Countering method bias in questionnaire‐based user studies

Abstract

Purpose

This paper seeks to discuss reliability problems associated with questionnaires, commonly employed in library and information science. It aims to focus on the effects of “common method variance” (CMV), which is a form of bias, and ways of countering these effects.

Design/methodology/approach

The paper critically reviews the use of existing tools for demonstrating reliability in questionnaire‐based studies. In particular, it focuses on Cronbach's alpha, “Harman's single factor test” and Lindell and Whitney's “marker variable” approach. The paper introduces an illustrative case study based on the work on metacognition and web searching. It goes on to make recommendations for researchers considering using a questionnaire‐based approach.

Findings

CMV is a problem affecting questionnaire‐based studies in different disciplines across social and information science. Where questionnaire items are more abstract, CMV has been found to be more of a problem. The widely used Cronbach alpha measure, of the reliability of a questionnaire, may often be affected by CMV. Where method bias dominates the results, a high alpha score may do no more than indicate that the questionnaire consistently allows participants to accede to their biases. “Harman's single factor test” is criticised as lacking sufficient foundation, and it is concluded that it should not be used. The marker variable approach is a useful option but must be applied with caution.

Practical implications

A number of practical implications for researchers are drawn. Questionnaire‐based work investigating abstract constructs should be assumed to carry a high likelihood of CMV, and therefore should attempt to avoid it and demonstrate the degree of success in this regard. Otherwise, interpretation of the results should assume CMV. A number of approaches to assessing and isolating CMV are discussed.

Originality/value

The paper draws attention to a problem that is arguably often overlooked in questionnaire‐based studies, namely method bias. It discusses a number of approaches whereby it may be identified and controlled.

Citation

Gorrell, G., Ford, N., Madden, A., Holdridge, P. and Eaglestone, B. (2011), "Countering method bias in questionnaire‐based user studies", Journal of Documentation, Vol. 67 No. 3, pp. 507-524. https://doi.org/10.1108/00220411111124569