To be or not to be in the sample? On using manipulation checks in experimental accounting research

Abstract

Purpose

Manipulation checks are a recommended for experimental accounting research. Usage of information gained by manipulation checks varies. In some studies, participants who failed the manipulation check are removed from the sample. Other studies report the results of the manipulation checks but still use the full sample. Some authors recommend removing participants who failed the manipulation check as a means to increase the power of the statistical analysis. Others warn that removing these participants endangers the randomization as a crucial precondition for gaining valid insights from experimental research. Until now, there is little research on how sensitive results react to exclusion of participants. The purpose of this paper is to study the effects of removing participants failing the manipulation checks on the evaluation of a hypothesis and the development of alternative usages of the information gained from manipulation checks.

Design/methodology/approach

Based on an analytical model and a simulation, the authors show how removing participants who fail the manipulation check affects experimental findings.

Findings

Simulations show that statistical results and conclusions drawn from an experiment differ substantially, depending on whether participants who failed the manipulation check are removed from the sample. As the participants who are removed are no random sub-sample, but share a certain property, the experimental results react strongly, typically showing significant results, where there are actually none.

Originality/value

This paper is, to the best of the authors’ knowledge, the first to address the sensitivity of experimental results to removing participants who fail the manipulation check from the sample and the implications for the validity of conclusions drawn from experimental accounting research. This paper’s contribution is a better way of using information gained in the manipulation check in the statistical analysis of the experimental data.

Citation

Kotzian, P., Stoeber, T., Hoos, F. and Weissenberger, B.E. (2020), "To be or not to be in the sample? On using manipulation checks in experimental accounting research", Accounting Research Journal, Vol. 33 No. 3, pp. 469-482. https://doi.org/10.1108/ARJ-06-2019-0128