Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024

Amazon Mechanical Turk (MTurk) has been one of the most popular platforms for online research in psychology and the social sciences in general. While concerns about MTurk data quality have been raised, the platform continues to be widely used. The question is whether the MTurk platform is suitable for research and, if so, whether it is used optimally. We conducted a systematic investigation of MTurk data quality and reliability, including main and replication experiments, with more than 1300 participants subdivided into three cohorts: (i) workers (i.e. participants on the MTurk platform) with master requirement (i.e. high-performing workers selected by MTurk), (ii) workers without master requirement, and (iii) workers without master requirement, but with a 95% or above approval rate. We found that master workers almost never missed attentional checks, exhibited high reliability and showed no tendency towards straightlining, therefore, these workers are recommended, especially when the naivety of participants is not a strong prerequisite and no large sample size is required. In contrast, the workers without restrictions or with a 95% or above approval-rate threshold missed many attentional checks, exhibited low reliability and showed a tendency towards straightlining, raising serious concerns about the suitability of these workers for research.