Ithaca College

The Crowdsourcing Hypothesis Tests project found that whether effects replicated had more to do with the effects themselves than with the skill of the replicating teams. It also showed that experts can predict which effects will replicate.

A team of 50 researchers at 15 institutions independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Professor Leigh Ann Vaughn was part of a separate group of 144 experts, the Crowdsourcing Hypothesis Tests Collaboration, who lent their expertise as evaluators of study quality and forecasters. The Collaboration also revised and approved the final version of this article.

This research is important because it showed that there was wide variation in the sizes of effects, almost none of which was attributable to the skill of the replicating teams and almost all of which was attributable to the effects themselves. Additionally, this research is important because it shows that the Collaborations’ predictions correlated significantly with study results, which indicates that experts can predict effect sizes of research with some degree of accuracy, even when they only know a little about the hypotheses and methods.

This article was published in the journal Psychological Bulletin on January 16, 2020.