Estimating the Reproducibility of Psychological Science

The Choice Factory: 25 Behavioural Biases That Influence What We Buy
by Richard Shotton
Published 12 Feb 2018

, by Evan Davis, John Kay, and Jonathan Star [London Business School Review, Vol. 2, No. 3, pp. 1–23, 1991] Marketers Are from Mars, Consumers Are from New Jersey by Bob Hoffman [2015] Bias 16: The curse of knowledge Made to Stick: Why Some Ideas Survive and Others Die by Chip Heath and Dan Heath [2008] The Wiki Man by Rory Sutherland [2011] Bias 17: Goodhart’s law Long and Short of It: Balancing Short- and Long-Term Marketing Strategies by Les Binet and Peter Field [2012] Management in 10 Words by Terry Leahy [2012] Leading by Alex Ferguson and Michael Moritz [2015] Bias 18: The pratfall effect: Social Animal by Elliot Aronson [1972] The Wasp Factory by Iain Banks [1984] Bias 19: Winner’s curse The Winner’s Curse: Paradoxes and Anomalies of Economic Life by Richard Thaler [1991] Originals: How Non-Conformists Move the World by Adam Grant [2016] ‘Harnessing naturally occurring data to measure the response of spending to income’, by Michael Gelman, Shachar Kariv, Matthew Shapiro, Dan Silverman, Steven Tadelis [Science, Vol. 345, No. 6193, pp. 212–215, 2014] ‘The Psychology of Windfall Gains’, by Hal Arkes, Cynthia Joyner, Mark Pezzo, Jane Gradwohl Nash, Karen Siegel-Jacobs, Eric Stone Eric [Organizational Behaviour and Human Decision Processes, Vol. 59, No. 3, pp. 331–347, 1994] On the Fungibility of Spending and Earnings – Evidence from Rural China and Tanzania by Luc Christiaensen and Lei Pan [2012] Bias 20: The power of the group ‘Humour in Television Advertising: The Effects of Repetition and Social Setting’, by Yong Zhang and George Zinkhan [Advances In Consumer Research, Vol. 18, pp. 813–818, 1991] ‘Feeling More Together: Group Attention Intensifies Emotion’, by Garriy Shteynberg, Jacob Hirsh, Evan Apfelbaum, Jeff Larsen, Adam Galinsky, and Neal Roese [Emotion, Vol. 14, No. 6, pp. 1102–1114, 2014] Bias 21: Veblen goods ‘Commercial Features of Placebo and Therapeutic Efficacy’, by Rebecca Waber, Baba Shiv, Ziv Carmon; Dan Ariely [Journal of the American Medical Association, Vol. 299, No.9, pp. 1016–1017, 2008] Bias 22: The replicability crisis ‘Why Susie Sells Seashells by the Seashore: Implicit Egotism and Major Life Decisions’, by Brett Pelham, Matthew Mirenberg, and John Jones [Journal of Personality and Social Psychology, Vol. 82, No. 4, pp. 469–487, 2002] ‘Rich the banker? What’s not in a Name’, by Tim Harford [2016], www.timharford.com/2016/11/rich-the-banker-whats-not-in-a-name ‘Estimating the reproducibility of psychological science’, by Brian Nosek et al. [Science, Vol. 349, No. 6251, 2015] ‘Comment on “Estimating the reproducibility of psychological science”’, by Daniel Gilbert, Gary King, Stephen Pettigrew and Timothy Wilson [Science, Vol. 351, Issue 6277, p. 1037, 2016] ‘Meta-assessment of bias in science’, by Daniele Fanelli, Rodrigo Costats, and John Ioannidis [Proceedings of the National Academy of Sciences, Vol. 114, No. 14, pp. 3714–3719, 2017] ‘Evaluating replicability of laboratory experiments in economics’, by Colin F.

Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth
by Stuart Ritchie
Published 20 Jul 2020

See also Le Texier’s reply to a more recent (at the time of writing unpublished) version: Thibault Le Texier, ‘The SPE Remains Debunked: A Reply to Zimbardo and Haney (2020)’, Preprint, PsyArXiv (24 Jan. 2020); https://doi.org/10.31234/osf.io/9a2er 26.  Open Science Collaboration, ‘Estimating the Reproducibility of Psychological Science’, Science 349, no. 6251 (28 Aug. 2015): aac4716; https://doi.org/10.1126/science.aac4716 27.  77 per cent: Colin F. Camerer et al., ‘Evaluating the Replicability of Social Science Experiments in Nature and Science between 2010 and 2015’, Nature Human Behaviour 2, no. 9 (Sept. 2018): pp. 637–44; https://doi.org/10.1038/s41562-018-0399-z 28.  

…

I’ve been stressing the importance of robust results, but in making the case that there’s a replication crisis, I’m relying on multi-study replication attempts that weren’t representative samples of all the scientific literature. The conclusion of ‘only about half of published results replicate’ might not generalise to all science. This was a point made in a critique of one of the replication survey studies: D. T. Gilbert et al., ‘Comment on “Estimating the Reproducibility of Psychological Science”’, Science 351, no. 6277 (4 Mar. 2016): p. 1037; https://doi.org/10.1126/science.aad7243. Whereas I disagree with many of the arguments made in this rejoinder (for some reasons to be sceptical of it, see Daniël Lakens, ‘The Statistical Conclusions in Gilbert et al (2016) Are Completely Invalid’, The 20% Statistician, 6 March 2016; https://daniellakens.blogspot.com/2016/03/the-statistical-conclusions-in-gilbert.html), the criticism about representativeness was fair.

Thinking in Bets
by Annie Duke
Published 6 Feb 2018

New York: Current, 2014. Oettingen, Gabriele, and Peter Gollwitzer. “Strategies of Setting and Implementing Goals.” In Social Psychological Foundations of Clinical Psychology, edited by James Maddox and June Price Tangney, 114–35. New York: Guilford Press, 2010. Open Science Collaboration. “Estimating the Reproducibility of Psychological Science.” Science 349, no. 6251 (August 28, 2015): 943 and aac4716-1–8. Oswald, Dan. “Learn Important Lessons from Lombardi’s Eight-Hour Session.” HR Hero (blog), March 10, 2014. http://blogs.hrhero.com/oswaldletters/2014/03/10/learn-important-lessons-from-lombardis-eight-hour-session.

Know Thyself
by Stephen M Fleming
Published 27 Apr 2021

“Birds Have Primate-Like Numbers of Neurons in the Forebrain.” Proceedings of the National Academy of Sciences 113, no. 26 (2016): 7255–7260. Onishi, Kristine H., and Renée Baillargeon. “Do 15-Month-Old Infants Understand False Beliefs?” Science 308, no. 5719 (2005): 255–258. Open Science Collaboration. “Estimating the Reproducibility of Psychological Science.” Science 349, no. 6251 (2015). Ortoleva, Pietro, and Erik Snowberg. “Overconfidence in Political Behavior.” American Economic Review 105, no. 2 (2015): 504–535. Palser, E. R., A. Fotopoulou, and J. M. Kilner. “Altering Movement Parameters Disrupts Metacognitive Accuracy.”

Restarting the Future: How to Fix the Intangible Economy
by Jonathan Haskel and Stian Westlake
Published 4 Apr 2022

utm_campaign=Matt%27s%20Thoughts%20In%20Between&utm_medium=email&utm_source=Revue%20newsletter. 18. Fukuyama 1995; Putnam 1994. REFERENCES Aarts, Alexander A., Joanna E. Anderson, Christopher J. Anderson, Peter R. Attridge, Angela Attwood, Jordan Axt, Molly Babel, et al. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349 (6251). https://doi.org/10.1126/science.aac4716. Abel, A. B., A. K. Dixit, J. C. Eberly, and R. S. Pindyck. 1996. “Options, the Value of Capital, and Investment.” Quarterly Journal of Economics 111 (3): 753–77. https://doi.org/10.2307/2946671. Acemoglu, Daron. 1999.

Randomistas: How Radical Researchers Changed Our World
by Andrew Leigh
Published 14 Sep 2018

Lalande, ‘A peculiar prevalence of p values just below .05’, Quarterly Journal of Experimental Psychology, vol. 65, no. 11, 2012, pp. 2271–9; Kewei Hou, Chen Xue & Lu Zhang, ‘Replicating anomalies’, NBER Working Paper 23394, Cambridge, MA: National Bureau of Economic Research, 2017. 46Alexander A. Aarts, Joanna E. Anderson, Christopher J. Anderson, et al., ‘Estimating the reproducibility of psychological science’, Science, vol. 349, no. 6251, 2015. 47This represented two out of eighteen papers: John P.A. Ioannidis, David B. Allison, Catherine A. Ball, et al., ‘Repeatability of published microarray gene expression analyses’, Nature Genetics, vol. 41, no. 2, 2009, pp. 149–55. 48This represented six out of fifty-three papers: C.

The Art of Statistics: Learning From Data
by David Spiegelhalter
Published 14 Oct 2019

Raftery, ‘Bayes Factors’, Journal of the American Statistical Association 90 (1995), 773–95. 9. J. Cornfield, ‘Sequential Trials, Sequential Analysis and the Likelihood Principle’, American Statistician 20 (1966), 18–23. CHAPTER 12: HOW THINGS GO WRONG 1. Open Science Collaboration, ‘Estimating the Reproducibility of Psychological Science’, Science 349:6251 (28 August 2015), aac4716. 2. A. Gelman and H. Stern, ‘The Difference Between “Significant” and “Not Significant” Is Not Itself Statistically Significant’, American Statistician 60:4 (November 2006), 328–31. 3. Ronald Fisher, Presidential Address to the first Indian Statistical Congress, 1938, Sankhyā 4(1938), 14–17. 4.

The Art of Statistics: How to Learn From Data
by David Spiegelhalter
Published 2 Sep 2019

Raftery, ‘Bayes Factors’, Journal of the American Statistical Association 90 (1995), 773–95. 9. J. Cornfield, ‘Sequential Trials, Sequential Analysis and the Likelihood Principle’, American Statistician 20 (1966), 18–23. CHAPTER 12: HOW THINGS GO WRONG 1. Open Science Collaboration, ‘Estimating the Reproducibility of Psychological Science’, Science 349:6251 (28 August 2015), aac4716. 2. A. Gelman and H. Stern, ‘The Difference Between “Significant” and “Not Significant” Is Not Itself Statistically Significant’, American Statistician 60:4 (November 2006), 328–31. 3. Ronald Fisher, Presidential Address to the first Indian Statistical Congress, 1938, Sankhyā 4(1938), 14–17. 4.

Calling Bullshit: The Art of Scepticism in a Data-Driven World
by Jevin D. West and Carl T. Bergstrom
Published 3 Aug 2020

Press release. National Aeronautics and Space Administration. January 31, 2018. https://www.nasa.gov/​feature/​nasa-twins-study-confirms-preliminary-findings. NORC General Social Survey. 2017. Data compiled by the Pew Research Center. Open Science Collaboration. “Estimating the Reproducibility of Psychological Science.” Science 349 (2015): aac4716. Pauling, L., and R. B. Corey. “A Proposed Structure for the Nucleic Acids.” Proceedings of the National Academy of Sciences 39 (1953): 84–97. Pauling, Linus. Vitamin C and the Common Cold. 1st edition. San Francisco: W. H. Freeman, 1970.

The Data Detective: Ten Easy Rules to Make Sense of Statistics
by Tim Harford
Published 2 Feb 2021

An alternative metric is to ask how many of the replication studies produced results that passed the standard (but rather problematic) hurdle of “statistical significance.” Only thirty-six did; ninety-seven of the original studies had cleared that hurdle. Open Science Collaboration, “Estimating the Reproducibility of Psychological Science,” Science 28, no. 6251 (August 2015), 349, DOI: 10.1126/science.aac4716. 12. Brief film on YouTube: “Derren Brown—Ten Heads in a Row,” ThinkSceptically, April 8, 2012, https://www.youtube.com/watch?v=n1SJ-Tn3bcQ. 13. Planet Money, episode 677. 14. F. J. Anscombe, “Fixed-Sample-Size Analysis of Sequential Observations,” Biometrics 10, no. 1 (1954), 89–100, JSTOR, www.jstor.org/stable/3001665; and Andrew Gelman’s blog post “Statistical Inference, Modeling and Social Science,” May 2, 2018, https://statmodeling.stat.columbia.edu/2018/05/02/continuously-increased-number-animals-statistical-significance-reached-support-conclusions-think-not-bad-actually/. 15.

Can It Happen Here?: Authoritarianism in America
by Cass R. Sunstein
Published 6 Mar 2018

Princeton, NJ: Princeton University Press, 2017. Camerer, Colin F., and Eric J. Johnson. “The Process-Performance Paradox in Expert Judgment: How Can Experts Know So Much and Predict So Badly?” Research on Judgment and Decision Making: Currents, Connections, and Controversies 342 (1997). Collaboration, Open Science. “Estimating the Reproducibility of Psychological Science.” Science 349, no. 6251 (2015): 10.1126/science.aac4716. DiPrete, Thomas A., and Gregory M. Eirich. “Cumulative Advantage as a Mechanism for Inequality: A Review of Theoretical and Empirical Developments.” Annual Review of Sociology 32, no. 1 (2006): 271–97. Dunning, Thad.

Rationality: What It Is, Why It Seems Scarce, Why It Matters
by Steven Pinker
Published 14 Oct 2021

Journal of Elections, Public Opinion and Parties, 29, 222–44. https://doi.org/10.1080/17457289.2018.1465061. O’Keefe, S. M. 2020. One in three Americans would not get COVID-19 vaccine. Gallup, Aug. 7. https://news.gallup.com/poll/317018/one-three-americans-not-covid-vaccine.aspx. Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science, 349. https://doi.org/10.1126/science.aac4716. Paresky, P., Haidt, J., Strossen, N., & Pinker, S. 2020. The New York Times surrendered to an outrage mob. Journalism will suffer for it. Politico, May 14. https://www.politico.com/news/magazine/2020/05/14/bret-stephens-new-york-times-outrage-backlash-256494.

Model Thinker: What You Need to Know to Make Data Work for You
by Scott E. Page
Published 27 Nov 2018

The Logic of Collective Action: Public Goods and the Theory of Groups. Cambridge, MA: Harvard University Press. O’Neil, Cathy 2016. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. New York, NY: Crown. Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349: 6251. Organization for Economic Co-operation and Development. 1996. The Knowledge Based Economy. Paris: OECD. Ormerod, Paul. 2012. Positive Linking: How Networks Can Revolutionise the World. London: Faber and Faber. Ostrom, Elinor. 2004. Understanding Institutional Diversity.