What Does the Replication Crisis Mean for Science?

This discussion will center around the current "replication crisis" (a.k.a. "reproducibility crisis") in several areas of scientific research and allegations that have been floating around in the news media over the past couple years that this crisis indicates "science is broken", plagued by ideological biases, or perhaps running into inherent limitations in a way that suggests science is headed for a "paradigm shift" or perhaps even "ending".

The articles & videos you see linked below are intended to give you a basic overview of the major points in the debates over technological progress. As usual, I certainly don't expect you to watch all the videos & read all the articles prior to attending our discussion. The easiest way to prepare for our discussion is to just watch the 4 videos linked below, which come to about 41 minutes total. Hopefully, that doesn't seem like too much time to invest if you're really interested in these type of issues. If you prefer reading articles to watching videos, I've listed some beneath each video that make some interesting points.

(Over the next few days, I'll jot some notes under each article to summarize their main points so that you don't have to read them all to get the general gist.)

In terms of the discussion format, my general idea is that we'll briefly cover the 1st section which simply introduces the topic, and then we'll address the next 4 topics in the order presented here, spending roughly 25-30 minutes on each section.

WHAT IS THE "REPLICATION CRISIS" IN SCIENCE? IS THERE A DIFFERENCE BETWEEN "REPLICATION" AND "REPRODUCIBILITY"? WHAT DO SCIENTISTS THINK ABOUT THIS PROBLEM?

1. Nature, "Is There a Reproducibility Crisis in Science?" (video - 2:03 min.)

https://www.scientificamerican.com/video/is-there-a-reproducibility-crisis-in-science/

2. Monya Baker, "1,500 scientists lift the lid on reproducibility" (short article w/ infographics)

http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970

In the spring of 2016, the scientific journal Nature conducted a survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research. You can see the results of this survey summarized in the infographics below...

https://secure.meetupstatic.com/photos/event/4/c/2/5/600_462139493.jpeg

https://secure.meetupstatic.com/photos/event/4/7/3/4/600_462138228.jpeg

https://secure.meetupstatic.com/photos/event/4/7/3/5/600_462138229.jpeg

https://secure.meetupstatic.com/photos/event/4/7/3/7/600_462138231.jpeg

https://secure.meetupstatic.com/photos/event/4/7/f/9/600_462138425.jpeg

https://secure.meetupstatic.com/photos/event/4/7/f/8/600_462138424.jpeg

3. Mark Lieberman, "Replicability vs. Reproducibility - or is it the other way around?" (medium-length blog post)

http://languagelog.ldc.upenn.edu/nll/?p=21956

Typically, "reproducibility" refers to whether or not other researchers can take a research study's data and re-analyze it and get the same results. This is relatively quick & inexpensive and can help spot cases where the original researchers made mistakes or used fraudulent methods in analyzing the data, but it can't detect mistakes or used fraudulent methods in collecting the data. "Replication" refers to doing the same type of experiment from scratch in an effort to see if they original researcher's hypothesis holds up. This is more expensive & time-consuming, and it may be impossible in certain cases (e.g. research trying to determine the cause of a particular event rather than a natural law) but it avoids relying on the original researcher's dataset and thus isn't subject to their mistaken or fraudulent data. However, replications run the risk of mistakes or fraud from the new researchers. (Unfortunately, as Lieberman notes in the above article, some researchers have more recently started using the same terms with the reference more or less switched which has created confusion.)

WHY DOES THE REPLICATION CRISIS APPEAR TO BE WORSE IN PSYCHOLOGY & BIOMEDICAL RESEARCH?

4. Andrew Gelman, "Why is the scientific replication crisis centered on psychology?" (short blog post)

http://andrewgelman.com/2016/09/22/why-is-the-scientific-replication-crisis-centered-on-psychology/

5. Roger Peng, "A Simple Explanation for the Replication Crisis in Science" (short blog post)
   https://simplystatistics.org/2016/08/24/replication-crisis/

https://secure.meetupstatic.com/photos/event/4/7/3/6/600_462138230.jpeg

IS THE REPLICATION CRISIS DUE TO INSTITUTIONAL FAILURES & PERVERSE INCENTIVES? HOW COULD PRE-REGISTRATION OF EXPERIMENTS HELP?

6. Derek Muller, "Is Most Published Research Wrong?" (video - 12:21 min.)

https://www.youtube.com/watch?v=42QuXLucH3Q

7. Neuroskeptic, "Is Science Broken? Let's Ask Karl Popper" (short blog post)

http://blogs.discovermagazine.com/neuroskeptic/2015/03/15/is-science-broken-lets-ask-karl-popper/

8. Neuroskeptic, "Fixing Science - System and Politics" (short blog post)

http://blogs.discovermagazine.com/neuroskeptic/2012/04/14/fixing-science-systems-and-politics/#.WUcz0oVKu2x

IS THE REPLICATION CRISIS IN THE SOCIAL SCIENCES DUE TO IDEOLOGICAL HOMOGENEITY & POLITICAL BIASES?

https://secure.meetupstatic.com/photos/event/4/7/f/7/600_462138423.jpeg

Registered voter ratios, Democrat to Republican, among social scientists at 40 top US universities (Source: Econ Journal Watch)

https://www.timeshighereducation.com/news/democrats-far-outnumber-republicans-social-sciences#survey-answer

9. David Sloan Wilson & Jonathan Haidt, "Who is More Phobic About Science--Conservatives or Liberals?" (video - 15:00 min.)

https://www.youtube.com/watch?v=n9kJkuuedw0

10. Annie Bui, "Study Finds Bias in Social Psychology Community" (medium-length article)

http://cornellsun.com/2015/10/06/study-finds-bias-in-social-psychology-community/

11. Piercarlo Valedsolo, "Fixing the Problem of Liberal Bias in Social Psychology. We should seek to reduce bias, not balance it out." (medium-length article)

https://www.scientificamerican.com/article/fixing-the-problem-of-liberal-bias-in-social-psychology/

IS THE REPLICATION CRISIS DUE TO THE INHERENT LIMITATIONS OF SCIENCE, LIKE THE "HALF-LIFE OF FACTS", DIMINISHING RETURNS ON THEORETICAL PARADIGMS, OR THE LIMITS OF HUMAN UNDERSTANDING?

12. Sam Arbesman, "The Half-Life Of Facts" (video - 12:00 min.)

https://www.youtube.com/watch?v=GaxYnvd7YAM

https://secure.meetupstatic.com/photos/event/4/a/1/a/600_462138970.jpeg

13. Daniel Yudkin, "Social Psych's 'Replication Crisis' Is Just Your Everyday, Honest-to-Goodness Paradigm Shift" (short blog post)
    http://www.danielyudkin.com/thequaliast/2014/8/10/social-psychs-replication-crisis-is-just-your-everyday-honest-to-goodness-paradigm-shift

14. John Horgan, "Why I Think Science Is Ending" (long article)
    https://www.edge.org/conversation/john_horgan-why-i-think-science-is-ending