Overview
Research is not typically a fast process. It can take years for an academic article to make it to print. This is a problem given the ever-quickening pace of change in our world today. By the time a set of research studies gets published, the results may no longer be relevant, or worse, no longer accurate.
Nevertheless, outdated research is still heavily cited. But are non-academic audiences more discerning? Perhaps we’ve learned to place less trust in the results of decades-old studies. To find out, we conducted a simple experiment.
The Experiment
We conducted a randomized controlled trial with 400 people on Amazon Mechanical Turk in which participants were asked to rate the believability of a research finding. Unbeknownst to participants, we had randomized whether the year in which the research was allegedly conducted was 1999 or 2019.
Participants read the text below:
Research is not typically a fast process. It can take years for an academic article to make it to print. This is a problem given the ever-quickening pace of change in our world today. By the time a set of research studies gets published, the results may no longer be relevant, or worse, no longer accurate.
Nevertheless, outdated research is still heavily cited. But are non-academic audiences more discerning? Perhaps we’ve learned to place less trust in the results of decades-old studies. To find out, we conducted a simple experiment.
The Experiment
We conducted a randomized controlled trial with 400 people on Amazon Mechanical Turk in which participants were asked to rate the believability of a research finding. Unbeknownst to participants, we had randomized whether the year in which the research was allegedly conducted was 1999 or 2019.
Participants read the text below:
The following claim may or may not be true:
A research study from [1999 / 2019] at Northwestern University suggests that people looking at higher-elevation sceneries are more willing to purchase new products.
A research study from [1999 / 2019] at Northwestern University suggests that people looking at higher-elevation sceneries are more willing to purchase new products.
Participants were then asked, "How strongly do you believe this research finding is true?" and responded by answering on a 1-7 scale (1 = Not at all, 7 = Very much)
Results
We found no significant difference in believability (1-7 scale) between the 1999 research study (avg. = 4.13) and the 2019 research study (avg. = 4.00) (p = 0.447). In fact, the 1999 article was rated as slightly more believable.
Results
We found no significant difference in believability (1-7 scale) between the 1999 research study (avg. = 4.13) and the 2019 research study (avg. = 4.00) (p = 0.447). In fact, the 1999 article was rated as slightly more believable.
Conclusion
It seems that neither scientists nor everyday people discount research studies from over 20 years ago. At least not in this context of consumer purchasing. Perhaps research in other fields, like computer science, are discounted more quickly. Or perhaps it takes longer than 20 years. It’d be interesting to see just how many years it takes for such discounting to kick in. Maybe it never does. I guess only time will tell.
|
Daniel BrownDaniel Brown is an award-winning researcher, social scientist, and founder of AB Labs. He holds a doctorate in Management from Harvard University and has conducted over 100 randomized controlled trials in social psychology.
|
Popular Experiments
COVID MasksDoes wearing a COVID mask affect how others think of you?
|
Video GamesAre video games more enjoyable than board games?
Does age or gender matter? |
Zero-Sum PoliticsDo Democrats or Republicans view society as win-lose?
|