Scientific Studies Should Be Taken With A Giant Grain Of Salt

An article in Nature last week brought up a fact that’s been getting more and more attention in the past decade: most published research findings are false. It all started when a paper with that very title was published in 2005, containing mathematical proof showing that it’s very easy for a study to be wrong, due to three main reasons:

  • Researcher bias: this can be confirmation bias, where the researcher, maybe even unconsciously, sets the experiment up to succeed out of fear of being proven wrong; there are plenty of ways to massage every experiment, or to interpret the numbers, so that the outcome is the one hoped for. The bias can also be incentive-based, in which the researcher skews the results because of what’s at stake: a good job, grant money, a promising career, or even prestige and scientific stardom.
  • Selective reporting: the vast majority of studies published have positive results — that is, they confirmed what the researcher was looking for, e.g. that cell phones cause brain cancer. But if the research shows no effect from cell phones, it doesn’t get sent in for publication. And even if it does, the journal may decide to not publish because it’s not sexy enough.
  • Poor study design: maybe the samples were too small, the study wasn’t double-blinded or not randomized, it tested too many relationships, or relied on self-reporting; it could be the question was too broad, the results were mis-analyzed, or any of dozens of possible flaws that can make its way into a study, usually due to constraints on time or money.

John Ioannidis

 

The paper became the most downloaded article in the PLoS Medicine journal. The author, John Ioannidis, was the subject of an in-depth profile by The Atlantic in 2010. He has made it his mission in life to root out bad studies and prove them wrong. From the article, emphasis added:

Ioannidis laid out a detailed mathematical proof that, assuming modest levels of researcher bias, typically imperfect research techniques, and the well-known tendency to focus on exciting rather than highly plausible theories, researchers will come up with wrong findings most of the time. Simply put, if you’re attracted to ideas that have a good chance of being wrong, and if you’re motivated to prove them right, and if you have a little wiggle room in how you assemble the evidence, you’ll probably succeed in proving wrong theories right. His model predicted, in different fields of medical research, rates of wrongness roughly corresponding to the observed rates at which findings were later convincingly refuted: 80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials. (The Atlantic, November 2010)

Ioannidis then published a study in the Journal of the American Medical Association, which confirmed his predictions: 22% of the 49 most widely cited medical studies in the most widely cited journals were never even replicated; and 29% of the ones that were, were proven wrong. Therefore, his advice is to largely ignore studies: besides the fact that they often contradict each other, there are so many factors at play in something as complex as the human body, that researchers most often find flukes in the large, but limited data sets — not actual facts. It’s somewhat like claiming words in a giant bowl of alphabet soup have some significance.

 

But even if researchers stumble onto something significant, the study can only predict effects in a test environment, not in the real world. And very few studies go on for long enough to see if a factor affects important things, like death rate. (The few that do, generally contradict shorter studies.) Then, even in a magical unicorn of a perfect study, the results are averages over hundreds or thousands of people and are not even remotely tailored to our individual needs. And finally, even if you are the unlikely beneficiary of a perfect study done on a subset of the population that you belong to, the effects found are generally meager. In fact, we can’t even be sure that a study hasn’t already been refuted: sometimes it takes over ten years for researchers to stop citing one that was proven to be wrong. For example, people still think plastic water bottles leech toxic chemicals, even though the original study was severely flawed.

“The odds that anything useful will survive from any of these studies are poor,” says Ioannidis—dismissing in a breath a good chunk of the research into which we sink about $100 billion a year in the United States alone. (The Atlantic, November 2010)

And these problems exist in all fields of research, not just in bio-medicine. The New Yorker also interviewed Ioannidis in 2010, as part of an article on a related problem in scientific research, known as the Decline Effect. A very curious phenomenon, it makes significant findings of studies disappear over time — and it happens a lot. A study will find a drug very effective, then a subsequent study will find it less effective, and a third one even less so; the effect is generally explained as flukes being worked out of the studies over time, known as regression toward the mean.

 

It has even been shown that all economic studies might be wrong, and throughout history, instances where reality didn’t agree with the laws of physics have caused our understanding of the universe to become more and more complex — from the Earth sitting on the back of a turtle, to being suspended in the void by a force we can measure, but do not quite understand. And after centuries of discovering the laws of physics and building the complex language of advanced mathematics to describe them, scientists are now faced with the similar feat of discovering the laws that govern the remaining important and extremely complex systems: our bodies, our environment and our society.

All of the evidence points to us not even being close to understanding those systems. But eventually, we will get to the same level of clarity about them that we have about classical physics. Until then, Ioannidis came up with some rules for judging truthiness:

A study is more likely to be wrong if

  • it is small
  • the effect sizes are small
  • there are large financial interests, or prejudices
  • the scientific field is flexible with respect to study design
  • the scientific field is teeming with competing teams

See also:

 

From Nature, PLoS Medicine, Journal of the American Medical Association, The New Yorker and The Atlantic

3 Comments.

  1. The People Who Live Long Are The Happy Ones | Apt46 - pingback on May 30, 2012 at 6:35 pm
  2. Those Who Exercise More, Make More Money | Apt46 - pingback on June 8, 2012 at 5:39 pm

Trackbacks and Pingbacks: