Category Archives: Science

Scientific Studies Should Be Taken With A Giant Grain Of Salt

Posted by apt46 on May 14, 2012 None comments

An article in Nature last week brought up a fact that’s been getting more and more attention in the past decade: most published research findings are false. It all started when a paper with that very title was published in 2005, containing mathematical proof showing that it’s very easy for a study to be wrong, due to three main reasons:

  • Researcher bias: this can be confirmation bias, where the researcher, maybe even unconsciously, sets the experiment up to succeed out of fear of bring proven wrong; there are plenty of ways to massage every experiment, or to interpret the numbers, so that the outcome is the one hoped for. The bias can also be incentive-based, in which the researcher skews the results because of what’s at stake: a good job, grant money, a promising career, or even prestige and scientific stardom.
  • Selective reporting: the vast majority of studies published have positive results — that is, they confirmed what the researcher was looking for, e.g. that cell phones cause brain cancer. But if the research shows no effect from cell phones, it doesn’t get sent in for publication. And even if it does, the journal may decide to not publish because it’s not sexy enough.
  • Poor study design: maybe the samples were too small, the study wasn’t double-blinded or not randomized, it tested too many relationships, or relied on self-reporting; it could be the question was too broad, the results were mis-analyzed, or any of dozens of possible flaws that can make its way into a study, usually due to constraints on time or money.

John Ioannidis

 

The paper became the most downloaded article in the PLoS Medicine journal. The author, John Ioannidis, was the subject of an in-depth profile by The Atlantic in 2010. He has made it his mission in life to root out bad studies and prove them wrong. From the article, emphasis added:

Ioannidis laid out a detailed mathematical proof that, assuming modest levels of researcher bias, typically imperfect research techniques, and the well-known tendency to focus on exciting rather than highly plausible theories, researchers will come up with wrong findings most of the time. Simply put, if you’re attracted to ideas that have a good chance of being wrong, and if you’re motivated to prove them right, and if you have a little wiggle room in how you assemble the evidence, you’ll probably succeed in proving wrong theories right. His model predicted, in different fields of medical research, rates of wrongness roughly corresponding to the observed rates at which findings were later convincingly refuted: 80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials. (The Atlantic, November 2010)

Ioannidis then published a study in the Journal of the American Medical Association, which confirmed his predictions: 22% of the 49 most widely cited medical studies in the most widely cited journals were never even replicated; and 29% of the ones that were, were proven wrong. Therefore, his advice is to largely ignore studies: besides the fact that they often contradict each other, there are so many factors at play in something as complex as the human body, that researchers most often find flukes in large, but limited data sets — not actual facts. It’s somewhat like claiming words in a giant bowl of alphabet soup have some significance.

 

But even if researchers stumble onto something significant, the study can only predict effects in a test environment, not in the real world. And very few studies go on for long enough to see if a factor affects important things, like death rate. (The few that do, generally contradict shorter studies.) Then, even in a magical unicorn of a perfect study, the results are averages over hundreds or thousands of people and are not even remotely tailored to our individual needs. And finally, even if you are the unlikely beneficiary of a perfect study done on a subset of the population that you belong to, the effects found are generally meager. In fact, we can’t even be sure that a study hasn’t already been refuted: sometimes it takes over ten years for researchers to stop citing a study which was proven to be wrong. For example, people still think plastic water bottles leech toxic chemicals, even though the original study was severely flawed.

“The odds that anything useful will survive from any of these studies are poor,” says Ioannidis—dismissing in a breath a good chunk of the research into which we sink about $100 billion a year in the United States alone. (The Atlantic, November 2010)

And these problems exist in all fields of research, not just in bio-medicine. The New Yorker also interviewed Ioannidis in 2010, as part of an article on a related problem in scientific research, known as the Decline Effect. A very curious phenomenon, it makes significant findings of studies disappear over time — and it happens a lot. A study will find a drug very effective, then a subsequent study will find it less effective, and a third one even less so; the effect is generally explained as flukes being worked out of the studies over time, known as regression toward the mean.

 

It has even been shown that all economic studies might be wrong, and throughout history, instances where reality didn’t agree with the laws of physics have caused our understanding of the universe to become more and more complex — from the Earth sitting on the back of a turtle, to being suspended in the void by a force we can measure, but do not quite understand. And after centuries of discovering the laws of physics and building the complex language of advanced mathematics to describe them, scientists now faced with the similar feat of discovering the laws that govern the other important and extremely complex systems: our bodies, our environment and our society.

All of the evidence points to us not even being close to understanding those systems. But eventually, we will get to the same level of clarity about them that we have about classical physics. Until then, Ioannidis came up with some rules for judging truthiness:

A study is more likely to be wrong if

  • it is small
  • the effect sizes are small
  • there are large financial interests, or prejudices
  • the scientific field is flexible with respect to study design
  • the scientific field is teeming with competing teams

See also:

 

From Nature, PLoS Medicine, Journal of the American Medical Association, The New Yorker and The Atlantic

Sleeping Pills Tied To Higher Risk Of Death

Posted by apt46 on February 28, 2012 None comments

A new study in the British Medical Journal looked at the mortality rates associated with the use of “hypnotic drugs,” or sleeping pills. Older studies have looked at this before and found that people using sleeping pills were more likely to die, but had some shortcomings which this study corrected. The results:

  • Patients receiving prescriptions for zolpidem, temazepam and other hypnotics suffered over four times the mortality as the matched hypnotic-free control patients.

  • Even patients prescribed fewer than 18 hypnotic doses per year experienced increased mortality, with greater mortality associated with greater dosage prescribed.

  • Among patients prescribed hypnotics, cancer incidence was increased for several specific types of cancer, with an overall cancer increase of 35% among those prescribed high doses

 

The researchers adjusted the data to remove effects caused by age, gender, smoking, body mass index, ethnicity, marital status, alcohol use and prior cancer. However, without more effort, research like this finds correlation, not causation: it’s hard to tell whether the sleeping pills caused the higher risk of death, or whether the patients with sleep problems have a higher risk of death to begin with and would’ve ended up the same without the sleeping pill.

To combat this problem, the study also controlled for known causes of death and the association between sleeping pill use and mortality remained significant. Or, in science speak: “neither the level of individual health nor the presence of particular categories of comorbidity explains the bulk of the hazard associated with the use of hypnotic medications.” However, they still can’t be sure that they didn’t leave something out, and an unknown factor is causing both the insomnia and higher mortality. Still, about 8% of Americans use sleeping pills, so caution is warranted. The paper closes with this — emphasis added:

The meagre benefits of hypnotics, as critically reviewed by groups without financial interest would not justify substantial risks. A consensus is developing that cognitive-behavioural therapy of chronic insomnia may be more successful than hypnotics. Against meagre benefits, it is prudent to weigh the evidence of mortality risks from the current study and 24 previous reports, in order to reconsider whether even short-term use of hypnotics, as given qualified approval in National Institute for Clinical Excellence guidance, is sufficiently safe.

 

See also:

 

From The British Medical Journal, via Slashdot

The Faster-Than-Light CERN Speeds Were An Error

Posted by apt46 on February 22, 2012 None comments

Back in September of 2011, experiments at CERN showed neutrinos traveling faster than the speed of light. This would imply that we could send messages backward in time, so it should come as no surprise that the whole thing was an error, in this case due to a loose cable. Since the cable in question wasn’t tightened enough, the data arrived 60 nanoseconds later than it should’ve; the neutrinos in question arrived 60 nanoseconds earlier than they should’ve. So it looks like carelessness wins again. Of course, they’ll have to confirm this with a new round of experiments, but for now, the dinosaurs are still safe.

The Neutrinos from Dimension X, on Teenage Mutant Ninja Turtles

 

Update, 17 March 2012: more evidence that neutrinos can’t travel faster than light has been uncovered.

From ScienceMag, via Slashdot

Math And Plants

Posted by apt46 on February 7, 2012 None comments

One of the coolest things about math is when it can actually be applied to the real world, like when it helps you figure out in what direction to shoot a cannon ball to hit your nemesis’ house, or how to beat the house at Black Jack. But even those applications are almost always about physics or chemistry and rarely about biology, since life can’t be bothered with such rational things as math.

The video below is therefore very cool because it shows that the leaves in plants are generally spaced in a mathematical series, because that’s the optimal way for leaves to grow such that they fill in the most available space. And because of that simple fact, plant growth can be predicted mathematically, which makes math that much more interesting.

From YouTube, via Neatorama

Correlation Is Not Causation

Posted by apt46 on December 13, 2011 1 comments

Business Week has a series of graphs which use statistics that lead to ridiculous conclusions. Their point: just because two things appear to be related, it doesn’t meant they actually are; and even if they are, it doesn’t mean that one caused the other.

From Business Week, via Neatorama

Fertility In Women Declines Rapidly After 30

Posted by apt46 on December 1, 2011 None comments

A survey of a thousand mostly urban professional women (around 30 years old, mostly white, college-educated with a full-time job and medical insurance, and had no kids, nor were actively trying) found a few gaps in knowledge of fertility, so the paper points out some things everyone should know:

  • Fertility declines after 30, and is very low after 40: only 10% of women under 30 are infertile, but that number jumps to 25% for women aged 35-44. In fact, because of the big impact age has, a woman over 35 is considered infertile if she hasn’t conceived after 6 months of trying; for those under 30, doctors wait 12 months to make that call. In over 70% of couples that have fertility problems, the woman is over 40.
  • Other potential risk factors include smoking, drinking too much alcohol, having a poor diet, exercising too much (apparently, yes, you can do that) and having a “social disease“
  • The pill is not a risk factor for fertility: there’s no link between any hormonal contraceptive and infertility
  • It general, it’s harder to get pregnant than most people think: about 5 months for a 20-year old, 6 months for a 30-year old and 7-12 months for a 40-year old. At the end of a month of trying (that is, attempting to fertilize one egg), the chances of pregnancy during that month will have been less than 30% for women under 30, and less than 10% for the 40 year old.
  • In any given couple with fertility issues, the chances that the problem is the man, the woman, or both are split equally between the three.
  • Besides in-vitro fertilization, there are two other main methods of dealing with infertility: hormonal injections and pills.
  • The chance of getting pregnant via in-vitro fertilization is only 20-29%.
  • The age of the egg used for in-vitro matters a lot: a 35-year old woman doing the procedure has the best chance by using a 20-year old’s egg

Things those surveyed knew:

  • Menopause tends to happen between the ages of 50 and 54
  • Fertility declines 10+ years before menopause
  • Age, heredity, stress, and being over- or underweight are risk factors for fertility
  • About 10-29% of all couples are infertile

Other findings:

  • Women tend to want to have their first kid about 7 years after their mother had her first, and they tend to want 2 kids: one at 32 and another at 35.
  • 75% of the women weren’t worried about trying to conceive
  • 75% of the women were on the pill; 86% were using some kind of family planning
  • 1% of babies born in the US were conceived in-vitro

NPR has a story on the subject which points out that a lot of women wait to have a baby after 35, only to find out that the odds are very much against them. Unfortunately, it’s a tough issue to educate people on, because it’s so touchy. An ad campaign back in 2001 with a baby bottle shaped like an hourglass drew a lot of criticism from women’s rights groups.

 

This blurry picture is all that remains on the Internet of the controversial 2001 baby hourglass ad campaign

 

It’s worth mentioning that the survey was done by EMD Serono, which is the US arm of Merck KGaA, which is a German pharmaceutical. Given that the target of the survey was women around 30 who had no kids, they were probably just trying to figure out how to sell more pills to rich white women: either pills to prevent birth (did you know they don’t affect fertility?) or pills to promote birth (did you know we sell fertility hormones?). Drugs: the cause of, and solution to, all our problems.

Surprisingly, the German Merck KGaA is not connected in any way to the American pharmaceutical Merck & Co: during World War I, the American assets of the German Merck were seized by the US government and set up as a completely separate company. The two companies were also in the news earlier this week, because American Merck wanted German Merck’s Facebook page; Facebook initially agreed and transferred the page over, but after German Merck complained, it decided to go the Solomon route and give the page to neither of them.

From EMD Serono, via NPR and IT World

There’s No Such Thing As Renewable Energy

Posted by apt46 on November 29, 2011 1 comments

Previously, we’ve seen that there’s no such thing as sustainable energy because all energy use produces heat, which at some point (around the year 2200, at current rates) will make the planet too hot for us. Along the same vein, The Bulletin of the Atomic Scientists has an article which makes the point that there’s no such thing as renewable energy either. So even if we could use all the energy we wanted — which we can’t because eventually, we’d boil — we are never going to have a source of energy that’s inexhaustible.

At this point, most people say “ok, the sun isn’t technically inexhaustible, but it’ll be around for a couple billion more years, so for all intents and purposes, it’s renewable.” And it’s true, sunlight in and of itself is virtually renewable, but the problem is that we can’t make direct use of sunlight: we need solar panels, and they’re made using non-renewable resources like neodymium. They also have a shelf-life, and need maintenance. Solar plants, like all power plants, are designed for a couple of decades of use, and require a lot of groundwater during maintenance, for cleaning and cooling; in the desert, where solar plants generally live, groundwater is not renewable.

Wind power requires wind mills which are built with steel, concrete, and rare earth metals. Same with hydropower, which needs dams and turbines. Geothermal power and biomass also need turbines and engines; and what’s more is that unlike solar, wind and hydro power, these two energy sources tend to be used at a rate faster than they can be renewed. So even though the source of the energy is renewable, it doesn’t renew quickly enough. After all, trees are biomass, and until oil became the primary energy source, deforestation was a very real concern.

The truth is, there’s no horn of plenty when it comes to energy, and at our current usage, even if the entire planet switched to 100% solar power tomorrow, we will eventually run out of energy. It might take longer and damage the environment less (except for heating up the planet), but it will happen. So ‘renewable,’ ‘sustainable,’ and ‘green’ are serious misnomers when talking about even the most politically correct of energy sources. At best, we can say that alternative energy sources are more efficient and clean — but win/win solutions, they are not.

Now, it could be that some magical new technology will be invented that’s made from abundant elements like carbon and hydrogen, doesn’t create heat as a byproduct, and uses sunlight as an energy source. But until we can plug in our iPhones into ficus trees, it’s better if we just assume that won’t happen.

From The Bulletin of Atomic Scientists, via Slashdot

Night Owls Are Smarter

Posted by apt46 on November 29, 2011 None comments

Back in 1999, a couple of psychologists at the University of Sydney did a study in which they asked 420 people if they were night owls or early birds, then gave them IQ tests — all to see if Ben Franklin was right, and early to rise did make people wise. (In technical terms, they were assessing the correlation between diurnal preference and intelligence.) The study concluded that “contrary to conventional folk wisdom, evening-types are more likely to have higher intelligence scores.”

From The National Library of Medicine, via Neatorama

How To Turn Off Your Gag Reflex

Posted by apt46 on November 10, 2011 None comments

Just fold your thumb inside a fist and squeeze, as pictured below. For you know…. when you go to the dentist. [cough.] They even did a study on it.

From The Journal of the American Dental Association, via Reddit and Lifehacker

Makeup Is An Even Better Lie Than You Thought

Posted by apt46 on October 18, 2011 1 comments

Proctor & Gamble funded a study done by Harvard in which they, of course, found out that makeup makes women look prettier than they actually are. But the really interesting thing is that it makes them look more competent, too. It’s unclear how much of it is due to the halo effect, since without information to the contrary, we tend to assume that good looking people are also nice, smart, great tennis players, etc. But if you’re trying to make people superficially think you know what you’re talking about — regardless of if you actually do or not, — then make-up is your friend. Unless you’re Erin Andrews, in which case makeup can only hurt. Also unless you’re a guy, because that may backfire.

The study's four makeup-aided looks: bare face, natural, professional and glamorous.

 

It’s kind of amazing how in the photo, the girl’s likely profession goes from WNBA player to hotel clerk to bank teller to pharmaceutical rep. The study didn’t allow the women to look in the mirror after the makeup was applied on them, so they wouldn’t alter their facial expression based on how they looked. And they had two different groups of mostly women grade them: the first only looked at the pictures for a blink of an eye (a quarter of a second), while the others looked as much as they wanted. The results were the same for both groups, which goes to show you just how snappy snap judgments are.

From Harvard, via The New York Times and Neatorama