Why do we need double-blind studies?

In #science

I was recently reading some articles about dubious medical procedures, and the claims of miraculous healings. I was struck by a level of credulity with some of these claims, but at the same time felt the emotional tug of a healing claim and some internal push-back on the insistence on double-blind studies. I can understand how the people making the claim might balk at skepticism. Why do we always need double-blind studies in medicine anyway? Can't you do a procedure, especially if there is an obvious positive outcome (e.g. someone can walk who couldn't before, see when they were blind before, etc...), and see the outcome and not have to go through all of the hassle of making a double-blind study?

Not being a medical doctor, I started to read about it. I've learned a few interesting examples that I read here.

One about the effectiveness of a surgical procedure:

For a particularly dramatic example of this phenomenon, consider what happened when orthopedic surgeon Bruce Moseley, team physician for the Houston Rockets, decided he needed to properly evaluate the efficacy of an operation commonly used to treat the pain caused by arthritic knees. This surgery involves scraping away rough areas in the knee's cartilage. It is widely believed to be effective, and as many as 400,000 such surgeries are performed each year.

Mosely decided to see if the surgery really worked. He conducted a study in which five patients were given the real surgery and five were given fake surgery consisting of little incisions over the knee. He then followed his patients for two years.

The results were amazing. Interviews showed that pain and swelling were reduced just as much in the placebo group as in the group that received the real surgery. Four out of the five participants who experienced the fake surgery said it was so helpful they'd gladly recommend it to a friend. Glowing testimonials, in other words, mean nothing.

(the study is here)

Another about prostate size (placebo gave the same improvements even after 2 years).

So how does this happen? The article points out (with references) common examples which include:

  • Observer bias -- the person doing the measurement inadvertently noticing where the effect is, or not, because they expect it
  • Rosenthal Effect (aka the observer-expectancy effect)-- the observer inadvertently influencing the participants actions, so they act in the way the observer expects. this indirect communication can be remarkably subtle and effective.
  • Reinterpretation Effect - when observers reinterpret the result after the fact, given the expectations.
  • Placebo Effect - when no-treatment gives a positive result.
  • Memory Distortion - the observer's memory of results changes over time
  • Natural Course of the Disease: Illusion of Agency - natural improvements can possible lead to misattributing the treatment effectiveness
  • Study Drop Outs - participants with bad results may drop out of a study, leaving the good results remaining

To this list I'd add

  • Multiple treatments -- if a participant gets multiple treatments simultaneously, the observer could misattribute the effectiveness of one of the treatments.

Each of these problems are clearly possible with any of the miraculous healing claims I've read about and reviewed earlier. In the presence of a community supporting faith-healings, you get observer bias, Rothenthal effects, the illusion of agency, and placebo effects occuring. When the reports come some time after the events, the memory distortion, reinterpretation effects, and multiple treatment effects. Further, faith-healings have a massive drop-out problem -- many people are prayed for, and the people who don't get healed don't step forward.

How do you make sure that the treatment you are studying actually works? You have to rule out each of these effects:

  • keep all of the participants (rules put drop-out effects)
  • keep accurate pre-post observations (rules out reinterpretation and memory distortion effects)
  • use placebo control groups (rules out natural course)
  • keep all parties blind to the treatment and placebo groups (rules out observer and Rosenthal effects)
  • keep all treatments narrow and quantitative (rules out multiple treatment and memory distortion)