Shaking One’s Faith in RCTs?

David Freedman in the November issue of The Atlantic profiles John Ioannidis, one of the world’s most respected and sought-after experts on the credibility of medical research.

Ioannidis argues that much of what biomedical researchers conclude in published studies is misleading, exaggerated, and often flat-out wrong. “Gold-standard” randomized controlled trials (RCTs) are not immune: 25 percent of published RCT results are subsequently convincingly refuted. When Ioannidis narrowed the focus to the very pinnacle of the research pyramid—49 of the most highly regarded medical research findings of the past 13 years—he found the same problems.

The causes are varied and complex, including researcher bias (and sometimes even outright fraud), publication bias that highlights certain types of findings and buries others, inadequate statistical procedures, and of course financial conflicts of interest when pharmaceuticals are tested.

“Medical research is not especially plagued with wrongness,” says Freeman. “Other meta-research experts have confirmed that similar issues distort research in all fields of science, from physics to economics (where the highly regarded economists J. Bradford DeLong and Kevin Lang once showed how a remarkably consistent paucity of strong evidence in published economics studies made it unlikely that any of them were right).”

Does any of this bear on the credibility of microfinance RCTs? I could speculate about some differences. For instance, I wouldn’t expect financial conflicts of interest to very much of a problem in current microfinance RCTs. And publication bias may be lesser; after all, a major reason that Dean Karlan and his buddies started IPA was to carry out research that academics would have a hard time getting published.

But I’m a statistical illiterate with no business opining on these issues. Watch this space tomorrow for Jonathan Morduch’s much more informed reflections on the matter.