Hype, hope, and how to tell the difference (2025)

A few thoughts on hype and hope in medical research.

I've seen a fair number of thrilling new treatments come and go over the years. Sometimes things don’t work out because of unforeseen technical or implementation difficulties; sometimes there’s fraud involved; sometimes you simply never figure it out.

Often, closer reading of a much-hyped paper will give you a sense of stormclouds ahead. Effect sizes might be smaller than one would expect from the headlines; control groups can inflate the outcomes because of, for example, a “nocebo” effect of not receiving the intervention; and the participants included in a trial might not reflect patients in a real-world setting.

There are also some less tangible aspects of much-hyped advances that curb my enthusiasm. What follows is subjective and anecdotal, but you might find it useful.

Multiple indications, few robust trials; or, a solution looking for a problem: Some interventions start off with indications for one condition, but rapidly spread their remit far, far beyond. Robust testing for the original indication is delayed, while a blizzard of early-phase trials for other indications appears in the literature.

Related to this, sometimes a single drug, nutritional substance, or other intervention is tried for all sorts of different conditions, despite a gradual accretion of negative results—surely it must be good for something!

Bundled evidence: Related to the above. Scraps of evidence for an intervention or the significance of a particular mechanism are bundled together in a rapid-fire, overwhelming narrative. The individual bits of data might be marginal or even contradictory. Bundling them together without proper evaluation of quality gives the impression that weak evidence is made strong if you just cite enough of it.

Gnome logic: Term taken from the South Park episode. I’m not saying that we always need complete understanding of a mechanism to have confidence that an intervention is effective. But it’s best to be cautious when poor understanding of a mechanism is obscured by jargon and hand-waving, or when it’s assumed that a proposed intervention will lead to a desirable outcome because, well, just because. Mental health science in particular is the graveyard of things that made intuitive sense and ought to have worked.

Finally, the bookstore test: If you go to the health section of your local bookstore, is there a proliferation of books on this topic? Do they make promises of radical change that will transform everything you thought you knew? If so, beware.

If you have any other rules—actually, they’re more like guidelines—to add to this list, do let me know.