(2023-05-31) Alexander All Medications Are Insignificant In The Eyes Of God And Traditional Effect Size Criteria

Scott Alexander: All Medications Are Insignificant In The Eyes Of God And Traditional Effect Size Criteria. SSRI antidepressants like Prozac were first developed in the 1980s and 1990s... skeptics found substantial bias in these early trials; several later analyses that corrected for this all found effect sizes (compared to placebo) of only 0.30.

Is an effect size of 0.30 good or bad? The usual answer is “bad”. The UK’s National Institute for Clinical Excellence used to say that treatments were only “clinically relevant” if they had an effect size of 0.50 or more

Recently a Danish team affiliated with the pharma company Lundbeck discovered an entirely new way to get into fights about this. I found their paper, "Determining maximal achievable effect sizes of antidepressant therapies in placebo-controlled trials"", more enlightening than most other writing on this issue. They ask: what if the skeptics’ preferred effect size number is impossible to reach?

The Danes simulate several different hypothetical medications. The one I find most interesting is a medication that completely cures some fraction of the people who take it. (intervention roulette)

Only D, E, and F pass NICE’s 0.50 threshold. And only F passes Kirsch’s higher 0.875 threshold. So a drug that completely cured 40% of people who took it would be “clinically insignificant” for NICE

Clearly this use of “clinically insignificant” doesn’t match our intuitive standards of “meh, doesn’t matter”.

What’s gone wrong here? The authors point to three problems.

First, most people in depression trials respond very well to the placebo effect.

Second, this improvement in the placebo group is inconsistent; a lot do recover completely, but others don’t recover at all. That means there’s a large standard deviation in the placebo group

Third, many patients (often about 30%) leave the study partway through for side effects or other reasons.

Some of our favorite medications, including statins, anticholinergics, and bisphosphonates, don’t reach the 0.50 level.

This doesn’t even include some of my favorites. Zolpidem (“Ambien”) has effect size around 0.39 for getting you to sleep faster. Ibuprofen (“Advil”, “Motrin”) has effect sizes between from about 0.20 (for surgical pain) to 0.42 (for arthritis). All of these are around the 0.30 effect size of antidepressants