(2021-11-03) Kerwin Nothing Scales

Jason Kerwin: Nothing Scales. I recently posted a working paper where we argue that appointments can substitute for financial commitment devices.

Our results are very clear, and we tell a clear story that teaches us something very important about self-control problems in healthcare. Appointments help in part because they are social commitment devices, and—because there are no financial stakes—they don’t have the problem of people losing money when they don’t follow through. The paper also strongly suggests that appointments are a useful tool at encouraging people to utilize preventive healthcare—they increase the HIV testing rate by over 100%.

Maybe we should try appointments as a way to encourage people to get vaccinated for covid, too? Well, maybe not. A new NBER working paper tries something similar for covid-19 vaccinations in the US. Not only does texting people a link to an easy-to-use appointment website not work, neither does anything else that they try, including just paying people $50 to get vaccinated.

The most likely story is that this is a different group of people and their treatment effects are different

the Chang et al. study specifically targets the vaccine hesitant, whereas men in our study mostly wanted to get tested for HIV

The vaccine-hesitant aren’t procrastinating; by and large they just don’t want to get a shot.

But trying to analyze this is very rare, which is a disaster for social science research. Good empirical social science almost always focuses on estimating a causal relationship: what is β in Y = α + βX + ϵ? But these relationships are all over the place: there is no underlying β to be estimated!

Treatment effect heterogeneity also helps explain why the development literature is littered with failed attempts to scale interventions up or run them in different contexts.

to a first approximation, nothing we try in development scales. (social program, economic development)

Why not? Scaling up a program requires running it on new people who may have different treatment effects. And the finding, again and again, is that this is really hard to do well.

we rarely actually run the original intervention at larger scale. Instead, the tendency is to water it down, which can make things significantly less effective

First, we need a better understanding of how to get policymakers to actually implement interventions that work.

Time and again, we have real trouble just replicating actual treatments that work—instead, the scaled-up version almost always is watered down.

Second, every study should report estimates of how much treatment effects vary, and try to link that variation to a model of human behavior.

Moreover, almost all of our studies are underpowered for understanding heterogeneous treatment effects.