The Limits of Nudges and the Role of Experiments in Applied Behavioral Economics

In a recent article in Nature, Evidence from a statewide vaccination RCT, authors found that eight different nudges previously shown to be effective for encouraging flu and COVID vaccination failed to show impact when tested on more reluctant populations. I think a knee jerk reaction is that maybe nudges aren't effective ways to increase vaccination after all. But that completely misses a very important aspect of nudges. Nudges work in most cases because humans can be sensitive to context. And applied behavioral design processes work to understand this context and test the impact of interventions to know if they are effective in a given context. At the highest level I think this paper is less about the ineffectiveness of nudges per say and more about the important role of changing context on behavior. 

I'd like to try to unpack more by focusing on the following: 

• The importance of testing. You can’t blindly chuck nudges over the fence at your customers and simply assume they will be effective just because they worked in prior published studies or in other businesses. In this paper they tested 8 different nudges. If they had just scaled any or all of these without testing we would not have learned anything about effectiveness or the other lessons that follow relating to why they may not have worked. And vice versa - just because something failed to replicate in one context doesn't invalidate prior work or imply it won't work in yours. We just know findings are not generalizable across all contexts. The only way to really know about your context is to test. That is part of the value of business experiments.

• Context matters. In the paper they discussed important differences in context between late stage COVID vaccination and vaccination earlier in the pandemic, as well as differences between flu and COVID.

• The utility of behavioral personas and behavioral mapping to guide our thinking about why a given nudge may work or not. To take context a bit further, authors discussed differences in populations (age) and different challenges to flu vs COVID vaccinations and the differential impact related to how both logistical and psychological barriers may have been addressed in different populations and different contexts with different designs. All of these are things that we can point to or think about in the framework of behavioral mapping. Other issues related to 1) different kinds of hesitancy and changing norms over time, 2) whether some participants may have already been vaccinated (and not mentioned perhaps how prior infection may have changed the sense of effectiveness or urgency). These things may relate more to the kinds of personas that any given nudge may speak to. Although the paper doesn't discuss behavioral mapping or developing personas their utility here seems palpable.

• Behavioral design frameworks. Additionally, authors discussed the impact of things like message saturation and novelty effects in addition to timing. These are things that I tend to think about in the context of Stephen Wendel’s CREATE action funnel as a design framework that speaks to issues like the importance of Cue and Timing. (Actually every aspect of CREATE speaks to almost all of the aspects of this messaging in some way).

• The importance of operationalizing applied behavioral science through repeatable iterative cycles of learning. Even if one constructed behavioral maps and personas in the design of these nudges, the findings in this paper (and in many instances where we leverage experiments to test impact) dictate that we go back and revise our maps and personas based on learnings like these.

There has also been some recent discussion about the failure of nudges because they focus too much on individual behavioral (i-frame) vs. larger systemic issues  (s-frame). It seems to me that best practices in the 'diagnosis' phase of behavioral design process would be helpful in both of these areas if the behavioral lens is widened to include deeper thinking about the broader system (s-frame). As discussed in The Consitution of Knowledge: A Defence of Truth Jonathan Rouch discusses the challenges of changing behavior when beliefs and identity become tightly braided together. Sometimes people first have to be moved to a 'persuadable place emotionally' and their 'personal opinions, political identities, and peer group norms' have to be 'nudged and cajoled simultaneously, which is a long slow process.' To quote Jim Manzi, you can't test your way out of a bad strategy. It does not mean that we should give up on leveraging applied behavioral science to make a positive change in society, but it does make understanding of the larger ecosystem in the implementation of nudges all the more critical. 

As discussed in a recent article in The Behavioral Scientist:

"Our efforts at this stage will determine whether the field matures in a systematic and stable manner, or grows wildly and erratically. Unless we take stock of the science, the practice, and the mechanisms that we can put into place to align the two, we will run the danger of the promise of behavioral science being an illusion for many—not because the science itself was faulty, but because we did not successfully develop a science for using the science." 

The authors follow with 6 guidelines echoing some of the above sentiments above that are well worth reading. 

Reference: 

Rabb, N., Swindal, M., Glick, D. et al. Evidence from a statewide vaccination RCT shows the limits of nudges. Nature 604, E1–E7 (2022). https://doi.org/10.1038/s41586-022-04526-2

Chater, Nick and Loewenstein, George F., The i-Frame and the s-Frame: How Focusing on Individual-Level Solutions Has Led Behavioral Public Policy Astray (March 1, 2022). Available at SSRN: https://ssrn.com/abstract=4046264 or http://dx.doi.org/10.2139/ssrn.4046264