25 Scaling
Scaling is the process of expanding a successfully trialled intervention to the broader population of interest.
In his book The Voltage Effect, John List describes five barriers to scaling. These are:
1. False positives: The inference problem
A false positive is an erroneous sign that success will continue (or even increase) as your enterprise grows. … [I]nitial feedback might simply be a false positive. Think of a false fire alarm or a false signal that you have COVID after a personal test. … One simple way to weed out false positives is to replicate ideas that show early promise.
2. Representativeness of the population
No product is designed for everyone, everywhere, at all times. Knowing who your idea will work for, where, and when will help you figure out how far it can go. … Make sure your test groups at smaller scales reflect the larger population you’re aiming to reach.
3. Representativeness of the situation
If the driver is something unique or specific to a time and place, it may not be easily scalable. … [H]iring 30 excellent teachers might be easy, but hiring 30,000 excellent teachers is a whole different kettle of fish.
4. Unintended consequences and negative spillovers
When designing your idea early on, you must anticipate unintended consequences and negative spillovers and look for ways to engineer positive spillovers. For instance, in our preschool, we discovered that children who weren’t in our program were nonetheless performing significantly better. It turned out that just from playing with the students who were in the program aided their development—a wonderfully scalable spillover.
5. The supply side of scaling
[D]etermine how much the trajectory of costs change as you scale your idea, and if you or the market can bear this change. To ensure you don’t fall into the voltage drop of running out of money, you must account for two types of costs: 1) upfront fixed costs, like the one-time investment for the research and development to create a new product; and 2) your ongoing operating expenses.
25.1 Barriers to adoption
Other evidence on scaling comes from DallaVigna, Kim and Linos (2022), who examined 73 randomised controlled trials run across 30 cities in the United States. They wrote:
Compared to most contexts, the barriers to adoption are low. Yet, city departments adopt a nudge treatment in follow-on communication in 27% of cases. As potential determinants of adoption we consider (i) the strength of the evidence, as determined by the RCT itself, (ii) features of the organization, such as “organizational capacity” of the city and whether the city staff member working on the RCT has been retained, and (iii) the experimental design, such as whether the RCT was implemented as part of pre-existing communication. We find (i) a limited impact of strength of the evidence and (ii) some impact of city features, especially the retention of the original staff member. By far, the largest predictor of adoption is (iii) whether the communication was pre-existing, as opposed to a new communication. We consider two main interpretations of this finding: organizational inertia, in that changes to pre-existing communications are more naturally folded into year-to-year city processes, and costs, since new communications may require additional funding. We find the same pattern for electronic communications, with zero marginal costs, supporting the organizational inertia explanation. The pattern of results differs from the predictions of both experts and practitioners, who over-estimate the extent of evidence-based adoption. Our results underline the importance of considering the barriers to evidence adoption, beginning at the stage of experimental design and continuing after the RCT completion.
For more information, see DellaVigna et al. (2024), List (2022a), and List (2022b).