43 Heterogeneity
Experimental treatments can have different effects on different people.
In most randomised controlled trials, however, there is a focus on average treatment effects. This is partly a legacy of the sample sizes we use. Partition into smaller groups requires a larger total sample size for accurate inference.
There is considerable benefit in considering heterogeneity in experiments. It can provide insight into how the intervention can be targeted to those who will benefit most, how the treatment works, or how the treatment can be improved.
As one illustration, Costa and Kahn (2013) examined how people’s change in energy use in response to a home energy report varied with their political ideology. Their analysis is summarised in the following chart.
Knowledge of the political leanings of the recipients of the energy report could help targeting the reports and influence their design. It might also provide fodder to develop further hypotheses to make them more effective.
43.0.1 Heterogeneity across cultures
One way in which heterogeneity shows itself is across cultures. The artefactual field trial we examined earlier, where Joe Henrich and friends (2001) played the ultimatum game in small scale societies, demonstrates this.
However, there is also a commonality across cultures. In the Many Labs 2 study (Klein et al., 2018), researchers performed replications of various findings in behavioural science across a range of different countries and cultures. A core theme of the result was that the same studies tended to replicate across countries and cultures, or tended to fail everywhere.
One important implication of this is that, contrary to some claims, heterogeneity of study participants is unlikely to be an explanation for the replication crisis.