8  Control

A primary requirement of a good experiment is that it tests or controls for alternative hypotheses. If you were to test the success of an intervention to increase savings in December without controlling for the possibility that expenditure tends to increase in the lead up to Christmas, you are going to have a poor experiment.

There are two ways in which we achieve control: directly and indirectly. These are not either/or options. Rather, we tend to use both direct and indirect control together in an experiment.

8.1 Direct experimental control

In a lab experiment, you can directly control many variables. You choose which to keep constant across the experimental participants, such as the rules of the game that they play or their initial endowment. Variables held constant are control variables.

You also choose which variables you will vary. Variables that are changed are called treatment variables. If you want to to test the effect of a text message to some people and not others, that controlled variation is your experimental treatment.

Typically you will test hypotheses or behavioural interventions by changing one variable at a time. You only change variables which are directly relevant to the hypothesis being tested, otherwise holding the environment fixed. This can help to avoid confounds.

In the field, you control fewer variables, although you maintain direct control of the treatment effects.

8.2 Indirect experimental control

Many variables are difficult to control directly, particularly in the field. For our advertising example above, it’s hard to cancel Christmas. You might think that you could compare sales year-on-year, but is this Christmas the same as the last (a salient problem this year!). More subtlety, the customers who will see your intervention may have different propensities to save that those who don’t. You can’t directly control that.

You can measure variables which you think may affect the propensity to save: gender, age, income, etc, and adjust the analysis after the fact. But there will be more factors than you capture, and likely some important ones that you don’t even realise are relevant. These uncontrolled factors will ultimately undermine your experimental conclusions.

There is, however, an indirect way to achieve control: randomisation. By randomly assigning experimental participants to different treatments, we can eliminate differences between the subjects as a cause of differences between treatments.