1 Why experiment?
For several decades, adults with severe head injury were treated using steroid injections. This made perfect sense in principle: steroids reduce swelling, and it was believed that swelling inside the skull killed people with head injuries, crushing their brain. However, these assumptions were not subject to proper tests for some time.
Then, a decade ago, this assumption was tested in a randomised trial. The study was controversial, and many opposed it, because they thought they already knew that steroids were effective. In fact, when the results were published in 2005, they showed people receiving steroid injections were more likely to die: this routine treatment had been killing people, and in large numbers, because head injuries are so common. These results were so extreme that the trial had to be stopped early, to avoid any additional harm being caused.
This is a particularly dramatic example of why fair tests of new and existing interventions are important: without them, we can inflict harm unintentionally, without ever knowing it: and when new interventions become common practice without good evidence, then there can be resistance to testing them in the future.
Haynes et al. (2012) Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials
Understanding what works is difficult. The world is complicated. Logic and theory can provide insight, but they do not always lead us to the right answer.
Further, we often believe that we want to believe. As Richard Feynman said, “The first principle is that you must not fool yourself – and you are the easiest person to fool.”
One approach to understanding the world might be to develop a model (a description of the relationships between the variables you are interested in) and then gather data to test the model. However, it is challenging to make causal statements without a counterfactual as to what would otherwise occur. Would those with head injuries have recovered if they had not been given steroids? There are a variety of approaches to making cause-effect statements in those circumstances (the subject of the other unit this session, Principles of Causal Inference), but these all rest on assumptions of varying robustness.
One major problem is what Jim Manzi (2012) calls “causal density”. Study the movement of a planet, and you can assume a single causal factor, gravity. If you examine a new way of disclosing information about your credit card to customers, there are so many possible causes of behaviour that, no matter how sophisticated your tools and models, there is always the possibility of some uncontrolled factor causing what you observe. The high causal density makes inferring causation nearly impossible.
Experiments provide a more direct way of creating a counterfactual and untangling the causally dense environment by constructing a control group against which to compare outcomes. This provides a foundation for us to move beyond mere statements about correlation and to make causal statements.
Importantly, experiments don’t provide a definitive answer. But even where there is still room for experts to disagree, the room for reasonable disagreement is often narrowed. They can protect us from being fooled into believing what we want to believe.