24  Alternative trial design processes

Beyond the set of steps for randomised trials that we have just explored, there are many other conceptions of the steps in a randomised controlled trial.

24.1 Test, Lean, Adapt

One of the best known in applied behavioural science is described by Haynes et al. (2012) in Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials. This paper is written in relation to public policy problems, but the process can be applied equally to business problems. They describe nine steps:

24.1.1 Test

Step 1: Identify two or more interventions to compare. The purpose of a trial is to differentiate between different interventions when we do not know which is most effective.

Step 2: Define the outcome that the intervention is intended to influence. We will discuss this on the next page.

Step 3: Decide on the randomisation unit.

Step 4: Determine how many units are required for robust results.

Step 5: Assign each unit to one of the interventions using a robustly random method. By “robust”, we need to protect against bias creeping into the trial. We cannot allow those with vested interests to lean on which units are allocated to which treatments, or which units might be excluded altogether outside of those identified to be excluded as part of the original plan.

Step 6: Introduce the interventions to the assigned groups. This may sound easy, but monitoring is required to ensure that the interventions are being introduced in the way intended to the right units. There also needs to be a consistency of treatment between the trial and what would be done at scale. For example, is a test of gold-plated interventions delivered by enthusiastic, recently trained practitioners going to give a fair measure of the effectiveness of an intervention delivered month-after-month by those not involved in the experiment? If specific effort is applied during the trial but cannot be applied when scaled, the trial is likely to be a poor guide as to effectiveness when scaled.

24.1.2 Learn

Step 7: Measure the results and determine the impact of the interventions. Your trial may show that none of the interventions are successful, but that’s a success. You have learnt something. The method of measurement should have been decided before randomisation and captured in the pre-analysis plan. We will cover pre-analysis plans in week 4.

24.1.3 Adapt

Step 8: Adapt your policy intervention to reflect your findings.

Step 9: Return to step 1. This process worth is iterative. The trial gives you information that you may be able to implement at scale to improve outcomes, but it does not provide the perfect solution. You can continue to iterate toward better outcomes. Further, for many interventions, what is most effective may also change over time.

24.2 Other processes

A selection of processes in around the development of a trial include:

ideas42

ideas42’s behavioural design process describes a five-stage process:

  • Define the problem and try to remove any embedded assumptions about why it may be occurring.
  • Diagnose what behavioral bottlenecks may be driving the problem.
  • Design interventions that directly address the key bottlenecks we’ve diagnosed.
  • Test the intervention to see whether the design successfully addresses the problem, typically using a randomised controlled trial.
  • Scale the solution to a larger population or adapt it to other contexts if an intervention proves effective.

BETA 4D

BETA’s 4D framework, a four stage process:

  • Discover: Define the policy, program or service delivery issue and develop a behavioural problem statement.
  • Diagnose: Understand the current behaviour and its drivers and develop a clearly defined hypothesis of behaviour change.
  • Design: Design an intervention to address the behavioural problem. Design an evaluation to test the intervention.
  • Deliver: Implement the intervention and evaluation and share the results.

BETA has also published a guidance note on Developing behavioural interventions for randomised controlled trials: Nine guiding questions, which provides more detail on undertaking the first two stages of the 4D framework.