7 Types of experiments
One major experimental consideration is whether to experiment in the lab or the field.
7.1 Criteria that define a field experiment
Harrison and List (2004) described six criteria that can be used to delineate between lab and field experiments:
The nature of the subject pool: Is the subject pool made up of standard subjects (e.g. students), a more representative subject pool, or a target population? Decision making may vary across pools.
Information the subjects bring to the task: Do participants being with them experience in the commodity or task that forms the basis of the experiment? Decisions may vary with the experience brought to the task.
The nature of the commodity: Does the experiment use actual rather than abstractly defined goods? The artificiality of abstract goods can affect behaviour.
The nature of the task or trading rules applied: Is the task the same that is naturally undertaken in the field? As field experience can enable the development of task specific heuristics, variation in the task can shift decision making.
The nature of the stakes: Are the stakes in commensurate to that in the field? Stakes in the laboratory and field can vary markedly, which might be the difference between indifference and engagement.
The nature of the environment that the subject operates in: Is the environment the one in which the decision naturally occurs? Different settings might engender role playing, or suggest particular strategies or heuristics to use in making decisions.
7.2 A taxonomy of experiments
This criteria can be used to develop a more refined taxonomy of experiments than simply “lab” or “field”, although this is a spectrum rather than a precise taxonomy. Harrison and List developed a spectrum as follows:
Conventional lab experiments: Lab experiments typically use a standard subject pool of students (cheap and convenient), an abstract framing, and an imposed set of rules.
Artefactual field experiments: Artefactual experiments mimic lab experiments, except that they use non-standard subjects such as people from the market or context of interest.
As one example, Michael Haigh and John List got traders from the Chicago Board of Trade to play games to test whether they exhibit myopic loss aversion. (You might recall from 23713 Behavioural economics and corporate decision making that myopic loss aversion was Thaler and Benartzi’s explanation for the equity premium puzzle.) Would professional traders with experience trading make better decisions? Surprisingly, they found that the traders were even more myopically loss averse in a lab environment than the typical student subjects.
Another example is work by Joe Henrich and colleagues, who recruited subjects from 15 small-scale societies to play the ultimatum game. (Recall that the ultimatum game involves a proposer offering a split with a respondent. If the respondent accepts, they keep the split. If they reject, they both get nothing.) These subjects came from three foraging societies, six societies that practice slash-and-burn horticulture, four nomadic herding groups, and three sedentary agriculturalist societies.
Although no group exhibited the optimal game theoretic solution (offer the smallest possible non-zero sum, accept), Henrich and colleagues found that there was material behavioural variability across the groups. No group offered, on average, less than 25%. But two groups had average initial offers over 50%. Rejection rates also varied materially. Henrich and colleagues argued that economic organisation and the degree of market integration explained a substantial portion of this variation.
Framed field experiment: Framed field experiments incorporate elements of the naturally-occurring environment, such as the commodity, tasks, stakes and information sets of the subjects. Subjects understand they are taking part in an experiment and that their behaviour is recorded and scrutinised.
As an example framed field experiment, John List tested expected utility and prospect theory at a sportscard show. He endowed experienced traders with a mug or bar of chocolate and give them an opportunity to trade (as per the famous experiments reported by Daniel Kahneman, Jack Knetsch and Richard Thaler). List found that, among inexperienced consumers, prospect theory adequately explained their behaviour. However, those with intense market experience behaved in accordance with neoclassical predictions.
Natural field experiments: Natural field experiments are similar to framed field experiments except that the environment is one where the subjects naturally undertake these tasks and where the subjects do not know that they are in an experiment.
Natural field experiments are different from “natural experiments”, a subject you will look at in depth in Principles of Causal Inference. Natural field experiments use treatments constructed by the experimenter to test a hypothesis. Natural experiments use naturally created randomness across treatments to draw conclusions from naturally occurring data. The experimenter has less (usually no) control in natural experiments. Field experiments might be seen as providing a bridge between lab experiments and naturally occurring data, giving mixture of realism and control that you can’t achieve otherwise.
You came across a natural field experiment in a previous unit where we saw Marianne Bertrand and Sendhil Mullainathan’s tests of labour market discrimination. They sent manipulated CVs tin response to ads in Boston and Chicago, finding that white names received 50% more callbacks for interviews than African-American soundings names.