Draft of November 2004
Earlier versions presented at the workshop on “The Role of Experimental Methods in Economics” at the University of Nottingham, September 13-14, 2003; and at the INEM Conference in Amsterdam, 10-11 August 2004.
Models and experiments, as used in a variety of research areas, interact in many ways, they reciprocally inform and constrain one another. The following observations do not address the issues related to these interactions. They rather focus on the very concepts of model and experiment in relation to one another. The questions are: What are models? What are experiments? The answers are constrained by the strategy to consider them in relation to each other. What do we learn about the concept of model and that of experiment by viewing them from the perspective provided by the other – as well as by their relationship to the real world? A rather weak and unsurprising – and correct - answer would be to suggest they are in many ways similar, there are important positive analogies between them. A stronger and perhaps somewhat surprising answer will be to suggest a closer connection: models are experiments, and experiments are models. In order to ground such a claim, we need to examine the two concepts a bit more closely and to identify two more general background concepts – those of representation and isolation by manipulation.
Models as representations and isolations
For the present purposes, I take models to be representations (of theories or of reality, for example). I take representation to have two major aspects: the representative aspect and the resemblance aspect (see Mäki 2001). Models are representatives of what they represent: they represent by adopting the role of representative. Another way of putting this is to say that models serve as substitute systems of the target systems they represent. They are substitute systems in the sense that one does not directly examine the target systems, one rather focuses on the properties and behaviour of the representatives. The epistemic point of this activity is that the properties of such substitute or surrogate systems are directly examined in order to indirectly acquire information about the properties of the systems they represent. Animal subjects are representatives that are examined with the purpose of learning about human beings. Large systems of mathematical equations are representatives that are examined in order to learn about the Big Bang.
The reasons for employing and examining substitute systems are various, including ethical (using animals in medical research instead of human subjects since the latter would be ethically objectionable), economic (examining the target system directly would be too expensive), and epistemic (the target systems are inaccessible in full since they are too small, too large, too far away in space or time, or too complex). A familiar way of putting one major idea is to say that the systems in the real world are characteristically all too complex to be tractable targets for direct examination, therefore they are represented by much simpler model systems. It is within, and in terms of, such simple representatives that questions about the complex real system can be recast so as to make them tractable and answerable. Theoretical practice in economics is in line with this notion of representation: “let us examine what happens in this model” thereby hoping to gain some insight into the whys and hows of what happens in the real world.
Not just any arbitrary substitute systems will successfully serve the purpose of learning about those target systems of which they are representatives. It is also required that models represent by resembling those real systems in certain respects and to certain degrees. Thanks to the resemblance, the direct study of models may succeed in indirectly providing information about what they represent. The resemblance aspect of representation is far more contested and difficult to deal with than the representative aspect. While medical researchers may often regard animals as adequate representatives of human beings, major problems arise amongst economists and other social scientists as to whether there is sufficient resemblance between animal behavior and human behavior to justify using the former as representatives of the latter. In economics, the main methodological issue for the last two centuries has been whether the resemblance between models and reality has been sufficiently close. It is relatively easy to construct tractable and convenient substitute systems, models as representatives, and to examine them so as to generate definite solutions, but it is somewhat harder to do this while meeting the constraint of resemblance. There is a long tradition of blaming economics for failing to meet the latter requirement and for focusing its efforts on building theoretical model worlds far removed from the real world – and then for conflating those substitute systems with the far more complex real systems.
I want to suggest that these criticisms and controversies play a criterial role in identifying something as a model. Whether something is a representative of what it represents, whether it is a model as representative, is revealed by whether it gives rise to issues of resemblance. Obviously, resemblance issues will not arise, even potentially, in cases where the thing in question is not intended as a representative of some other thing but something unrelated to it, and where the two things are identical or close to identical. Issues of resemblance are dependent on the model character of a system.
Thus models involve a semantic aspect (characterized by the notions of representation and resemblance) and an epistemic aspect (characterized by the aim of acquiring information about the target system by examining a representative substitute system). In addition, models also essentially involve pragmatics, dealing with their functions. Models conceived as representations can be considered four placed: Model M is an entity used by agent A to represent target system S for purpose P. The inclusion of purpose or function suggests pragmatic constraints on the required respects and degrees of resemblance. The desired sort of resemblance is a function of the uses to which models are put, the purposes they are supposed to serve.
These ideas about the general characteristics of models imply that the ontology of models must be very flexible (see Mäki 2001). Almost anything can serve as a model of almost anything else. Models may be made of almost any stuff, as it were. They may be physical objects, diagrams, drawings, verbal statements, systems of mathematical equations, abstract objects, and so forth. With a broad enough conception of the world, we may say that bits of the world (models) are used by other bits of the world (modellers) to represent further bits of the world (modelled). This flexibility is among the preconditions of the models = experiments equation to be argued.
Another precondition of the models = experiments equation is based on recognizing a prominent feature of conventional experiments: they involve manipulation. An experimental design typically suggests ways of manipulating a number of potentially influential variables of a system so as to neutralize them, to prevent them form making an impact on the outcome. In other words, a system of entities is manipulated in order to accomplish effective isolations of a limited set of properties and causal relations from the rest of the world. This is done in order to utilize and enhance the capacity of such experimental systems to serve as epistemically successful substitute systems, as resembling representatives. The equation models = experiments is supposed to hold precisely for such manipulable and manipulated systems. The equations holds for representations which involve the representative and resemblance aspects and which are subject to manipulations that effect isolations.
Experiments are models
Let us first approach the equation by examining the notion of experimental system in the conventional sense, denoting an arrangement of things typically in a laboratory. Such experimental systems, having been first designed and then built, are systems that are not, at the end, examined for their own sake. They are artificially designed and constructed substitute systems that are directly examined in order to indirectly generate information about the real world outside the laboratory – such as economic systems and behaviour “in the wild” (as well as theories about such systems and behaviours). Given the account of models given above, this means they are representatives of real systems: they are material models of aspects of the rest of the world. This is not to say experimental set-ups are unreal other than in the sense of being artificial, man-made. They are bits-in-the-world and at the same time bits-about-the-world. As Plott suggests,
“The relevance of experimental methods rests on the proposition that laboratory markets are ‘real’ markets in the sense that principles of economics apply there as well as elsewhere. Real people pursue real profits within the context of real rules. The simplicity of laboratory markets in comparison with naturally occurring markets must not be confused with questions about their reality as markets.” (Plott 1982, 1520)
In order to construct a substitute system, the experimenter modifies and purifies some relevant chunks of the world. Experimental models isolate fragments of the world by way of controlling for the rest of it – or more accurately: for other potentially causally significant things - by causally manipulating those other things, thereby preventing them from interfering with what happens in those substitute systems.
Using Vernon Smith’s vocabulary, theory (or a theoretical model) depicts three components - environment, institution, and behaviour – and yields results concerning behaviour given the environment and the institution. In testing such a model against field data, the environment and the institution are not controlled, thus we have a composite test of the theory’s assumptions about all three components jointly. Failures and successes in such tests are not conclusive about any of the three components. In an experimental set up, on the other hand, one seeks to control the environment and the institution and proceeds to test assumptions about behaviour (Smith 1989, 154).
The relevant connections between the three components are isolated from possible disturbances by employing various controls. On Smith’s account, four conditions must be met to ascertain that some of the potentially causally most significant things are neutralised (Smith 1982): non-satiation of preferences, saliency or lucidity of the connection between rewards and choices, dominance of payoffs in the laboratory over other possible costs or benefits, and privacy of choices. By way of manipulating the situation so that these conditions hold, relatively effective material isolations are hoped to be forthcoming.
Experiments operate with substitute systems that are manipulated for certain purposes. For most purposes, the issue of resemblance between such material models and what they are representatives of “in the wild” cannot be escaped. Do the conclusions of the examination of the substitute systems also apply to the target system? The resemblance aspect of representation is a particularly pressing issue in experimental economics: do experimental systems resemble real economic systems “in the wild” in sufficient respects and to sufficient degrees, given the purposes of experimentation? This is the issue of “external validity” or “parallelism” (see Guala 1999, 2002; Siakantaris 2000; Loewenstein 1999; Starmer 1999). This implies that experimental systems qualify as models: they meet the criterion of giving rise to the issue of resemblance between the representative and the represented.
Models are experiments
Models are the laboratories of economic theorists. This is a claim most economic theorists will accept, and many of them have explicitly made it. Just as laboratory scientists design and examine the artificial worlds of experimental situations in their laboratories, economic theorists design and examine the artificial worlds of their theoretical models. Lucas draws this analogy explicitly:
“One of the functions of theoretical economics is to provide fully articulated, artificial economic systems that can serve as laboratories in which policies … can be tested out ….” (Lucas 1980, 696)
I am not suggesting that all sorts of models are to be viewed as experiments. But as implied by what I said earlier, there is a species of models that possess the key characteristics of experiments: those based on theoretical isolation (see Mäki 1992). In earlier work, I have adopted the vocabulary of ‘isolation’ using it of processes and products of theoretical research, partly inspired by the analogy with material isolation as in experimentation (and partly also by Marshall’s early use of the term to characterize partial analysis). The expression I have used, the “method of isolation” is intended to generalize, to capture what is shared by conventional experimental isolation and theoretical isolation.
Consider material experimentation as based on causally isolating fragments of the world from the rest of it so as to examine the properties of those fragments free from complications arising from the involvement of the rest of the world. The analogy with theoretical modelling is obvious: while material experimentation employs causally effected controls, theoretical modelling uses assumptions to effect the required controls. Assumptions are used to neutralise, in the model worlds, the involvement of other things by assuming them to be constant, absent, in a normal state, between certain intervals, and so on. Assumptions play a key role in the construction of theoretical models as substitute systems. Such idealizing assumptions, if interpreted as statements about the real world, are characteristically false. Yet they are necessary for effecting the required theoretical isolations. (See also Mäki 2004a, 2004b.)
I have called models based on theoretical isolation “thought experiments” in analogy to “material experiments” (Mäki 1992). This sounds like justified given the strong structural similarities between material experimentation and theoretical modelling. The structure of experimentation, involving manipulation and isolation, is the same, while what is different is the way these controls and isolations are effected: by way of thinking and assuming, and by way of material or causal manipulation.
Given this difference, it is not surprising that theoretical models are capable of effecting isolations more stringently than material models. In theoretical modelling, one simply assumes away all disturbances and complications. As Vernon Smith rightly notes, “the abstractions of the laboratory are orders of magnitude smaller than those of economic theory…” (Smith 1982, 936). In thought experiments, the controls can be made as tight as one wishes, whereas material experiments are forced to leave many of the possible interferences uncontrolled or just weakly controlled (failure in causally effective controls can then be compensated for by way of assumptions). An unsurprising implication of this slack between the two is that the testing of theoretical models in terms of material experiments is bound to be imperfect (but naturally not useless).
The issue of resemblance is the hottest methodological issue in and about theoretical economics. Models and their assumptions are being criticized for being unrealistic and defended as sufficiently realistic or inconsequentially unrealistic. The traditional complaint is that the representatives do not sufficiently resemble what they represent, and that the gap between the two is ignored by treating the substitute systems as if they were the real system. Lucas is aware of these issues and guards against this latter error:
“To serve this function [as a laboratory] well, it is essential that the artificial ‘model’ economy be distinguished as sharply as possible in discussion from actual economies. Insofar as there is confusion between statements of opinion as to the way we believe actual economies would react to particular policies and statements of verifiable fact as to how the model will react, the theory is not being effectively used to help us to see which opinions about the behavior of actual economies are accurate and which are not.” (Lucas 1980, 696)
I have suggested that models = experiments insofar as theoretical models and material experiments share the characteristics of representations that are manipulated in order to effect isolations. In this regard they share the general characteristics of models as well as the general characteristics of experimentation. There are two major differences. One is qualitative: the manipulations in one case are causal, while in the other, they are theoretical. The other is one of degree: theoretical models are capable of providing much tighter isolations than experimental models. This is possible thanks to the qualitative difference: those tighter isolations are easy to implement as they only have to be assumed.
The foregoing has suggested that theoretical models are experiments and that material experiments are models. To justify this thought I did not need much in the way of conceptual preliminaries: just the notion of representation as involving the representative and resemblance aspects, and the observation that experiments characteristically involve isolation by manipulation. The equation models = experiments is not suggested to hold for all specifications of the two concepts, that of model and that of experiments. The equation rather boils down to two more specific claims: many theoretical models = experiments, and many material experiments = models.
As implied already, we can choose to let all this shape our vocabulary. We can talk about two kinds of models: theoretical and material models; about two kinds of experiments: theoretical experiments (or thought experiments) and material experiments; and more generally about two broad classes of representations: theoretical and material representations. When talking about economic experiments, we need to be somewhat more relaxed about the meanings of ‘material’ than when referring to experiments in physics or chemistry.
I have not addressed in any detail the issue of what these theoretical and material representations are – or are intended to be – about: what are they representatives of, and exactly what kinds of realities might they resemble? “Aboutness” is an essential condition for something to qualify as a representation, but this does not yet tell anything about the sorts of thing any given representation is about. This is a crucial issue for debates around “external validity” in experimental economics.
As for the representative aspect, I think it is safe to suggest that both theoretical and experimental models are representatives of the economy “in the wild” – that impure mixture of all sorts of ingredients and influences where the conditions of isolation do not hold (or hold at most partially, approximately, temporarily). But what about the wild world are the models supposed to resemble? In the isolated mini-worlds of theoretical and experimental models, things are shown to happen in an orderly fashion. Is it these orderly happenings that are claimed to resemble what happens in the non-isolated maxi-worlds, implying that the models can be expected to exhibit good predictive performance also in non-isolated circumstances? I do not think this is a very good idea. Another possibility would be to counterfactually suggest that the happenings in the model worlds resemble what would happen in the real world if only those isolations were implemented. Yet another possibility – one that I am attracted to – is to conjecture that, at least for some purposes, the causal powers and mechanisms isolated and identified by the models resemble those that function outside those theoretical and material models (cf. Sugden 2002). Experimentation alone is insufficient for testing such conjectures for their truth and falsity.
Guala, Francesco (1999) “The problem of external validity (or ‘parallelism’) in experimental economics”, Social Science Information, 38, 555-573.
Guala, Francesco (2002) “On the scope of experiments in economics: comments on Siakantaris”, Cambridge Journal of Economics, 26, 261-267.
Loewenstein, George (1999) “Experimental economics from the vantage point of behavioural economics”, Economic Journal, vol 109, no 453, F25-F34.
Lucas, Robert, Jr. (1980) “Methods and problems in business cycle theory”, Journal of Money, Credit and Banking, 12, 696-715.
Mäki, Uskali (1992) "On the method of isolation in economics", Poznan Studies in the Philosophy of the Sciences and the Humanities, 26, 319-354.
Mäki, Uskali (2001) “Models”, in The International Encyclopedia of Social and Behavioral Sciences. Volume 15, pp. 9931-9937. Elsevier 2001.
Mäki, Uskali (2004a) "Theoretical isolation and explanatory progress: Transaction cost economics and the dynamics of dispute", Cambridge Journal of Economics, 28 (3), May 2004, 319-346.
Mäki, Uskali (2004b) "Realism and the nature of theory: A lesson from J.H. von Thünen for economists and geographers", Environment and Planning A, 1719-1736.
Morgan, Mary and Morrison, Margaret, Editors (1999) Models as Mediators. Cambridge: Cambridge University Press.
Morgan, Mary (2002) “Model experiments and models in experiments”, in Model-Based Reasoning: Science, Technology, Values, edited by L. Magnani and N.J. Nersessian. Dordrecht: Kluwer. Pp. 41-58.
Plott, Charles (1982) Industrial organisation theory and experimental economics”, Journal of Economic Literature, 20, 1485-1527.
Siakantaris, Nikos (2000) “Experimental economics under the microscope”, Cambridge Journal of Economics, 24, 267-281.
Smith, Vernon (1982) “Microeconomic systems as an experimental science”, American Economic Review, 72, 923-955.
Smith, Vernon (1989) “Theory, experiment and economics”, Journal of Economic Perspectives, 3, 151-169.
Starmer, Chris (1999) “Experiments in economics (Should we trust the dismal scientists in white coats?)”, Journal of Economic Methodology, 6, 1-30.
Sugden, Robert (2002) “Credible worlds. The status of theoretical models in economics”, in Fact and Fiction in Economics. Models, Realism, and Social Construction., edited by Uskali Mäki. Cambridge: Cambridge University Press. Pp. 107-136
The linguistic practice of economists often does not distinguish between ‘theory’ and ‘model’ but for many purposes it is useful to think of models not only as representing the world but also as representing theories, as their reduced or enlarged representatives. Models in this sense “mediate” between theories and the world or the data (see Morgan and Morrison 1999). My focus here is on models as representations of the world.
The full quote refers to the relatively low costs of theoretical modelling as one of its justifications: “One of the functions of theoretical economics is to provide fully articulated, artificial economic systems that can serve as laboratories in which policies that would be prohibitively expensive to experiment with in actual economies can be tested out at much lower cost.”
See also Morgan (2002) who pursues a similar idea. I became aware of her essay after having prepared this paper.
An obvious qualification is needed. This is because the expression “thought experiment” is often used in a more restricted sense, denoting mental operations that deal with particular cases, whereas this restriction is not a condition for something to qualify as theoretical modelling. A thought experiment in this narrow sense is put in terms of concrete illustrations. This makes the analogy between such concrete thought experiments with ordinary material experiments quite strong for the simple reason that the latter are also conducted in terms of concrete particularities, in particular space-time situations dealing with concrete materials and set-ups. A negative analogy is that the idea of control is not among the key elements in the narrow notion of thought experiment. The notion of thought experiment that I have been employing is not necessarily concrete in such a way, and it essentially involves the element of control. All that is needed for something to qualify as a thought experiment in my sense is that it is a mental operation of representation, isolation and manipulation, employing a theoretical framework, and having the generic structure of experiment.