|Realist Activity Theory for Digital Library Evaluation:
Conceptual Framework and Case Study
Mark A. Spasser1
Jewish Hospital College of Nursing and Allied Health
306 S. Kingshighway Blvd.
St. Louis, MO 63110-1090
In this article, I use the term digital library (DL) in a very relaxed and inclusive manner, borrowing Elliott and Kling’s (1997) definition of digital libraries as “information systems (IS) and services that provide electronic documents — text files, digital sound, digital video — available in dynamic or archival repositories” (p. 1023). The Flora of North America (FNA) project entails the construction of a digital library, in terms both of this definition and of Sanchez’s (1998) definition of a digital library, as “… a virtual space in which scholars conduct research, collaborate and publish their work” (p. 1). Both definitions are consonant with the conception of digital libraries as, in some way, integrating technology, content, and services (Bishop & Star, 1996), and all three can be underwritten by a realist understanding of complex (i.e., structured and thus irreducible, and relatively enduring) 'things,' as interweaving peoples’ situated choices and the collective resources on offer.
Moreover, a critical and heretofore unexamined facet of DL design and use is how library content is assembled and vetted, which in turn has profound implications for subsequent DL usefulness and usability. This article presents a social realist evaluation framework that is then applied to a study and review of the FNA DL and of its context of development and use — specifically, of the organizational issues (or, contradictions) that made its construction and use problematic. Consequently, a significant part of the present analysis focuses on the publication subsystem of the FNA DL — Collaborative Publishing Services — and on how understanding problems related to its design and use facilitates our ability to explain FNA not only as a functioning DL project, but as an organizational form in contradiction-driven expansive development (more will be said both about the role of contradictions and about expansive development later in the article).
THE FLORA OF NORTH AMERICA PROJECT
Flora of North America (FNA) is a project undertaken by the community of North American botanists to provide a wide range of users — including scientific/academic entities, government agencies, private industry, and amateur enthusiasts — with authoritative information on the names, relationships, characteristics, and distributions of all plants that grow outside of cultivation in North America, north of Mexico. The FNA project is gathering and making accessible, in a variety of media, scientifically authoritative and current information on the names, characteristics, relationships, and distributions of the approximately 20,700 species of vascular plants and bryophytes needed for decision-making, resource management, and innovative research. Thus, FNA is intended to serve as a means of identifying plants of the region, as a means of delineating taxa and geographic areas in need of additional study, and as a systematic conspectus of the North American flora. The Project first received funding in 1987 and is expected to be completed around 2009 (Magill et al., 1999; Morin, Wheatstone, Wilkin, & Tomlinson, 1989; Morin, 1991).
The major product being produced by the Flora of North America is a comprehensive set of taxonomic treatments for all North American plant species and infraspecific taxa. These treatments are published electronically on the Web and as 30 printed volumes by Oxford University Press. FNA treatments include accepted names, synonyms, bibliographic references, keys for identification, descriptions, economic uses, conservation status, weed status, and distribution data. The nomenclatural and taxonomic data compiled for FNA form the backbone of the Project and constitute a primary information product that will soon be made public. This list of plant names will become an entry point or portal into FNA’s electronic information and other related botanical information. Despite their synoptic format, many of the treatments present, for the first time, knowledge from a systematist’s lifetime of study (Schnase et al., 1997). In Figure 1 are the description and key sections of the treatment for Berberis Linnaeus:
3. BERBERIS Linnaeus, Sp. Pl. 1: 330. 1753; Gen. Pl. ed. 5, 153. 1754 - Barberry,
Oregon-grape, berbéris, algerita [Mediaeval Latin barbaris]
Mahonia Nuttall, name conserved; Odostemon Rafinesque
Shrubs or subshrubs, evergreen or deciduous, 0.1-4.5(-8) m, glabrous or with tomentose
stems. Rhizomes present or absent, short or long, not nodose. Stems branched or
unbranched, monomorphic or dimorphic, i.e., all elongate or with elongate primary stems
and short axillary spur shoots. Leaves alternate, sometimes leaves of elongate shoots
reduced to spines and foliage leaves borne only on short shoots; foliage leaves simple or
1-odd-pinnately compound; petioles usually present. Simple leaves: blade narrowly
elliptic, oblanceolate, or obovate, 1.2-7.5 cm. Compound leaves: rachis, when present, with
or without swollen articulations; leaflet blades lanceolate to orbiculate, margins entire,
toothed, spinose, or spinose-lobed; venation pinnate or leaflets 3-6-veined from base.
Inflorescences terminal, usually racemes, rarely umbels or flowers solitary. Flowers
3-merous, 3-8 mm; bracteoles caducous, 3, scalelike; sepals falling immediately after
anthesis, 6, yellow; petals 6, yellow, nectariferous; stamens 6; anthers dehiscing by valves;
pollen exine punctate; ovary symmetrically club-shaped; placentation subbasal; style
central. Fruits berries, spheric to cylindric-ovoid or ellipsoid, usually juicy, sometimes dry,
at maturity. Seeds 1-10, tan to red-brown or black; aril absent. x = 14.
Species ca. 500 (22 in the flora): almost worldwide.
Many species of Berberis are grown as ornamental shrubs. Some species harbor the black
stem-rust of wheat (Puccinia graminis Persoon); the sale or transport of susceptible or
untested species is illegal in the United States and Canada. Data on susceptibility of Berberis
spp. to infection by Puccinia graminis was supplied by Dr. D. L. Long, U.S. Department of
Agriculture (pers. comm.).
The berries of many species are edible and frequently are used for jam and jelly.
The genus Berberis as recognized below is divided into two genera, Berberis and Mahonia, by
some authors (e.g., L. Abrams 1934). Species 1-5 below represent Berberis in the narrow sense
(characterized by dimorphic stems, with elongate primary stems and short axillary shoots;
leaves of primary stems modified as spines; foliage leaves simple; and inflorescences
usually rather lax, with acuminate bracteoles and 1-20 flowers; most species susceptible to
Puccinia). Species 13-22 represent the segregate genus Mahonia (with stems never regularly
dimorphic; stem spines absent; leaves pinnately compound; and inflorescences dense, with
rounded or obtuse [rarely acute] bracteoles and 25-70 flowers; never susceptible to
Puccinia). Species 6-12, traditionally included in Mahonia when that genus is recognized
(L. Abrams 1934), are actually intermediate, resembling Berberis proper in their dimorphic
stems, inflorescence structure, and susceptibility to Puccinia, and Mahonia in their spineless
stems and compound leaves. Species showing different combinations of the characteristics
of the two groups are found in other parts of the world (J. W. McCain and J. F. Hennen 1982;
R. V. Moran 1982), so these segregate genera do not seem to be natural. Mahonia is often
recognized in horticultural works, but it is seldom recognized by botanists.
Abrams, L. 1934. The mahonias of the Pacific states. Phytologia 1: 89-94. McCain, J. W. and
J. F. Hennen. 1982. Is the taxonomy of Berberis and Mahonia (Berberidaceae) supported by
their rust pathogens Cumminsiella santa sp. nov. and other Cumminsiella species (Uredinales)?
Syst. Bot. 7: 48-59. Moran, R. V. 1982. Berberis claireae, a new species from Baja California; and
why not Mahonia. Phytologia 52: 221-226.
1. Stems spiny; leaves simple; plants deciduous or evergreen.
2. Plants evergreen; leaf blades thick and rigid, each margin with 2-4 teeth or shallow
lobes, each tooth or lobe 1-3 mm, tipped with spine 1.2-1.6 × 0.2-0.3 mm; stems
tomentose. ..... 5. Berberis darwinii
2. Plants deciduous; leaf blades thin and flexible, margins entire or each with 3-30
teeth, each tooth 0-1 mm, tipped with bristle 0.2-1.4 × 0.1-0.2 mm; stems glabrous.
3. Inflorescences of solitary flowers or umbellate; margins of leaf blade entire.
..... 4. Berberis thunbergii
3. Inflorescences racemose; margins of leaf blade entire or toothed.
4. Bark of 2d-year branches gray; each margin of leaf blade with (8-)16-30
teeth; racemes 10-20-flowered. ..... 3. Berberis vulgaris
4. Bark of 2d-year branches brown, purple, or reddish; leaf blade entire or
each margin with 3-12 teeth; racemes 3-15-flowered.
5. Leaf blade oblanceolate or sometimes narrowly elliptic, apex
rounded or rounded-obtuse; surfaces adaxially ± glaucous. ..... 1.
5. Leaf blade narrowly elliptic, apex acute to obtuse or rounded;
surfaces adaxially not glaucous, often shiny. ..... 2. Berberis fendleri
1. Stems not spiny; leaves compound; plants evergreen.
6. Racemes loose (rather dense in B. harrisoniana), 1-11-flowered; bracteoles acuminate.
7. All leaves 3-foliolate; terminal leaflet sessile.
8. Terminal leaflet blade 0.9-2 cm wide; berries red. ..... 6. Berberis
8. Terminal leaflet blade 2.2-3.2 cm wide; berries blue-black. ..... 7. Berberis
7. Leaves 5-11-foliolate (sometimes a minority of leaves 3-foliolate); terminal
leaflet stalked on most or all leaves.
9. Marginal spines of leaflet blade 0.4-1.2 × 0.1-0.15 mm.
10. Bracteoles (at least proximal ones) leathery, spine-tipped; berries
white or red, somewhat glaucous, 9-16 mm, usually hollow; c Texas.
..... 11. Berberis swaseyi
10. Bractoles usually membranous, seldom spine-tipped; berries
yellowish red to red, not glaucous, 5-6 mm, solid; s California. ..... 12.
9. Marginal spines of leaflet blade 0.8-3 × 0.2-0.3 mm.
11. Berries dry, inflated, 12-18 mm. ..... 8. Berberis fremontii
11. Berries juicy, solid, 5-8 mm.
12. Blade of terminal leaflet mostly 2-5 times as long as wide;
berries purplish red. ..... 9. Berberis haematocarpa
12. Blade of terminal leaflet mostly 1-2.5 times as long as wide;
berries yellowish red. ..... 10. Berberis higginsiae
6. Racemes dense, 25-70-flowered; bracteoles obtuse or acute.
13. Bud scales persistent, 11-44 mm; leaflet blades 4-6-veined from base; anther
14. Shrubs 0.1-0.8(-2) m; teeth 6-13 per blade margin, 1-2(-3) mm, spines
0.1-0.2 mm thick; native, Pacific Coast states, B.C., and Idaho. ..... 21.
14. Shrubs 1-2 m; teeth 2-7 per blade margin, 3-8 mm, spines 0.3-0.6 mm
thick; locally naturalized, se United States. ..... 22. Berberis bealei
13. Bud scales 2-8(-14) mm, deciduous; leaflet blades 1-3-veined from base
(sometimes 1-5-veined in B. amplectens); distal end of each anther filament with
pair of recurved teeth (status of this character in B. amplectens unknown).
15. Leaflet blades abaxially smooth and somewhat shiny (outer surface of
cells of abaxial epidermis of leaf plane).
16. Blade of terminal leaflet 1.3-1.9 times as long as wide; lateral
leaflet blades elliptic to ovate or broadly lanceolate. ..... 20. Berberis
16. Blade of terminal leaflet 1.7-2.5 times as long as wide; lateral
leaflet blades lance-ovate or lance-elliptic. ..... 19. Berberis aquifolium
15. Leaflet blades abaxially papillose and very dull (outer surface of cells
of abaxial epidermis of leaf strongly bulging).
17. Leaflet blades thin and flexible; teeth 6-24 per blade margin,
0.1-0.25 mm thick; plants 0.02-0.2(-0.6) m. ..... 18. Berberis repens
17. Leaflet blades thick and rigid; teeth 2-15 per blade margin, 0.2-0.6
mm thick; plants 0.3-2 m (0.1-0.4 m in B. pumila).
18. Leaflet blades adaxially glossy.
19. Teeth 6-12 per blade margin; n California and Oregon.
..... 17. Berberis piperiana
19. Teeth 3-5 per blade margin; Arizona and New Mexico.
..... 16. Berberis wilcoxii
18. Leaflet blades adaxially dull, ± glaucous.
20. Blade margins strongly crispate, each margin with 3-8
teeth. ..... 13. Berberis dictyota
20. Blade margins plane to undulate or, if crispate, each
margin with 9-15 teeth.
21. Plants 0.2-1.2 m; each blade margin with 9-15
teeth. ..... 14. Berberis amplectens
21. Plants 0.1-0.4 m; each blade margin with 2-10
teeth. ..... 15. Berberis pumila
Copyright (c) 1996 Flora of North America
Figure 1. The description and key of the FNA treatment Berberis Linnaeus.
While all treatments include geo-referenced distribution maps and many include illustrations, the present study refers only to the textual portions of treatment production to simplify and focus the exposition.
FNA then, is a compilation of the best knowledge available on the patterns of biodiversity among plants in the continental US and Canada. In terms of the number of participants and their geographic distribution, FNA is one of the country’s largest scientific collaborations. In sum, FNA provides the only comprehensive, scientifically authoritative treatment of all the plants of North America throughout their range. FNA thus provides a unified conspectus of the flora, is an essential tool for plant identification, and provides systematic discussions of foundational problems and promising new research and exploration.
The FNA Publishing Process: Organization, Medium, and Problems
Authors are invited by the FNA Editorial Committee2 to prepare treatments describing various taxa; and collections of taxonomic treatments, including distribution maps and illustrations, are then reviewed, databased, and assembled into published volumes. The FNA Editorial Committee is responsible for identifying experts, soliciting their participation, and managing the various review processes.
The Project’s daily activities have been coordinated at the Missouri Botanical Garden in the FNA Organizational Center. A mix of mostly paper and some electronic documents are used throughout. A total of five distinct review processes (taxonomic, regional, nomenclatural, bibliographic, and technical) that range from review of scientific content and style to evaluation of a taxon’s conservation status, are performed (sometimes repeatedly) on each treatment once it is submitted. In fact, there are approximately 100 discrete events associated with the publication of a single manuscript, and each event must be tracked and coordinated with other events, occurring serially and concurrently (see Figure 2).
Figure 2. Manuscript flowchart for the FNA project showing steps involved in the edit and review process for each manuscript (adapted from Schnase et al., 1997).
In Figure 2, boxes labeled technical editor, map illustrator, illustrator, prep for OUP and for WWW, project coordinator, and artist all signal Organizational Center activities. Boxes labeled Taxon Editor and Technical Editor, for example, refer to the same person/role, but at different stages of the manuscript vetting process, demonstrating that the same person, for example the Technical Editor in the Organizational Center, handles the manuscript multiple times. Finally, boxes to each side of the center vertical axis illustrate that steps occur concurrently as well as sequentially. For example, both the Taxon Editor and Technical Editor would see the manuscript at least four different times, while various reviewers would be reviewing the manuscript. Excluded from the figure are some of the author's functions as well as multiple back-and-forth exchanges between roles/people.
As many as 300 manuscripts can go into a single volume, and it is necessary to coordinate progress across several volumes simultaneously. This means that participants in the FNA project must effectively articulate a vast number of intra-document, inter-document/intra-volume, and inter-volume interdependent relationships among activities and participants.
Recently, the FNA project has undergone extensive reorganization. A major strategy has evolved to establish semi-autonomous editorial and service centers, the latter providing and coordinating various service activities (bibliographic, nomenclatural, artistic, GIS mapping, and portal), making it possible for editorial teams to work independently and in parallel on major taxonomic groups through the entire publication process, from author solicitation, through edit and review, to electronic and print publication (see Figure 3).
Figure 3. Overview of the major work processes in the newly organized FNA project. Authors (A1- An) prepare treatments and submit manuscripts (M1 - Mn) that eventually make their way through the appropriate editorial center (each editorial center supervising the publication of multiple, or Mn, manuscripts). Arrows depict major paths of information flow between the treatment preparation and database and publishing components within the Project’s editorial centers (EC1-n) (Adapted from Schnase et al., 1997).
The Project’s organization can now be depicted as semi-autonomous editorial centers directly communicating and coordinating their work with each other as needed. This distributed work arrangement replaces the centralized arrangement that previously prevailed.
In sum, until recently the FNA project has represented an attempt to adapt traditional methods of small-scale print publishing to a large-scale databasing and electronic publishing effort. While the Project has moved forward (the first three volumes have been published, and intensive work is well under way on volumes 22-24 and volume 4), with its 800+ participants scattered across North America involved in a decades-long effort and with hundreds of manuscripts in various stages of review by different sets of participants at any one time, traditional publishing methods have proved inadequate and inefficient. The present research constitutes an initial attempt to explain the expansive reorganization of the FNA project and the role of Collaborative Publishing Services in its collective reconstruction.
Collaborative Publishing Services (CPS)
In terms of hope and vision, FNA is a big project — a densely stratified work space comprising many collaborators with differing, and sometimes conflicting, agendas distributed across heterogeneous and (semi) autonomous information infrastructures. Because of the huge scope of the Project and the staggering number of inter-task and interpersonal dependencies that must be articulated in publishing thousands of manuscripts contributed by several hundred geographically dispersed scientists, traditional methods of small-scale print publishing have not scaled. Consequently, it was decided that new tools to enable new work processes must be developed if FNA was not to fall even further behind its planned publication schedule (See Spasser, 2000; Tomlinson, Spasser, & Schnase; and Tomlinson et al., 1998 for research on CPS development- and evaluation-in-use).
CPS is one of a new generation of web-based coordination environments that is being designed both to help reduce the complexity of database publishing in cases, such as FNA, where complexity arises from the inefficiencies in the publishing practices themselves, and to improve the speed and quality of global scientific information gathering and the community construction of large, collaborative scientific databases. It is relatively lightweight, modular, extensible, and scalable. As a web-based environment, it attempts to integrate communication, information sharing (through creation of a common information object repository), and coordination support features and is accessible by unmodified web browsers across heterogeneous, autonomous, and distributed information technology infrastructures. With regard to the last, CPS helps reduce the cognitive load entailed by Project participation (Tomlinson, Spasser, & Schnase, forthcoming), as well as effectively manage Project-wide collaborative load (Spasser, 1998).
In developing the first version of the web-based project coordination and publishing environment called Collaborative Publishing Services (CPS), FNA management, in consultation with CBI, brought together a multi-institutional team of sociologists and information scientists to help develop a strategy for streamlining — and, in particular, making simultaneous and parallel — the distributed operations of the FNA project. The experience has resulted in a promising new approach to large-scale project coordination. CPS provides a way of managing project information by means of dynamically constructed activity-and-information spaces, or role-based views. Role-based views are derived from the socially constructed roles (such as Project Editor, Taxon Editor, Taxonomic Reviewer, Author, etc.) that already exist within the Project. Through these role-based views, Project participants organize information and perform the tasks required in order for the FNA scientific process to work. The various role-based views are delivered through dynamically constructed, personalized web pages.
CPS accomplishes three things that are significant in terms of streamlining the publishing process:
1. CPS lessens individual cognitive load by delegating information and task organizational duties to the interface. CPS does the organization "behind the scenes," and presents to the user information that is in the right place at the right time and in a form that is more explicit and easier to use. CPS transforms the task the scientist confronts by representing it in such a way that the user can readily see exactly how to perform it. This cognitive offloading results in: "What you see is what you need to do."
2. CPS enhances system performance by enabling massively parallel simultaneous use of a large information space via mapping of permissible views plus suites of operations onto individual and group knowledge resources and capabilities. Distributed cognitive systems like the Flora of North America achieve their computational or information-processing power by superimposing several kinds of representations, or representational structures, on a single framework. In our case, the framework is a single, very large information space.
3. CPS structures not just the information but also the tasks. It simultaneously affords and constrains opportunities for the user to interact with the information. Concentrating information organizational complexity in this manner is a particularly efficient way to facilitate simultaneous and tailored access to, and use of, information over a single, large information space.
CPS V1.1 has been fully operational for the bryophyte component of FNA since September 1997. It was extended to the Poaceae group in mid-1998 and to the remaining vascular plant groups in late 1998 on a trial basis. CPS V1.3 became available Project-wide to support FNA’s distributed centers in early 1999 (For a detailed history of CPS development and early deployment, see Spasser, 1998).
CPS is the primary document management system for FNA and essentially provides a web-based, platform-independent mechanism for storing, retrieving, and tracking treatments as they make their way through the FNA publishing process. CPS allows editors, authors, and reviewers of individual families or groups of families, considered “publishing groups,” to carry out their work independently. Within CPS, Project participants can determine the status of a treatment (i.e., where the treatment is in the FNA process, such as “submitted,” “out for taxonomic review,” or “accepted”) and can download copies of all documents pertinent to a treatment, including versions of the manuscript, reviews, email exchanges between authors and editors, electronic copies of maps and illustrations, etc. Access is determined by a person's role. Editors and editorial center staff, for example, can view all documents. Authors can see only their own files, and reviewers can see only designated copies of the manuscript to review.
The Nature of Social Realist Evaluation
Realist evaluation is a relatively new evaluation paradigm (Henry, Julnes, & Mark, 1998; Pawson & Tilley, 1994, 1995, 1997), positing that outcomes (i.e., outcome patterns or regularities – the things and behaviors that interest us as social scientists) follow from mechanisms (sets of internally-related practices and/or objects) acting in contingently configured contexts. Unpacking the "realist" part of the phrase, we find that realist evaluation is about the real, employs a realist methodology, and has realistic outcomes as its goal. First, evaluation should concern the real, but a reality that is stratified and tensed, involving the interplay between the individual and institution, agency and structure, or the lifeworld and system in contingently configured circumstances over time. Social interaction creates interdependencies (Archer, 1995 refers to these, when reproduced, as situational logics) that develop into real-world customs, rules, and divisions of labor which, in turn, condition — enable and constrain — the interests and opportunities of a given cohort of actors. It is these realities that programs, initiatives, or system implementations seek to change. The key explanatory resource for social realism is not that programs work or that computer-based information systems such as digital libraries are deployed, but that such social forms constitute a spiral of new ideas and transforming social conditions and thus, when successful, introduce the appropriate ideas and opportunities to participants and users in the appropriate cultural and socio-organizational conditions — all else follows from such explanatory propositions.
Second, evaluation should follow a realist methodology, one that is scientific (i.e., systematic and rigorously eclectic) and strives to register the influence of the objective, as well as the situationally emergent. However, scientific evaluation is not method- or measurement-driven, but instead suggests a more extensive role for theory in the formulation of evaluation methodology (i.e., realist methodologies are theory-led). Finally, evaluation needs to be realistic, which means both that it is a form of applied research pursued to inform the practice and work of designers, users, managers, etc. and that it modestly attempts to perfect a particular method of evaluation that will work for a specific class of project in well-contextualized circumstances (i.e., the scope of realistic evaluation is middle-range).
Realistic perspectives have been widely adopted in the human sciences, such as accounting (Manicas, 1993), economics (Lawson, 1989, 1994), education (Henry, Julnes, & Mark, 1998); history (McLennan, 1981), human geography (Sayer, 1985), linguistics (Pateman, 1987), nursing (Ryan & Porter, 1996; Wainwright, 1997), psychoanalysis (Collier, 1981; Will, 1980), psychology (Manicas & Secord, 1983), social psychology (Greenwood, 1994; Harré & Secord, 1972), and sociology (Archer, 1995; Keat & Urry, 1982; Pawson, 1996; Pawson & Tilley, 1997; Stones, 1996). Moreover, realism has been employed by several scholars as a comprehensive philosophy of social science (Bhaskar, 1986, 1989; Layder, 1994; Manicas, 1987; Outhwaite, 1987; Pawson, 1989). Finally, the scientific realist perspective is a dominant approach in philosophy of the natural sciences (Aronson, Harré, & Way, 1995; Bhaskar, 1978; Harré, 1970, 1986)3. Thus, according to Sayer (1992), “… realism is a philosophy of and for the whole of the natural and social sciences [emphasis in original] (p. xi).
Social Realism in Practice: Methodological Considerations4
The strength of evaluation research depends upon the perspicacity of its view of explanation. Because realistic evaluation is at once analytic, stratified, processual, and oriented toward explaining change, it blends into one coherent framework objectivity, intentional agency, and contextual sensitivity. Interventions and systems are always embedded in a range of attitudinal, individual, institutional, and societal processes, and thus observable outcomes are always generated by a range of micro and macro forces that are ineluctably interwoven to produce observed situated activities. Stakeholders’ capacity for choice is always conditioned (i.e., constrained and enabled) by the power and resources of their ‘stakeholding.’ Human activity must be understood in terms of its embeddedness, its location within different (i.e., distinct yet interdependent — non-conflated5) layers of social reality.
Social realist theory, in general, rests on two commonsensical assumptions: that which we observe in the world is real and it is a product of complex and contingent causal6 mechanisms that may not be directly accessible to us. In particular, realistic evaluation seeks to understand for whom and in which circumstances a program works through the study of contextual conditioning.7 Context is, of course, much more than spatial, geographical, and institutional location; it refers, in addition, to the prior set of social rules, norms, values, roles, etc. and their interrelationships that condition information system usefulness and usability. A key act of design and analysis is thus to identify the people and situations for whom the system is useful/usable by drawing on success and failure rates of different subgroups of subjects within and between implementations.
Realistic evaluation is pragmatic and outcome-oriented. Social realists recognize actors as being active agents, who could always do otherwise (but usually have good reasons not to). Thus, realist evaluation is sensitive to these agents’ motives, objectives, and/or goals. Evaluators need to understand what the outcomes of an initiative are and how they are produced. Programs or systems cannot be understood as undifferentiated wholes, or as ‘things’ because they trigger (or, fire) multiple mechanisms having different effects on different subjects in different situations, and so produce multiple situated outcomes. Outcomes are not inspected simply to verify whether systems are used, but are analyzed to discover if conjectured mechanism/context theories are confirmed and the extent to which they transfactually apply. In effect,
The key is to empty the notions of mechanisms, contexts and outcomes of their architectonic splendour and … to see them as describing local resources, bounded capacities, specific choices, habitual forms of reasoning, which then act as the routine ingredients for hypothesis-making. … What is needed is a method which gets closer to the now-you-see-them-now-you-don’t patterns of social activities. We need a middle-road strategy for theory-construction so that it becomes neither a jumble of ad hoc stories about why particular events are connected, nor a set of critical claims for the ubiquitous (if metaphorical) presence of master mechanisms (Pawson, undated, pp. 15-16).
Purpose(s) of the Study, or Why study FNA?
The purposes of this research are threefold:
To contextualize the use of CPS in terms of the changing organizational form of the FNA Project;
To study the organizational issues that made the adoption of CPS as well as the construction of the FNA DL problematic; and finally
To explain, in social realist terms, the re-organizational pressures faced by the FNA Project and the role of CPS in its organizational reconfiguration.
Thus, the overarching focus of this research is to identify the underlying mechanisms generative of the inter-organizational dysfunction that has been observed.
A Minor Digression on Activity Theory
Being a mechanics of explanation, i.e., supplying only a scaffolding for hypotheses-generation and an ontological chassis for collection of evidence (Pawson, undated), social realism itself makes no assumptions about the content or substance of social reality. Thus, scientific realism is substantively neutral; it ushers in or presumes no specific theory of social life. Accordingly, realism has been conjoined with structuration theory (Pawson, 1996; Pawson & Tilley, 1997), with Habermasian critical theory (Morrow & Brown, 1994), and, sketchily but suggestively, with activity theory (Hjørland, 1997). Building upon the latter’s work, I propose that realism provides a solid foundation for activity theoretic analyses of social life in general and for work and technology in particular, while activity theory provides a conceptually and substantively rich vocabulary for explanatory reasoning about technologically mediated social praxis.
Activity theory8 is a philosophical and cross-disciplinary framework for studying different forms of human practices, and of social life in general, in a multi-level, stratified manner developmentally in time and through space. Activity theorists (see, for example, Engeström, 1987, 1990, 1991) consider the activity (or, activity system) to be the minimally meaningful unit of study. An activity system involves an activity undertaken by a human actor (either individual or collective) motivated towards/by an object (in the sense of an objective), mediated by artifacts/tools, and conditioned by emergent community structural or institutional properties, such as rules, conventions, and roles/divisions of labor.
The basic model of an activity is depicted by Engeström (1987) in Figure 4: