Ana səhifə

Grounding in computer-supported collaborative problem solving


Yüklə 1.41 Mb.
səhifə9/18
tarix25.06.2016
ölçüsü1.41 Mb.
1   ...   5   6   7   8   9   10   11   12   ...   18

The whiteboards


Initially, we used a separate groupware system to provide the whiteboard functionality. This was BeingThere for the Macintosh. We also used a quicktime movie recorder for capturing the whiteboard images 1 frame/second, for later analysis. Muddweller was our MOO client of choice the Macintosh platform. Later on(from pair 10), we switched to the TKMOO-lite client, implemented in TCL/TK, and available for Unix/X, PCs, and Macs. This client also included a whiteboard which works through the MOO, using the same kinds of transmissions between client and server as the other communication and action. This also allowed us to dispense with the movie recorder and use the same style of automatic transcription.

Both whiteboard supports elementary drawing: boxes, lines, with different colors, thickness (for BeingThere) and with or without arrows, plus text frames. It does not include free drawing and does enable users to edit displayed text. Users can move, remove, resize or change color to the objects created by their partner. They cannot see each other's cursor. They can copy and paste in the whiteboard. The size of the whiteboard window is 14 X 19 cm (as the MOO window). The MOO and the whiteboard are side by side, they split the screen vertically in two equal area. Both users see the same area, there is almost no scrolling inside the fixed window size. The two detectives are provided with a map of their virtual environment, so that the schema focuses on the inquiry itself instead of on a (trivial) spatial representation of their environment. Globally, these two whiteboards were rudimentary. Several subjects complained about their conviviality, especially for editing objects.





Figure 1: A subset of whiteboard drawings (from Pair 5)
    1. Conditions


  • The subject were provided with two sheets of instruction, one regarding the usage of the MOO and one regarding the task itself. The task instruction sheet included a map of the auberge (cfr Appendix 1).

  • The subjects were invited to work collaboratively (and not competitively), i.e. to agree on the solutions. There were not asked to stay together (in virtual space) during the whole task.

  • The task was programmed in English. All objects names and suspect answers were in English. We wanted to be able to exchange our protocols with a group of colleagues working on computers and collaboration11. The subjects (65% were native French speakers) were invited to interact in English if this was not too hard, they were allowed to talk in French as well, and several pairs did. They were also told that they could ask us about any English word in the game that they would not understand.

  • Pairs 1 and 2 interacted by voice, they were in the same room but could not see each other screen. These two voice experiments aimed to give us a basis for appraising the data collected in the MOO, but not for carrying a systematic comparison of voice versus typed communication. For most other experiments, the subjects came to our building, met briefly before the experiment and then solved the task in two different rooms. In a few pairs, the subject referred to as Sherlock was working in remote condition, somewhere in Geneva (pair 12), Bern (pair 17) and in the USA (pair 18). In pair 21, both subjects were in remote conditions, one in Geneva and one in the USA. In remote conditions, the subjects were provided with the same instructions sheet. We also applied the same rules regarding to the size of the whiteboard, insuring that each subject sees completely his partner whiteboard.

  • From pair 3, subjects have been familiarized with the MOO and the whiteboard through a training task, in which they explore a area of 7 rooms, draw a map of these rooms on the whiteboard, on which they report the objects they have found (and their color). In most cases, the warm-up task was carried out a few days before the experiment itself.

  • The whiteboard BeingThere was used for pairs 1 to 10, after which TkMOOlight was used. In pair 10, the initial whiteboard was not empty but included a table which will be mentioned in the results.

  • The subjects received 30 Swiss Francs to do the two tasks (warm-up task and experiment task).

  • The technical conditions were satisfactory, we encountered no network lag problems, even for pairs working in remote conditions.
    1. Data


We did not include all protocols in this report. We included the final state of the whiteboards in Appendix 3 and one complete protocol in Appendix 2. However, all protocols are available on World Wide Web12. The intermediate states of the whiteboard are also available.
    1. Variables


The variables used to describe interaction patterns are: frequency of interactions, the rate of acknowledgment, the delay in acknowledgment, the co-presence in interactions (whether the two subjects are in the same room), the content of interactions, the modality of interaction (MOO dialogues, whiteboard drawings or MOO action), spatial sensitivity and all categories created by the interaction between these variables (e.g. the ratio inference-in-whiteboard / inferences-in-talk)

The variables used for describing problem solving behaviors are: the navigation (how subjects move in the MOO), the type of questions that subjects ask, various forms of redundancy among questions, when they switch from data collection to data analysis, etc. Most of the variables used in result description are obvious, but some of them need some explication regarding to how we counted them.

We explain now how we computed these variables and specify how we parsed the protocols to count different items. Despite the use of coding rules described below, the coding is not free from subjectivity. Given the size of the corpus (40 hours of interactions), we had not the time or resources necessary to ask a second judge to code the protocols. Our strategy is rather to collect information on the basis of a single judge coding and refine the analysis on the basis of the hypothesis which can emerge from this first exploratory study. Double coding will be carried out next spring on the most interesting items.

Most statistics are based on 18 pairs (excluding pairs 1 and 2, voice interactions), the statistics involving actions on the whiteboard do not include pair 4 for which the movie of whiteboard interactions was lost13. Most quantitative values are presented by pair, since, even if they are sometimes counted individually, most interaction variables make sense when aggregated by pair. Hence, our number of data is reduced from 40 (36) to 20 (18). In the statistical analyses we perform here, we do however perform F tests in all cases, for consistency. We did however systematically made a T test when the number of data was low, but we encountered no case in which the F was significant but not the T.

To associate quantitative and qualitative data, we often use excerpts from the protocols. These are presented in for rows indicating respectively the time, the room where action occurred , the subject who performed the action and what the subject actually typed. Times are indicated in fractions of minutes (3 min. 30 seconds = 3.5 minutes). The subject is indicated by 'H' for Hercule and 'S' for Sherlock. In the rooms 'K' means 'kitchen', 'priv' means 'private residence', and the other rooms are numbered (r1, r2, ...) (see the Auberge map in Appendix 1).

120.2

Priv

H

page sherlock I think indeed that the husband has a motive and actually he has perhaps been to the room when he left the restaurant





Time

Room

Subject

Action or interaction

The examples provided do not correspond exactly to the data in the protocols, since we deleted columns and rows which were not relevant for the example being presented. In some cases, we translated the messages from French to English, but we indicated it.


      1. Space sensitivity in dialogue


The 'space sensitivity' variables evaluates if the subject uses the communication verbs appropriated regarding relative MOO positions. It is computed as the sum of 'say' commands performed when the subjects were in the same room plus the number of 'page' commands when detectives were in different rooms, divided by the total number of messages.
      1. Acknowledgment rate


We computed the rate of acknowledgment, i.e. the ratio between the number of acknowledge interactions and the total number of interactions. We parsed the 20 protocols and associated utterances by pairs [U1 - U2] when U2 can be interpreted as acknowledging U1. Actually, we do not only code acknowledgment through verbal interactions, but also through whiteboard actions and even MOO actions. We hence have pairs [A1 - A2]. We apply the following rules in coding:

  • We do not count self-acknowledgment, i.e. when A1 and A2 are performed by the same subject, because do not indicate the elaboration of mutual knowledge. However, from a cognitive perspective, it would be interesting to study the role of self-acknowledgment14. Some of them really take the form of a dialogue.

  • We do not count failed acknowledgment, i.e. when A2 is not perceived by the speaker who uttered A1, because mutuality is only established if the speaker receives the acknowledgment. Failed acknowledgment is due to typing errors in commands or to spatial problems, e.g. when Sherlock uses 'say' while Hercule is in another room.

  • Some messages seems to acknowledge each other, but when one considers the time stamp, it appears that they have actually been typed simultaneously.

  • When an utterance is acknowledged by two utterances, we count it as one acknowledgment.

  • When several utterances are acknowledged by a single utterance, we consider that each of them as been acknowledged.

  • When we hesitate whether one utterance acknowledged one or another, we choose the best one with respect to content. An error at this level will impact on the computation of acknowledgment delay but not on acknowledgment rates.

  • When a subject types several times the same sentence, we count it as one utterance.

  • On the whiteboard, we counted that when Hercule moves an object drawn by Sherlock, he acknowledges Sherlock's drawing. This is true in some cases, but sometimes moving objects simply aims to reorganize space on the whiteboard.
      1. Content of interactions


The coding of the content is based on 4 categories and different sub-categories. Given the ambiguity between sub-categories, we gathered the data by categories and not by sub-categories, except for the knowledge level. For this category, the quantitative differences are wide enough to by-pass classification errors.

We count here utterances and whiteboard notes, not content units. An utterance or a whiteboard note may convey several facts or inferences. Concerning the whiteboard, this will be especially true for pairs 6 and 7 which put one note per room, each note summarizing the information gathered in that room. For the other pairs, most information on the whiteboard is made of short sentences.



When an utterance of category content X was acknowledged by a message which was neutral with respect to content, such as 'ok', we allocated this 'ok' to the same content category as the acknowledged utterance.

  • The knowledge level

The knowledge level includes all interactions about the knowledge involved in the problem solving process, i.e. the data collected though actions in the MOO and the inferences drawn from these data. We referred to the former as 'facts' and to the latter as 'inferences'. A fact presents the information as it was collected. It often reproduces word by word the answer given by a suspect. An inference involves some interpretation by the subject. The border between these two categories is of course sometimes difficult to draw, as in example 3, in which a simple "but" may turn the fact expressed by Hercule into a counter-argument on Giuzeppe opportunity to commit the murder. Very often, an utterance includes both one or more facts and an inference, the former supporting the latter. In this case, we count the utterance in the inference category.

120.2

Priv

H

page sherlock I think indeed that the husband has a motive and actually he has perhaps been to the room when he left the restaurant




121.8

Priv

H

page sherlock but Giuzeppe said he went to the bar immediately after the restaurant with the loretans

Example 3: Borderline case between facts and inferences (from Pair 19, translated)

  • The management level

The 'management' category includes all interactions describing the evolution of the problem solving process. When we coded the protocols, we distinguished two sub-categories, 'strategy' and 'position'. The 'position' sub-category refers to simple utterances to ask or to tell where the partner or oneself stands in the MOO. The 'strategy' sub-category includes utterances where subject discuss how to proceed: how to collect information (which suspects, which rooms, which questions, ...), what has been collected so far ("anything interesting?"), how to organize collected data, how to prune the set of possible suspects and how to share roles (who does what in the pair). Actually, because of the spatial metaphor used in the MOO, many utterances in this 'strategy' sub-category concern positions in the MOO as well. The difference between the 'position' and the 'strategy' subcategories is that the former addresses current position while the latter addresses future positions. However, since we found very few cases of utterances in the 'position' category, we merged the data in the results being presented here.

We faced some cases of ambiguity between inferences and management: on the whiteboard, when a subject crosses one by one the suspects they discard, they both share an inference (this suspect is not the murderer) and update the problem state (how many suspects are left). In this case, the 'inference' aspect is however more salient than the strategical aspect, and this type of actions has hence be coded as inference.



  • The meta-communication level.

This category also originally contained two sub-categories. Meta-communication in dialogue refers to utterances about the interaction itsetlf, for instance for tuning delay in acknowledgment as in example 4 (Sherlock complains that he is waiting) and 5 (Hercule apologizes because he did not acknowledged Sherlock's previous messages).

80.9

K

S

oh yes. She doesn't seems to know much...




81.2

K

H

a solution :

82.2

K

S

I am waiting...

82.3

K

H

Heidi threw her drink to HW at 7.30

Example 4: Interaction about interaction (from Pair 16)

18.1

r5

S

' are you accusing oscar?




19

Priv

H

page sherlock Sorry I was busy with the whiteboard

19.4

Priv

H

page sherlock I am not accusing him. Just found a motive

Example 5: Interaction about interaction (from Pair 5)

Meta-communication in interaction around the whiteboard involves the negotiation of the graphical codes used in the whiteboard as in example 6.



20.9

Priv

H

page sherlock We should probably use a color coding




23.2

Bar

H

Blue border - Motive/ Yellow Border - Weapon/ Green Border = Opportunity/ / Something Like that. What do you mean ?

Example 6: Interaction about graphical codes (from Pair 5 )

Here as well we faced ambiguous cases as in example 7 illustrates: Hercule (#337) questions the graphical form of an object (#337), Sherlock justifies the graphical code being used (#338), but Hercule repairs this misunderstanding and explains that was he questioned was the information being coded, i.e. Giuzeppe motive to kill (#339).



#337




H

Why did you put a second arrow?




#338




S

Because, these are those who could have killed

#339




H

But why Giuzeppe? He had n reason to kill

Example 7: Ambiguity between negotiating graphical code or content. (from Pair 1, translated)

  • The 'technique' category

The 'technique' category includes utterances where one subject asks his partner how to perform a particular action in the MOO. We did not include in this category the cases where one subject asked help (via the MOO) to the experimenters.
      1. Data collection method


The best way to describe the problem solving strategy in the chosen task is to analyze the sequence of questions asked by the detectives. When we compare dialogue and action in this report, we count these questions as actions, not as utterances. One could object that these questions are also interactions, but with an artificial agent. However, in this project, the form of these questions and their central role with respect to the task make them more relevant for describing data acquisition than for describing some communication behaviour with an artificial agent.

In this task, the subjects express questions with two parameters, the suspect and the object of the questions (e.g. 'ask Oscar about last night', 'ask Helmut about gun'....). The matrix of all questions (suspect X object) can be explored along these two axes, i.e. by suspect or by object. We describe the method of data collection by counting horizontal moves (same suspect) and vertical moves (same object) in the questions matrix (suspect X object) (Table 2).



We compute a coefficient which indicates the main axis of exploration, in the following way. We add 1 when two successive questions concern the same suspect with different objects, -1 when they concern the same object for different suspects and 0 in the other cases. We divide the sum by the number of questions. The result varies between -1 and 1. A method "by suspect" gives a coefficient around .8 (4 successive questions to one suspect then one move towards another suspect) . A method "by object" would similarly give a coefficient of -8. However, the spatial metaphor pushes detectives, even in a method "by object", to take the opportunity, when they are somewhere, to ask more than one question. Hence, the coefficient for a strategy 'by object' will be closer to 0. We later refer to this coefficient as the 'questions matrix path' (QMP) coefficient.

Mona

Night

Gun

Jacket







Marie










Rolf










Claire










Lisa







Lisa









Claire









Rolf

















Rolf









Claire






Claire







Lisa















Lisa






Lisa







Colonel












Colonel












Colonel




Lisa












Lisa












Lisa




Heidi












Heidi












Heidi

  1. Table 2: A subset of the data acquisition matrix for Hercule (Pair 10). The columns indicate the object of the question, the cells contain the name of the suspect to who the question is asked. This detective uses first a method "by object" and then "by suspect".
      1. Redundancy of questions


We computed the number of redundant questions by any detective. We refer to it as 'redundancy'. We also computed several variations of this coefficent.

  • Cross-redundancy is the number of times Hercule asked a question that Sherlock previously asked. Self-redundancy refers to the the number of times that a subject asks a question he previously asked himself. Self-redudancy may witness memory problems. Cross-redundancy may indicate bad coordination and/or group memory problems.

  • We counted differently the redundant questions asked within a maximum interval of 5 minutes (immediate redundancy) from other (long term redudancy). The threshold of 5 minutes was chosen as the inflection point in the distrubtion curve of all delays betwen redundant questions.

Redundancy indicates some sub-optimal functionning of the pair. However, some subjects may have considered that it was a good strategy to ask several times the same question to a suspect to see if it gives the same answer, as in real police interviews. We will see also cases where redundnacy does not indicate mis-coordination or memory failures (section 6.7)
1   ...   5   6   7   8   9   10   11   12   ...   18


Verilənlər bazası müəlliflik hüququ ilə müdafiə olunur ©atelim.com 2016
rəhbərliyinə müraciət