Ana səhifə

New tools for assessment in distance education

Yüklə 47 Kb.
ölçüsü47 Kb.

New tools for assessment in distance education

Liane Tarouco

Graduate Program Informatics on Education

University Federal of Santa Catarina


Luciano Hack

Graduate Program on Comuter Science

University Federal of Rio Grande do Sul


Abstract: This paper describes a strategy for improving the quality of distance education assessment provinding a set of complementary tools to help on formative evaluation. Regular forms to assess learning are analyzed and Kirkpatrick's model for evaluation is described as well as a set of tools to provide capability to asses according this model. The set of complementary tools includes consensus, tracking, voting and self-evaluation.


Learning assessment became a challenge to those working with distance education. In its definition of learning assessment, Bloom, Hastings and Madaus (Mendes 1998), proposed that the evaluation process should includes a great variety of evidences that go beyond the traditional final examination pencil and paper based. In a face-to-face context, professors use more than just formal mechanisms to evaluate students. Body language, participation, quality of questions proposed by students are good indicators of learning, In distance education what normally occurs it is the use of formal mechanisms only. It occurs not only because of the lack of accessible mechanisms that can assist instructors on this task, but also, because many professors prefer to work with more traditional methods of evaluation.

However, thanks to evolution of networks and computers and especially the Internet, mechanisms, that can fill this gap started to show up. Tools became available, to keep track of students activities and interactions with learning environment, with colleagues and with the professor. These kind of information present new opportunities for monitoring the way students learn and for learning assessment.

The use of the Internet for distance education assessment offers many advantages:

  • Reduction of the distribution costs;

  • The corrections and updates are simpler, therefore they are carried through in an only site, being immediately spread to all students;

  • Several techniques for evaluation are possible using multimedia for the between professor and students and between students;

  • The Internet facilitates collaborative writing;

  • The student has more easiness in sending feedback allowing formative and evaluation

A research conducted by Dirks [Dirks] with instructors of distance learning MBA programs presented some questions like: Considering all the time you spend on this course (100%), how much of that time do you think you spend on student assessment (writing tests, grading, giving feedback, and reporting scores)? Answers indicated that the newer teachers, 2-9 years teaching, spent over half their time on assessment (56%), those with 10-19 years experience spent less time on assessment (46%), and the career instructors, 20+ years teaching, spent the least amount of time doing assessment (35%). The primary reasons given for having assessments include providing feedback, giving grades, and motivation. The four types of assessment most commonly mentioned are: case studies, exams, papers, and projects. All of them requires extensive time to grade and are very difficult to get automated help to do the evaluation.

Other researches point also that what is often used for assessment in distance education are the following:

  • individual works developed individually and sent by regular mail or by email

  • assessment based on contributions for group discussions

  • tests (automatically handled by computer program)

  • term papers (analysed by professor or assistants)

  • oral or written tests conduced in the presence of the instructor (some times through videoconference) or with a remote assistant.

Learning theories state that group learning has significant relevance and must be supported also in distance education. Participation in group activities, cooperation and collaboration must be supported and graded. But there are a lack of good tools to help evaluate the participation of the distance education student in group activities.

Slowly, thanks to evolution of networks and computers and especially the Internet, mechanisms, that can fill this gap started to show up. Tools became available, to keep track of students activities and interactions with learning environment, with colleagues and with the professor as indicated in figure 1.

Figure 1: Internet as mediating tool

These kind of information present new opportunities for monitoring the way students learn and for learning assessment.

As stated by Tucker (Tucker 1998) the CMDL (Computer Mediated Distance Learning) classroom is routinely a far more scrutable educational environment than the physical classroom. The virtual classroom's electronic data storage, retrieval and exchange system (i.e., the text of student and faculty transactions, communication logs, file structures and information presentation algorithms that exist on the file server's hard disk drives) represent concentrated, structured and highly accessible artifacts of the learning transactions. These artifacts can be retrieved, analyzed and reported via highly economical, automatic processes operating in the background of the communications system.

This work presents a proposed solution for assessment on distance education mediated by Internet using innovative approaches based on Kirkpatrick model for evaluate training problems and will describe how they where implemented.

2. Assessment models and tools

There are today some systems, available in the market, that provide detailed information on the evolution of the student working at distance mediated by Internet. Examples of this kind of system include, CyberQ, WebCT and AulaNet. Statistics of usage and regular approaches for testing like multiple choice quiz and term papers are usually possible in these kind of systems.

Donald Kirkpatrick developed a model of evaluation, used in training programs. It recommends a division of the evaluation process in four levels (Kirkpatrick 1998):

  • Level 1 : Reaction - level measures how those who participate in the program react to it. It may be called a measure of student satisfaction.

  • Level 2 : Learning - can be defined as the extent to which participants change attitudes, improve knowledge, and/or increase skill as a result of attending the program.

  • Level 3 : Behavior - can be defined as the extent to which change in behavior has occurred because the participant attended the training program

  • Level 4 : Results - can be defined as the final results that occurred because the participants attended the program.

Mendoza is other researcher that that it is also relevant to investigate student context to find out which aspects of course contents he/she is in the know (Mendoza 1999). So an aspect to be also considered is the lack of suitable tools to asses their current level of knowledge and skills. Mendoza recommends utilization of techniques, other that conventional ones, to collect data and suggest formative evaluation as well as summative evaluation. To achieve formative evaluation is important to make easy for the students not only inform their opinions and feelings about how course is going on but see this contributions being handled and considered.

To help in all these tasks and also support evaluation as proposed in Kirkpatrick model one should have a tool that would help organize polling and receive data in a suitable form to allow counting and summarizing results. To do this, it was designed a system aimed to support flexible and dynamically built of enquete for polling. The system has a WWW interface that allows designing of different kinds of multiple choice questions. Basic data informed are question and alternatives. Depending on type of desired measure the system ask for:

  1. Feedback for each alternative

  2. Indication of correct answer

  3. Nothing more, just counting incidence of selections on each alternative

Option a is used for questions inserted in the learning path to ask for active response. Option b is used for testing and give grades. Option c is used for polling purposed. This system was designed to complement regular tools used to assess distance education students. The following mechanisms have been selected to assist in the evaluation process in this framework:

  • Tool of consensus: based in the Delphi technique, allows on time creation of enquete proposing instigating questions, as well as the collection and tabulation of responses. The tool has the capability to keep track of each student's level of contribution to the discussion as well as enquete responses.

  • Tracking tool: the register of learning activities (accessed page with dates and time) is important because it allows the professor to monitor the progress of students trough the course. The collected information will show that type of access the student has made (visited units, tools used, etc), as well as the time spent on it.

  • Voting tool: aimed to provide conditions for a fast feedback for the instructor on specific subjects.

  • Self-evaluation tool: allows the dynamic building of enquetes for self-evaluation by professor and by students.

It is important to stand out that this proposed work is not an alternative approach to assessment rather it is a complementary one and is supposed that may be used together. Each on of this tools are supposed to be used to help evaluate at more than one level in Kirkpatrick model

To implement the tracking tool, a generic interface was designed to intermediate all the interaction of the student with the server. Interactions are processed in real time (for authentication purposes) and periodically in batch to generate records in a student data-base.

To implement the voting, consensus and self-evaluating tools a set of CGI programs where developed to allow dynamically create pages with questions and propositions (HTML forms) that are presented to students and collect their answers (Tarouco 1997).


It is our believe that formative evaluation plays an important role to get distance students motivated because they feel a sense of not being lost in space. Just the opposite, they feel that being closely monitored and having their opinions being asked often is more like having a personal tutor that pay attention on way they do and what they say. This kind of attention provided by the system helps to minimize solitude that sometimes is referred as main desisting factor from students.


  • Dirks, M.. 1998 How is Assessment Being Done in Distance Learning? .

  • Kirkpatrick, Donald 1998. Evaluating Training Programs - The Four Levels.. Berret-Koehler Publisher, Inc - San Francisco.

  • Mendes, Marcel. 1998. Conceito e significados de avaliação de aprendizagem.

  • Mendoza, Judith. 1999. Conceptualización de la evaluación de aprendizaje y su relación com el proceso de instrucción en los sistemas a distancia: vía para mejorar la calidad de la instrucción. In: Acerca de la distancia. 3er. Seminario Internacional de Educación a distancia. Pp (299-302) Eudecor SRL. Córdoba, Argentina

  • Tarouco, Liane. Pereira Luiz. 1997 Quiz Building on the Web without CGI or Java Programming. Interactive Multimedia 97. SALT. Washington-DC, 18-22 de agosto de 1997.

  • Tucker. R. 1998 Assessing the Virtual Classrooms: A Progress Report.

Verilənlər bazası müəlliflik hüququ ilə müdafiə olunur © 2016
rəhbərliyinə müraciət