Evaluation

 Introduction

Evaluation is the meaning or interpretation of the data from the assessment In an institutional setting and the evaluator may be students, faculty, administrators or accrediting agencies.  The results of an assessment process should provide information which can be used to determine whether or not intended outcomes are being achieved and how the ODL material can be improved.

In the assessment process it is important to distinguish between formative and summative assessment.  Formative assessment is the collection of data and the feedback of the results on an ongoing basis. Formative assessment is designed to provide information for the purpose of improving the ODL material being assessed. Summative assessment is designed to produce information that can be used to make decisions about the effectiveness of the ODL process.
 

 Preparation of an Evaluation Scheme

The following six phases present an outline for the development of an assessment plan. The stages are illustrated with examples on the following pages. The examples follow the progress of a unique objective like “... assure that learners can create homepages in the web”.



 Stage 1

 Identify goals and learning objectives.

 Question a
- What is to be achieved?

The first step is to write a statement describing the broad outcome desired. Program evaluation is intended to provide information on how well a ODL material performs relative to its specific objectives and, ultimately, its broad goals. A goal should be far reaching and describe the best situation that could possibly be hoped for.

 Question b
- Under what circumstances will you know the goal has been achieved?

The learning objective is a statement derived  from  the  goal  that  defined the circumstances by which it will be known if the desired change in learning skills has occurred.  These learning objectives are precise in stating expected change, how the change should be proved, the expected level of change, and over what time period the change is expected.



 Stage 2

 Develop performance criterion for each learning objective.

 Question
- What will students be able to do, or be, or possess when the goal is accomplished?

The performance criterium is a specific statement identifying the outcomes required to meet the objective and must be proved through  evidence. It is important to identify criteria against which results will be measured.   The indicator of performance are not easily apparent so therefore indicators are used to provide evidence of whether or not the desired performance exists.



 Stage 3

 Determine the practice to be used to achieve goals.

 Question
- What will be done to achieve the goals and the learning objectives?

It is important to define the practice to be followed in the classroom or by the individual learner prepared to obtain the specific performance for each learning objective.



 Stage 4

 Select and apply assessment methods for each learning objective.

Question
- What are the methods to obtain the information about each learning objective?

The specified assessment methods to collect proof about the quantification of the learning indicators are used and the results are analyzed and compared against the performance criteria defined beforehand.  The methods should be consistent with the criteria adopted. These can be enquiries, rating sheets,   interviews and focus group protocols, etc., as appropriate.



 Stage 5

 Question
- How to establish the communication channels to get the information about the ODL materials evaluation?

The implementation of the feedback channels which provide information at appropriate timings to facilitate continuous improvement of practices, provide information for decision making, and provide basis for evaluation of the ODL material.  One of the results of the evaluation is that faculty may determine that is necessary to provide intervention and/or provide complementary training.  When results indicate that students are performing consistent with established criteria, faculty may extend the current practice to other students and to other courses.



 Stage 6

 Question
- Have the performance criteria been met and the objectives achieved?

Since evaluation is the process of classifying meaning and value to assessment results it usually occurs during the continuous improvement process (formative evaluation) and at the end of a project (summative evaluation).  A decision will need to be made concerning how many of the performance criteria must be met to say that the learning objective has been achieved.
 

 Examples

 1. Interactivity

The example is dedicated to measure interaction as a performance criteria of stage 2.
The interaction takes place between instructor and learners, between learners and distant sites available through videoconferencing or other synchronous media.  The questions asked will have to measure the level of interacticvibity between learners and tutors in a scale of 1 to 5 about the following issues:

 a) possibility for questions
 b) participation of discussions on-line

 c) engagement in learning activities

 d) choice of themes for discussion

 e) off-line discussions

 2. Learners evaluation

 According to stage 4 the measurement of learner satisfaction with the learning activities of the ODL sessions has to be rated in a similar manner to the face-to-face learning activities.  The rating should be done as a measurement on a given numeric scale of the following issues:

 a) active involvement
 b) activities were related with workplace tasks

 c) interesting challenge

 d) help to achieve learning objectives

 e) clairity of materials

 f) activities will help job tasks

 

3. Self-Evaluation Form:
Criteria :    Leadership

How the behaviour and the actions of the management team and others organization leaders encourage, support and promote the culture of  Quality  Management.

Definition of criteria:
How the management team inspires and controls the total quality as a fundamental organizational process for  continuous improvement.

Goals:
To learn to appreciate the implication and the commitment of the management and executive teams.

Example of criteria question
How leaders visibly show  their commitment with the  Quality Management philosophy.

Areas to approach:

  - How leaders form a model, managing by example

  - How leaders will became accessible, hearing and answering all  the members of the company.

  - How leaders improve their leadership.

  - Which are the strong points

  - Which are the improvement areas:

  - Further observations

  - What is the total score

 
 

 Subcriteria 1.a.:

How leaders visibly show  their commitment with the  Quality Management philosophy.

Areas to approach:

- How leaders form a model, managing by example
- How leaders will became accessible, hearing and answering all 

the members of the company.

- How leaders improve their leadership.

Strong Points :
 
 

Improvement areas:

Evidence:
Observations:
Score (to be filled in for the external team)

Focus
 

Developement: Overall score:





 
 

Task/Activity to do on your own

Now that you have gone through all the stages of an evaluation process, Your task is to evaluate two web courses.
We have some suggestions of sites with web courses for you, but feel free to pick any other courses.

Send your evaluation together with the address of the web site you choose to
avsoeiro@reit.up.pt
 
 


http://sorga03.aero.usyd.edu.au/~ahmad/index.htm
http://www.nald.ca/nbinternational/english/2132.htm

http://database.telecampus.com/

http://www.dipoli.hut.fi/org/humanities/

http://www.stac.ac.uk/edu/ODL/EN/decimal.htm