A formative evaluation of a web-based tutorial

Lecturers of the Department of Nursing Science of the University of Pretoria (UP) were requested to participate in an informal evaluation of a software program. The software evaluated were developed as a tutorial by the author of this article. The content of the tutorial concerns the current issues of the new education and training dispensation in South Africa, and specifically how to compile unit standards to meet the requirements of the South African Qualifications Authority (SAQA) and the National Qualifications Framework (NQF). The purpose of the evaluation was to assess the quality of the software. Participants were provided with a questionnaire, as well as criteria for the evaluation of software as presented in literature and information on the Web. The feedback has been useful and suggestions made by the participants have been used to improve and add to the product.


Introduction
Since the introduction of the Internet and the World-Wide-Web, information of all kinds became available to every person who is computer literate.To obtain information/knowledge is not the biggest issue, but rather what is being done with that information and how it is applied.The emphasis should now shift from obtaining information, to using the information/ knowledge at one's disposal to improve one's skills.All information available on the Web is not necessarily of high quality and will not enhance learning.The same applies to software developed for educational purposes.Before any soft ware can be incorporated into a program as part of lecturer or student support, it must first be evaluated.The ideal evaluator/assessor would be somebody who is com puter literate and who is a specialist in his/her subject.In order to ensure an objective assessment, more than one person should do the evaluation.

Problem Statem ent
The author has developed software, which is published on the Web.The subject of the software falls within the context of the new requirements of the South African government on educa tion and training, namely The Compilation o f Unit Standards as required by the National Qualifications Framework (NQF).All lecturers of the Department of Nursing Science at the Uni versity of Pretoria were invited to participate.The purpose of the software is for lecturers who have to develop curricula according to the SAQA requirements, to gain insight into the development of unit standards by first doing the tutorial which provides opportunity for exercises.

Background of the Problem
Computer literacy has become a key phrase in defining the aim of many educational institutions.It is, however, important to note that people must be literate, implying being able to read and write, before they will be able to become computer literate.Computers are limited in their teaching abilities, and man will always be the better teacher (Barlow, 1999).This statement can lead to many a debate, but the important issue here is that educational software will only be as good as the person who created it.Educational software/technology has a definite place in edu cation as it allows students to use it as often as they deem necessary, especially if a lecturer is not available.Computer literacy has become part of the general frame of reference of many educators.Information associated with computers and the Internet is the new focus point in obtaining information and acquiring new skills.Computer software can be employed as support tools to ob tain information regarding subject matter pertaining to a myriad of educational situations.It also serves as an excellent tool for in-service training or continued learning for adults/lecturers as they can study the content of a specific software program in their own time, as well as repeat the lesson/tutorial as often as they would like to.Tutorial-type computer software presents learners with infor mation or skills, and guide them through the application of the newly acquired information, knowledge and skills (Alessi & Trollip, 1991:17).Instructional tools/tutorials demand an alter native way of thinking regarding the use of computers in edu cation (Blignaut & Knoetze, 1999).Students must, however, not be forced to rely on computer skills only to meet the out comes of programs.Educational software alone cannot influ ence the cognitive development of a student.Human inter vention in the form of the educator is therefore imperative.What needs to be established is the quality of software used for continued learning or in-service training.All software pro grams do not contain useful information and are not necessar ily user-friendly.Therefore, many instructional designers have written on the subject of evaluating software regarding the applicability of content, interface design, and usability.
If one considers the computer and computer software as part of educational strategies, it could be described as "technol ogy " assisted education.The role that technol ogy should play in nursing education should be as Gentry (1987:3) describes it: "Technology should be the servant and not the master of in struction.It should not be adopted merely be cause it exists, or because an institution fears that it will be left behind the parade of progress without it".

Criteria for the evaluation of software
According to Trochim (1999:1), evaluation is prob ably mostly defined as ''...the systematic assess ment of the worth or merit of an object".How ever, evaluation does not always result in the assessment of worth.Trochim (1999:1) provides another, probably more accurate definition: "Evaluation is the systematic acquisition and assessment of information to provide useful feed back about some object."The "object" in this instance, is a web-based tutorial.Evaluation/assessment of educational software is not an easy task.There are two reasons for evaluating/assessing software: 1) to improve the instructional possibilities and 2) to improve the quality of the software per se.The evaluation can be done in both an informal and formal manner (Hannafin & Peck, 1988: 279).Evaluation provides feedback, useful or otherwise.There are mainly two forms of evaluation namely formative and summative evaluation.Forma tive evaluation assists the development of an object, as it can be developed and improved as people evaluate and give feed back on the quality, applicability and usability of the object.Summative evaluation is the final evaluation regarding the im pact of the object on outcomes (Trochim, 1999).It is of the essence to establish whether or not specific soft ware complies with set criteria, and also whether it will assist the user/learner/lecturer towards the achievement of pre-de- The possible fields and sub-fields to which the unit standards apply should be indicated termined goals, aims, objectives or outcomes.Various meth ods can be employed to establish whether the selected soft ware complies with the set requirements, and whether or not it can be used effectively for educational purposes.According to Hannafin & Peck (1988:279), the evaluation of computerassisted instruction is "an ongoing process that occurs con tinuously, drawing on a variety of information sources and techniques." The software evaluated The author requested lecturers who are computer literate to volunteer for this project and five lecturers decided to partici pate.The participants were asked to evaluate the Web based tutorial.The tutorial consisted of the following parts: A glossary containing relative terminology was added to the html document.The purpose of this is to assist the user in understanding the paradigm shift regarding education and mastering the new educational "language".The purpose of the slide show was to provide the user with additional informa tion on the compilation of unit standards.The tutorial was available on the server of the University of Pretoria (UP), and therefore the participants, who have computers in their offices linked to the Intranet of the UP.They were assisted by the author in accessing the Intranet, and to work through the tuto rial.No graphics were used in the creation of the web site.The "home" and "back" buttons were, however, animated.The background of the buttons reflected a light green, with the letters of the word "back" and "home" a darker green, and appearing one by one, again and again.The background col our of the site was dark orange, and the font used, Lucida sans Unicode 10 bold in royal blue.A one-pixel frame, with a cream background framed the headings of the different sections.The font that was used for the heading was in Parisian BT 24 bold.Links were made in bright yellow.See Figure 1 to assist de scription.The MS PowerPoint slide show, Compiling Unit Standards (A support too!) linked to the site, was created on a Microsoft template, and the original green background changed to blue.The background contains a few stationary "ribbons" in shades of blue, with a "block" floating from left to right over the top third of the screen for approximately three seconds each time the user clicked on the "next" button.The font used for the slide show is Gill sans 32 in pale blue, with some of the slides with pop-up boxes created in Visual Basic, and containing defi nitions and explanations of the concepts of specific slides.The background of the pop-up boxes was medium blue, with the text in a pale blue font of Gill Sans 12. See Figure 2 to assist description.

Purpose of the study
The purpose of the study was to assess the quality of the software that was developed, and to improve the product if and where necessary.

Methods and procedures
Participants were provided with evaluation criteria.The evalu ation form of Joseph (1999) was adapted and provided to as sist them in the evaluation of the tutorial, as they had no prior knowledge regarding the criteria for the evaluation of soft ware.Please see table one for these evaluation criteria.The software has been developed and finished in totality before the participants did the evaluation thereof.All five participants are computer literate, but on different lev els.Please see table two in this regard.They did not find it too difficult to work through the tutorial.The participants were allowed to ask each other for assistance when they did not understand how to continue with the tutorial.They spontane ously asked the author for assistance when it was needed.After the participants have worked through the tutorial a focus group discussion took place to get feedback.Field notes were taken and added to the data obtained from the evaluation forms.During the focus group participants were allowed to talk to each other and therefore the clarity of the findings were en hanced.The roles performed by the five participants were that of evaluator, advisor and critic.The feedback on this software served a formative purpose, as the author of the web-based tutorial used the information to improve the product and it will be used for the development of future nursing educational software programs.Information/evaluation criteria provided to participants The definition by Hannafin and Peck (1988:303) was provided: "Instructional adequacy is the extent to which the lesson pro vides the necessary kinds of support and features to accom plish the objective at hand".The following information on lesson procedure, clarity, efficiency and design of a lesson for the computer from Hannafin and Peck was also supplied: • Lesson activities must be easy to follow Provide sufficient information for the user to solve problems, reach conclusions, grasp a concept, and learn a skill or process".Additional information on the following to take into consideration when evaluating the software, was pro vided from http://hagar.up.ac.za/catts/ole/eel/concept-map/cosmet2.html:Specific criteria on screen design regarding the evalu ation of user-friendliness, the evaluation of visual display, such as the use of:

Data analysis
Findings The W eb-site Negative aspects: • The default setting of the font of the computers of the participants influenced the opinion regarding legibil ity, the reason for this being that one participant's opinion regarding the font size was that "the letters were too small and reading difficult".It seemed that the font was set as Times New Roman 10 regular and that the font used for the site was Gill sans 12 bold.The participant was, however, of the opinion that it might be the background and font colours causing the illegibility.
• One participant was actually expecting a lit tie more "excitement".She was, however, the participant with the most computer ex perience.She was obviously used to brows ing the Internet, although the author doubts whether the sites visited by this participant had all been educational in nature.

•
The animated home and back buttons both ered only one participant.Her comment was that the flickering was "distracting".

•
The grammar was bothering two participants as one found a spelling mistake, and the other found the sentences to be abrupt.

•
Most of the participants requested a con tact link to the author on the site but the author deemed this unnecessary.The soft- The slide-show

Negative aspects:
Aspects identified were: • The blue block (see fig ure 2) moving slowly over the top of the screen was irritating and creating frustration as one could not move to the next slide before it stopped moving.• Each link in the web site was linked to the home/ first page of the slideshow.The participants would have preferred it linked to the specific slide applica ble.However, the author was of the opinion that it was unnecessary, as the slide show contained a menu of the topics.

Positive aspects:
The majority of the comments on the slide show were positive: • Good colour combination • Information relevant, applicable and correct • Font type and colour makes reading easy

Changes
The following changes were made to the html document: • An informative, more personal introduction that addresses the user was written called Compiling unit standards according to the South African Qualifica tions Authority (SAQA) and the national Qualifica tions Framework (NQF).The background is white with the characters in a font of bold Gill Sans of 12 points.A link to the html document called The compi lation o f Unit standards as required by the National Qualifications Framework (NQF) is provided at the bottom of the screen (see figure 3).
• The sitemap of the html document was changed completely.
It was given a white background.The heading was put in a frame of one pixel with a blue background and the title in white in a font of Gill Sans bold 20 points.An icon was used for aesthetic pur poses before every link to the different parts of the document (see figure 4).ards (see figure 8).After every exercise the user can clear the form and start over again.
• A Microsoft Word document, Guidelines fo r using the Access software to com pile a unit standard, to guide the user on how to use the support tool created in MS Access (see figure 10).
To enable the user to use the MS Access tool it was necessary to provide the user with guidelines.These are provided in the MS Word document.A link on the html file pro vides information and a link to the document (see figure 9).The slides contain navigational links (see figure 5).

•
To assist the user in gaining additional information regarding unit standards, buttons to open text boxes containing that information was designed on the slides (see figure 6).
The following additions were made to the html document: • A Microsoft Access database, Compiling a Unit Standard (A support tool), that provides in forma tion as well as an opportunity to practice writing unit standards (see figure 7).This tool was also provided with buttons that can open text boxes contain ing information on aspects of unit stand- The evaluation that was done has been a posi tive experience for both the author of the soft ware, as well as the participants.The web site called The Compilation o f Unit Standards as required by the National Qualifications Framework (NQF) has been im proved.The initial tutorial lacked interactivity and opportu nity to practice.The evaluation of the tutorial resulted in a valuable learning experience for the author.The author is of the opinion that there is definitely a place for tutorials of this nature in a personnel development or further education pro gramme.This tutorial can be viewed at http://hagar.up.ac.za/ catts/leamer/1999/meyer_sm/m_ed/opdragte/oro/unitinfo.htmThe p u r p o s e a n d t h e v a lu e o f a q u a lif ic a t io n c a n b e u n d e r s to o d as a n a n s w e r t o th e q u e s tio n "W h y ?" as in "\ W iy t h is q u a lif ic a t io n ?" T h e S A Q A f o r m a t fo r r e q is t e r in q u n it s ta n d a r d s a a d q u a lific a tio n s r e q u ir e s t h a t th e p u r

Figure 1 :
Figure 1 : The introductory page of the web site on the Principles of the N Q F « * h N m l Nftlarcap*

Figure 2 :
Figure 2 : One slide of the M S Pow erPoint slide show

Figure
Figure 4 : Sitem ap for tutorial

FFigure 6 :
Figure 6: Slide of the slideshow with textbox open For the text Gill Sans in bold point 12 in blue was used.a t c r c -i t i r t j n c D s i c n b f s h > w • itm c if t^r a a c 'c o f v jtc c 4 n * w iB b e together with administrative and other necessary information.

Figure 7 :
Figure 7 : M S Access tool to practice writing unit standards EE ; (dl ^cw kncsl Q eoards Jocis IjM rxtow fcjefc l £ • w .& & V .' -e .v • -» l S i V 'H JHfPonie | ^MyC oiwb | | g j j f n m i j * .N I IBjtCTCTreda |[g } w te fo » o fl i t .» PowerPoint was used for the slide show.The template looks like a notebook.The colours are different shades of brown.The font used is Ariel in brown in a large point size.

Figure 9 :
Figure 9: The inform ation and link on the html tool to the M S Word docum ent «BlRjiansiigife-UiLaJ 3 ái -* rf Is E J Bdbed Hone 5««k £i Netscape Part S « c m y _ _ M H p o s e b e s t a t e d e x p lic i t ly , This s t a t e m e n t o f p u rp o s e s h o u ld s p e c if y th e a p p lie d c o m p e t e n c e t o b e a c h ie v e d a n d lin k th is d i r e c t l y t o an a c a d e m ic a re a o r fie ld o f le a f n in q a n d a n o c c u p a t io n a l r o le , in a d d it io n , t h e p u r p o s e s h o u ld in c lu d e an e x p la n a tio n o f th e n e e d f o r th e q u a lif ic a t io n .T he p u r p o s e a n d v a lu e o f t h e q u a lif ic a t io n p r o v id e s a m e a n s b Figure 1 0 : A M icrosoft Word document to guide the user on how to use the support tool created in M S Access •The Compilation o f Unit Standards as required by the National Qualifications Framework (NQF) was created in Netscape Composer.It also contains a resource list at the end of the site, with Universal Relative Links (URL's), providing the user with the opportunity of visiting the sites with information of SAQA, • A Microsoft (MS) PowerPoint slide show, Compiling Unit Standards (A support tool), has been linked to certain sections of the tutorial.