THE DEVELOPMENT OF A WEIGHTED OBJECTIVE RESEARCH CRITERION ASSESSMENT TOOL ( WORCAT )

The aim o f this study was to develop an instruinent to evaluate research reports objectively according to se t criteria, relative weights and critical errors; A tentative instrument was compiled and copies were sent to 144 university lecturers and senior lecturers at South African Nursing Science Departments in order to determine: • the face validity o f the assessment criteria o f the 18 stated aspects o f research reports • the weight or relative value o f each o f the aspects • hoM> relationships between aspects should be evaluated The relative value o f each o f the 18 aspects was determined by completing a matrix on which aspects were compared The Analytic Hierarchical Procedure was used to determine the weight o f each aspect. The WORCA T was developed from the findings and the most striking feature is the high relative weight allocated to each o f the following aspects o f research reports: recommendations = 20 % ; conclusions = 10 % literature study ^ 1 0 % ; \ problem staiement = 7 % The reliability o f the WORCA T still needs to be tested. OPSOMMING Die doel van die studie was om ’n instrument te ontwikkel om navorsingsverslae objektief volgcns gestelde kriteria, relatiewe gewigte en kritiese foute te evalueer. 'n Tentatiewe instrument is opgestel en aan 144 lektoreen senior lektore verbonde aan Departemente van Verpleegkunde aan Suid-Afrikaanse Universiteite gestuur o m : • die siggeldigheid van die ramingskriteria van die gegewe agden aspekte van ’n navorsingsverslag te bepaal • die relatiewe gewigte o f waardes van elke aspek te bepaal • te vas te stel hoe die verwantskappe tussen die aspekte evalueer behoort te word. Die relatiewe waarde van elke aspek is bepaal deur die voltooiing van ’n matriks waarop die aspekte teen mekaar opgeweeg is. Die gewig van elke aspek is per rekenaar bepaal en wel deur middel van dieAnalitiese Hierargiese Prosedure. D ie WORCA Tis ontwikkel uit die bevindinge waarvan die mees uitstaande kenmerk die hoe relatiewe gewigte is wat aan die volgende aspekte toegeken is : aanbevelings = 2 0 % ; literatuurstudie = 10 % : bevindinge = 1 0% probleemstelling = 7 % Nursi ng research is the scienti fi c approach that o ffers im portant resources for answ ering difficult clinical and health-related questions (Wilson 1983). Only by discovering these problems in our practice and finding solutions by m eans o f re sea rch can w e shape a profession based on meeting the needs of those However, when we judge research findings we do not primarily focus on answering research questions but rather on the research process and the product o f that endeavour (Duffy 1985). The focus should be on the merit and worth of the recommendations as well as the possib ility o f incorporation into clinical practice (Beck 1990). This means that a research report should not be judged merely by evaluation of the report against criteria based on the research process but that more emphasis should be placed on the conclusions and recommendations since they indicate how the problem manifests and could be solved. Just as the recommendation of solutions is of u tm ost im portance to the researcher, the m a tte r o f o b je c t iv i ty c a n n o t be overemphasised in the case of the evaluator. Although objective evaluation of a research report is extrem ely difficult it is vitally important; not only to be fair to students who submi t research reports as part of requirements in obtaining degrees but also for the awarding o f grants and prizes.


A B STR A C T The aim o f this study was to develop an instruinent to evaluate research reports objectively according to se t criteria, relative weights and critical errors; A tentative instrument was compiled and copies were sent to 144 university lecturers and senior lecturers at South African Nursing Science Departments in order to determine: • the face validity o f the assessment criteria o f the 18 stated aspects o f research reports • the weight or relative value o f each o f the aspects
• hoM> relationships between aspects should be evaluated

• die relatiewe gewigte o f waardes van elke aspek te bepaal
• te vas te stel hoe die verwantskappe tussen die aspekte evalueer behoort te word.Nursi ng research is the scienti fi c approach that o ffers im p o rtan t resources for answ ering difficult clinical and health-related questions (W ilson 1983).Only by discovering these problem s in our practice and finding solutions by m e a n s o f re s e a rc h can w e sh a p e a profession based on m eeting the needs of those However, w hen we judge research findings we do not prim arily focus on answering research questions but rather on the research process and the product o f that endeavour (Duffy 1985).The focus should be on the merit and w orth of the recom m endations as well as the p o ssib ility o f in co rp o ratio n into clinical practice (B eck 1990).This m eans that a research report should not be judged merely by evaluation o f the report against criteria based on the research process but that more em phasis should be placed on the conclusions and recom m endations since they indicate how the problem m anifests and could be solved.

D ie relatiewe waarde van elke aspek is bepaal deur die voltooiing van 'n matriks waarop die aspekte teen mekaar opgeweeg is. Die gewig van elke aspek is p e r rekenaar bepaal en wel deur m iddel van dieAnalitiese
Just as the recom m endation of solutions is of u tm ost im p o rtan ce to the researcher, the m a tte r o f o b j e c t i v i t y c a n n o t b e overem phasised in the case of the evaluator.A lthough objective evaluation of a research rep o rt is ex trem ely d ifficult it is vitally important; not only to be fair to students who submi t research reports as part of requirements in obtaining degrees but also for the awarding o f grants and prizes.

Problem statement
N o instrum ent w hich m akes provision for objective as well as weighted assesment of quantative research reports exists.

A im
T o d e v e lo p an in s tru m e n t to e v a lu a te q u a n ta tiv e re s e a rc h re p o rts o b je c tiv e ly according to set criteria and relative weights.

Objectives
• Carry out literature study and compile a tentative instrum ent com prising evaluation criteria for all the aspects of a research report.

Literature review
A research critique is viewed as a critical appraisal o f a research study that has been sy ste m atically rev iew ed b ased on som e known criteria (Beck 1990).Critical research appraisals have been addressed by several authors w ho agree on the im portance of appraising research critically but disagree on the best way o f doing so (D uffy 1985).
Leininger (1968) identified a few guidelines for doing a critique but does not define the acceptable criteria.Ward and Fetler (1979) go a step futher and provide an extensive checklist o f aspects to be considered.The value of the checklists is not only to remind the reader of the many features which should be appraised but should also im prove the d e g re e o f o b je c tiv ity (B in d e r 1 9 8 1 ).Although checklists are valuable they have shortcomings because they are developed by individuals to help them evaluate their own and o th e r s ' re p o rts and to help read ers understand and learn from reports.They are therefore not complete lists o f all criteria and a p p ra isa ls o f c rite r ia a re b ased on the evaluator's know ledge o f the research process (Duffy 1985).

Limitations
Rating of research criteria depends on the research expertise of the rater.
The completion o f the matrix w as not well understood which made it di fficult to complete and resulted in a low return rate.tentative evaluation instrum ent did not make provision for the evaluation of relationships between different aspects o f a research report, questions were set on how to deal w ith these relationships.

Method
A matrix was drawn up com prising 18 aspects o f a research report on both the horizontal and the vertical colum ns (Fig. 1).
R espondents w ere requested to assign weights to the aspects by compari ng each o f the aspects in the vertical colum n with each o f the aspects listed in the horizontal colum n.The weight should be the value of the vertical aspects relative to the value o f the horizontal aspects and indicated as in Fig. 2.    • The wei ght o f an aspect compared to itsel f will alw ays be 1 (shaded squares).
• If the aspect in the vertical colum n is considered to be more im portant than the aspect in the horizontal column, then the weight allocated will be 2, 3, 4 or 5.
Example -Square 1: If the aspect "title" (vertical co lu m n ) is considered to be twice as im portant for research report evaluation as "limitations" (horizontal column), ihe weight w ould be 2.
• If the aspect in the vertical colum n is considered to be less important than the aspect in Ihe horizontal column the weight will be 1 /2 ,1 /3 ,1 /4 , or 1/5, depending on how many times the aspect in the horizontal colum n is more important.
Example -Square 2: If eg."title" (horizontal colum n) is compared with "recom m endations" (vertical colum n) and "recommendations" is considered to be three lim es as important the "title", the w eight of "title" will be 1/3.A namelist o f all lecturers and senior lecturers at South African Universities w as obtained and a tentative research eval uation i nstrument, questionnaire and matrix were sent to 144 lecturers.

Data analysis
Only 33 lecturers responded which gives an extremely low return rate o f 23 %.O f these 33 only 23 completed the matrix correctly.Since the respondents' research involvement varied extremely it w as decided to divide the sample into experienced and inexperienced groups since the determ ination of relative w eights o f aspects o f a research report in particular required some experience.It w as decided arbitrarily that the criteria for the experienced group would be at least two years of teaching nursing research, 20 evaluations o f research reports done and one publication.Data on the two groups' involvement in research appear in Table 1.The relative weights for each of the 18 aspects o f a research report w ere determ inded by m eans of the A nalytic Hierarchical Procedure a softw are program m e developed by T.L. Saaty.The values o f the 18 weights add up to 100.The relative w eights allocated by the experienced and inexperienced groups are given in Table 2.
It was interesting to note that the two biggest di fferences i n rel ative wei ghts al located by the tw o g ro u p s w e re fo r th e p ro b le m and recom m endations.T he experienced group allocated 4 more for problem statem ent and 10 more for conclusions w hich underli nes the two main com ponents o f W ilson's view of nursing research i.e it is the approach that offers resources for answ ering questions (Wilson 1983).The relative w eights for each o f the 18 aspects of a research report, as determined by th e ex p e rie n c e d g ro u p , are in d icated as m aximum scores on the evaluation instrument (Fig 3).The m axim um score for each aspect w as divided arbitrarily between the criteria w hich w ere to be met.It would be possible to determine the relative weight o f each criterion o f an aspect in the sam e way as the w eights for th e a s p e c ts o f a re s e a rc h re p o rt w e re determined.
Respondents w ere requested to comment on relationships between different aspects o f a research report.Table 3 reveals their opinions on w hether the relationships betw een the s t a t e d a s p e c t s s h o u ld b e ta k e n in to consideration or not.Suggestions could be made for additional relations to be evaluated.-deducting a certain percentage from the total marks obtained from the criterion evaluation should aspects not relate.
Only 74 % o f repondents gave their opinion on how relationships betw een aspects of research reports should be evaluated: -59 % advocated the allocation o f a certain p e rc e n ta g e and the av erag e percentage suggested w as 29.
-The deduction of (on average) 14 % from the total marks w as suggested by 41 % of the respondents should relationships be inadequate.

Findings
Respondents' involvem ent in research varied to such extent that they were divided into an experienced group and an inexperienced g ro u p .C r it e r i a f o r in c lu s io n in th e experienced group w ere a minim um of two y e a rs o f te a c h in g n u rs in g re se a rc h , 20 e v alu atio n s o f research rep o rts and one published research report.
Very few additions were made to the tentative research assessm ent instrum ent by the two groups.The instrument comprised evaluation criteria for eighteen aspects o f a research report.
The relative w eights allocated to the eighteen aspects of a research report by the two groups varied considerably especially pertaining to problem statem ent and conclusions.The experienced group allocated 4 and 10 more marks respectively for these two aspects.The experienced group allocated a relative w eight o f 20 to recom mendations w hich m eant that 20 % of marks given for a research report will be for the recom mendations made.
T h e W O R C A T ( W e ig h te d O b je c tiv e R esearch C riteria A ssessm en t T ool) w as com piled by listing each aspect w ith its evaluation criteria together with its relative weight as maximum score.
In addition to the w eighted assessm ent o f criteria the relationships betw een aspects should also be evaluated.T his should be done by allocating 29 % o f the total m arks to relationships as recom m ended by 59 % o f the 17 respondents who responded to this part of the questionnaire.The rem aining 41 % w ere o f the opinion that 14 % should be deducted from the total marks.

Conclusions
The W O RCA T seem s to have face validity since few additions w ere made to the tentative instrument com prising evaluation criteria for all aspects of research reports.

*
Do a survey amongst lecturers of nursing research to determine the face validity of the evaluation criteria included in the instrument the relative w eight or relative value of each o f the aspects of a research report • C o m p ile th e W O R C A T ( W e ig h te d O b je c tiv e R e s e a r c h C r it e r i a A s s e s s m e n t T o o l) by a d ju s tin g th e tentative in stru m en t acc o rd in g to the findings

A
FIGURE 1. MATRIX: RELATIVE WEIGHTS OF ASPECTS OFA RESEARCH REPORT.
lecturing and evaluation o f research reports as well as number o f published papers.Since the Aspects Title Limitations Conclusions instument was com piled from resources and w as se n t to c o lle a g u e s to o b ta in th e ir evaluation o f the contents of the instrum ent (Burns and Grove 1987; Polit and llu n g ler 1983; Wilson 1983).
FIGURE 2. EXAMPLE OF WEIGHTING SOME OF THE ASPECTSOF A RESEARCH REPORT.
O f the experienced group 71 % have d o a o ral d e g re e s c o m p a re d to th e 25 % o f th e in experienced group.R espondents in the experienced group have on average 4 more years o f lecturing experience, evaluated 85 more research reports and published 3,6 more reports than respondents in the inexperienced group.Aspects o f a report that should be evaluated as w e ll a s th e ir a s s e s s m e n t c rite r ia w e re identified from literature (Burns & G rove 1987; Polit & Hungler 1983; W ilson 1983).These aspects were limited to 18 since the com puter programme used for determi ni ng the r e l a ti v e w e ig h ts c o u ld a c c o m o d a te a maximum o f 18 different aspects.The aspects and the assessm ent criteria for each are listed in F ig u re 3. T h e c rite ria added by the respondents to those identified from literature were few and are indicated with asterisks (♦).
The relative w eights allocated to different aspects o f a research report em phasized the importance o f identifying a relevant research problem clearly and m aking appropriate and valid recom mendations to solve the problem.O p in io n s o f the re sp o n d e n ts on how to evaluate relationships betw een aspects o f a research report varied w idely and neither of the tw o can be recom m ended.R ecom m endations Further research should be done to determine -th e v a lid ity a n d re lia b ility o f the W O RCA T -w ays and m eans o f deal i ng wi th relations betw een aspects o f a research report.REFERENCES BECK, CT.The Research Critique.General Criteria for Evaluating a R esearch Report.JO G N N 19:1 (1990) 18-22.B IN D E R , D M .( 1 9 8 1 ) : C r itiq u e : experimental study, in SD Kram pitz and N Pavlovich (Eds) R ea d in g s fo r N ursing Research, St Louis:C V M osby.BURN S, N A ND G RO VE, SK. (1987).The P r a c t i c e o f N u r s i n g R e s e a r c h Philadelphia: W B Saunders.D UFFY , ME. (1985).A Research appraisal checklist for evaluating research reports.N ursing and H ealth Care.539-546.L E IN IN G E R , M. (1968).The research c r i t i q u e , f u n c t io n a n d a r t .N ursing Research.17, 444-449.PO LIT, D A N D H U N G LER , R. (1983): N ursing R esearchP rinciplesandM ethods.Philadelphia : JB Lippincot.W ARD, M AND FETLER, M. (1979): W hat guidelines should be follow ed in critically e v a lu a tin g re se a rc h re p o rts?Nursing Research.28.120-125.W ILSON, HS. (1983): R esearch in Nursing.C alifo rn ia: Addison-W esley.