The Use of Experts and Their Judgments in Nursing Research: an Overview

Curationis 31(4): 57-61 Experts and their judgments are widely used in the fields of research, education, health care, law, commerce and technology. Expert judgment is known for its subjectivity and its potential for bias, which brings into question the accuracy and authenticity of judgmental data. At the same time there is acknowledgment of the valued contribution of judgmental data towards valid inferences in research and education. Maximizing the use o f experts and their judgments has therefore become an endeavour of educationists and researchers alike. Since this is not a research article its purpose is to guide and assist nurse researchers with important methodological and ethical decisions when using experts. Experts must be used in the context o f appropriate research methods such as the Delphi and Nominal Group techniques. Sampling of experts and sample size is determined by the type and quality of data and the availability of population data; purposive and maximum variation sampling techniques are recommended as appropriate when sampling experts. Universal research ethics must be applied with particular consideration of aspects which may influence the truth value o f consensus among experts and marginalization of minority or extreme viewpoints. Quantification of judgmental data is recommended and is important to minimize bias and to increase the authenticity of research findings. The content includes: design considerations when using experts, sampling issues, ethical rules to be considered when enlisting experts and their judgments, optimal data collection approaches and managing judgmental data.

Since this is not a research article its purpose is to guide and assist nurse researchers with important methodological and ethical decisions when using experts.Experts must be used in the context o f appropriate research methods such as the Delphi and Nominal Group techniques.Sampling of experts and sample size is determined by the type and quality of data and the availability of population data; purposive and maximum variation sampling techniques are recommended as appropriate when sampling experts.Universal research ethics must be applied with particular consideration of aspects which may influence the truth value o f consensus among experts and marginalization of minority or extreme viewpoints.Quantification of judgmental data is recommended and is important to minimize bias and to increase the authenticity of research findings.
The content includes: design considerations when using experts, sampling issues, ethical rules to be considered when enlisting experts and their judgments, optimal data collection approaches and managing judgmental data.

Introduction
The use of expert judgments has become increasingly popular in the fields of research, education, health care, law, commerce and technology.Experts are d efined as inform ed persons in a particular discipline or field, usually constituted as a panel, as individual m em bers or both (Burns & Grove, 2005:400).In this case the field is nursing research, defined as systematic enquiry designed to develop knowledge in the nursing profession (P olit & Beck, 2004:726).Experts are used mainly, but not exclusively, for the purpose o f judging the content-related evidence of ev alu atio n instrum ents and data collection schedules in education and research.Judging refers to experts giving th eir pro fessio n al opinion about desirable outcomes (Gomm, Needham & Bullman, 2000:81), During this (judging) process experts are asked how well each o f the concepts or item s has been represented in the instrument (Bums & Grove, 2005: 401) or the relative value (weight) of items.
E xpert ju d g m en t is know n for its subjectivity and its potential for bias, w hich brings into the question the accuracy and authenticity of judgmental data.At the sam e tim e there is acknow ledgm ent o f the valued contribution of judgmental data towards valid in ferences in research and education.Maximizing the use of experts and th eir ju d g m en ts has therefore become an endeavour of educationists and researchers alike.This paper discusses the best measures for selecting experts, for designing and implementing m ethods to co llect and m anage judgmental data and outlines the ethical considerations when using experts in research.Since this is not a research article its purpose is to guide and assist nurse research ers w ith im portant methodological and ethical decisions when using experts.

Literature Review
Judgment by experts has been used in research studies in a range o f diverse fields.Apart from nursing other fields of use include medical education (Downing, Tekian, & Yudkowsky, 2006:51), the military (Crawford & Williams, 1985:387) and in psychology and social science research (Miranda, 2001:87).In most of these fields problems have surfaced in both acquiring and managing judgmental data (Crawford & Williams, 1985:388).Additionally, there is mounting criticism that expert ju d g m en ts are less tran sp a ren t, biased tow ards overestimating the value o f an item/ concept and naturally so, subjective.In this regard authors (Crawford & Williams, 1985:387;Miranda, 2001:88) refer to the term "subjective judgment".As a result o f these problem s and criticism s researchers have made considerable strides towards the quantitative analysis o f subjective data to minimize inherent biases and to improve its accuracy.However, deciding when to use experts and who these experts would be are im portant precursors to generating judgmental data from experts.

Who is an "expert"?
Apart from being described as being knowledgeable or an informed person in a particular field or discipline there are no specific directives in the literature about the defining characteristics of an expert.The decision about who qualifies to be an expert and who will be selected as such, lies mainly with the researcher.It is th erefo re im portant that the researcher examines his/her own values and assumptions about what knowledge is, how it is acquired and developed, and w hat then does it m ean to be "knowledgeable Benner (1984:2) refers to the work of philosophers of science such as Kuhn and Polanyi in describing "knowing how" and "knowing th a t" and posits that m any o f our skills (know ing how) are learnt w ithout scientific formulations of "knowing that".
In a practice discipline such as nursing and as a consequence in nursing research, know ledge em bedded in practice becomes important for deciding who is an expert.If one uses Benner's (1984) seminal work "From novice to e x p e r t", expertise develops when someone tests and refines propositions, hypotheses and principles in actual practice situations; an expert then no longer relies on rules/principles for judgment or decision-making.
Experience is therefore a requisite for expertise (Benner, 1984: 3); the richness of experience leads an expert to have intuitive knowledge that arises from a deep understanding of the total situation (Benner, 1984: 32).Decisions around experience would thus be informed by how active the person is in the field and the nature of activity, or what the breadth/ scope of the experience is.Deciding who would qualify as an expert also depends on the purpose the data should serve.

When to use experts
Experts have been used for various purposes in a range of fields.In research and education experts have been used consistently for the design, development and testing of instruments to collect data.In particular, expert judgments have been used and quantified to determine content and construct validity and to assign component weights to assessment items that are assum ed to be o f equal importance or weighting.Through consensus building, experts may also be used to develop research agendas for determ ining research priorities in a particular field (Nathens, Cook& Machiedo, 2006:101).Developing research agendas in this way is p artic u la rly useful in research environm ents challenged by lack o f funding or where funds are best spent on ad dressing research questions already decided upon by experts and have been ranked through expert consensus.Experts are also useful for idea creation to im prove on the quality o f ideas generated and for idea building (Benjamin, Archibold & Suarez, 2004:1).In idea building it is the interaction between experts, constituted as a panel, that adds richness to the process and hence the data generated.Selecting some experts for their subject expertise and others for their creativity is an important sampling consideration (Benjamin et al., 2004:2).In health care, business and economics experts are increasingly being called upon to evaluate new technologies (Benjamin et al., 2004:2) and policies and to recom m end actions on m atters o f im portance.Panels o f experts and individual experts have been found useful when historical data are limited or where ethical and moral issues need to be teased out in a broad or complex problem (Yousuf, 2007:6).In management practice, experts play an important role in evaluation and selection processes and to provide future directions in planning.

Design and methods when using experts
Within both quantitative and qualitative approaches there are several research designs and methods that require the use o f experts as a data source.The most commonly proposed methods are the Delphi technique, the Nominal Group Technique and surveys (Bums & Grove, 2005:436;Polit & Beck, 2004:238;Steward, 2001:299;Yousuf, 2007:2).Surveys are used in the broadest sense to describe the overall design of a study using experts and not as a specific method to collect judgmental data.
The Delphi technique is essentially a group process whereby the opinions and judgm ents o f a group o f identified experts are elicited on a specified topic usually through a series o f questions (Yousuf, 2007:3).It may be used to gain expert consensus, called norm ative D elphi and to establish norm s and standards by giving structure to a set of properties (Yousuf, 2007:3).This is in contrast to a policy or decision Delphi, which seeks to elicit divergent responses from experts.
The Nominal Group Technique (NGT) is similar to Delphi, except that the NGT re q u ires the ph y sical p resen ce o f members in the form of an expert panel.The researcher would normally act as facilitator of the group process.The NGT is primarily a consensus building tool that assists to rank or prioritize issues or items.When deciding to use experts and their judgments the researcher must be clear about the purpose the data should serve.Knowing the purpose guides the research er to selecting the sam ple, choosing an appropriate data collection method and hence, data management strategies.

Sampling considerations
Sampling o f experts must be guided by the type and quality of judgmental data to be obtained.Purposive sampling, also known as judgmental sampling, has been w idely supported as an appropriate method of sample selection especially in qualitative research and sometimes in quantitative research (Bums & Grove, 2007:331).Purposive sampling is broadly defined as "Selectingparticipants based on personal judgm ent about which ones w ill be m o re r e p r e s e n ta tiv e o r informative... " (Polit & Beck, 2004:729).However, it is difficult to evaluate the accuracy o f the researcher's judgment (Bums & Grove, 2007:345); purposive sam pling therefore continues to be criticized for the lack o f methods in support o f the representativeness or typicalness o f the sam ple selected.P urposeful selectio n req u ires the identification of experts in disciplines or dom ains d irec tly and in d irec tly represented in the research instrument or topic under discussion, and who could provide rich data as evidence for the content-related validity of the instrument.Ideally, these disciplines or domains must also be sampled to ensure richness of judgm ental data.N esting purposive sam pling within maximum variation sam pling is one way to obtain representativeness and rich data by including a wide range of extremes (List, 2004:1;Patton, 2002:234).
Maximum variation or heterogeneity sampling is described as a special kind o f purposive sampling (List, 2004:2), which may be used to identify experts or cases (in qualitative research) to provide rich information (Patton, 2002:234).This sampling method aims to identify themes or patterns that run through a range of variations (Patton, 2002:234); it is informed by the logic that any common themes that emerge from great variation are valuab le in capturing the core experiences of a setting or phenomenon.In q u an titativ e term s this type o f sampling is also an extension o f the statistical principle o f regression towards the mean, which means that in a group of experts who is extreme in several different ways one is likely to find some who are average in other ways (List, 2004:2).Thus maximum variation sampling limits the possibility o f selecting a narrow sample (few cases) from one with wide variation.A ccording to Y ousuf (2007:4) heterogeneity o f participants must be preserved to arrive at valid results.When experts are similar they are more likely to agree than to disagree resulting in artificial consensus.In a group or panel situation domination by number or by personality may lead to the "bandwagon effect " (Linstone & T uroff, 1975:4).When to use maxim um variation sampling depends on the availability of population data and on sample size.

Sample size considerations
Sample size is determined by the study design (Yousuf, 2007:6) and the aims and purpose o f the experts' undertaking, which in turn inform s the sampling method.If consensus among experts is the aim for example in validation studies a sample o f between 5-10 experts is recommended (Bums & Grove, 2005:400).In areas where expertise is limited a m inim um o f three exp erts is recommended, which will require 100% agreement between them if a consensus Delphi is used.The lower the number of experts the greater the demand for 100% agreem ent; more experts allows for g re ater v aria tio n in th eir level o f agreement and hence, their judgment.The level o f agreement must be set by the researcher prior to implementation; gen erally 75-80% is the low est acceptable level o f agreement.
M axim um v aria tio n sam pling is appropriate for consensus groups o f approximately three within samples of 15-20 and for samples o f 20-50 for surveys of expert opinions (List, 2004:2).This type o f sampling is particularly useful if the target population from which experts may be drawn is not clearly defined or there a great v aria tio n in the dom ain or phenomenon to be studied.

Ways to collect data from experts
A fter identifying and selecting the experts the study design decided upon is followed to collect data from experts' judgments.To ensure good data quality it is important firstly, to provide necessary information about the process and give clear instructions to experts usually by way of an information letter or a meeting with them.Secondly and if applicable, decide on and include the scale to capture judgmental data for example: Likert scale, Visual Analogue Scale, Graphs; tracings and video recordings; this includes clear criteria or directives on how to apply the scale.Thirdly, if consensus is required decide on the level o f agreement between experts.Lastly, build in suitable time frames according to the study design and practicality, and to get commitment from experts.Within

Managing judgmental data
Quantifying judgmental data is a process that has become increasingly important to m ake optim al use o f e x p e rts' judgments.To begin this process the researcher must choose a suitable data management system e.g.Excel spread sheet to prepare judgm ental data for an aly sis.Some procedures for quantifying judgmental data sets are describ ed below.These include: calculating the Content Validity Index (Polit & Beck, 2004:423) and correlation coefficients, applying the Judgmental Policy Capturing procedure (Harvill, Lang & McCord, 2004:1) and the Subjective Judgment Model (Crawford & Williams, 1985:387) to experts' judgments.
The quality of measuring a particular construct is dependent on the degree to w hich the m easurem ent m ethod or instrum ent includes all the relevant elements o f that construct (Burns & Grove, 2005:400); that means determining the content-related validity for example, through the use o f expert judgm ent.Although there are several sources of evidence for content-related validity, calculating the content validity index (CVI) is a way to quantify experts' judgments.Using a 4-point Likert scale (Polit & Beck, 2004:423) experts rate the content relevance of each item and/or sub-item .Judgm ent quan tificatio n in volves firstly, determ ining the proportion of experts who must agree for content validity to be established and secondly, calculating the CVI.The actual CVI is the proportion o f items that received a rating of 3 or 4 by the experts; a CVI of .80 or more indicates high content validity (Polit & Beck, 2004:423).
For consensus data sets such as the above, Pearson's correlation coefficient (a statistic that expresses the size and direction of an association between two v aria b les) m ay be used to assess statistical relations between variables.For example between the mean score of each item/question and the percentage o f ex p erts who scored an item as important/relevant or very important/ very relevant, that is, at the higher end of the scale (Nathens et al., 2006:107).A high correlation coefficient (closer to +1 or -1) is desirable and rules out, to a large extent, the possibility of chance agreement.In ranked or prioritized items, increasing mean scores and decreasing variability in scoring within each item, as expressed by the Standard Deviation (SD), usually suggest a high degree of consensus for highest ranked items.
Judgmental Policy Capturing (JPC) is a procedure to statistically describe the inform ation processing strategies of experts (Harvill et al., 2004:1).During this process experts are required to review, independently, the com ponents o f a complex, multidimensional assessment (not for individual OSCE procedures or "test item s") and decide which of the components are more important than others (Harvill et al., 2004:2).By applying multiple regression analyses, appropriate weights are determined for components in an assessment.S ubjective Judgm ent Model (SJM ) enables subjective, pair-wise comparison of items by experts whose judgments are captured on visual analogue scales ( 100mm in length) or a judgment matrix.Usually data from visual analogue scales are based on an expert's educated guess on one variable; using the SJM requires the expert to judge the importance of one variable in relation to another, for example verbal expression vs. non-verbal cues (Figure 1).Mathematical procedures are applied to determine the relative weights of the variables or items.In contrast to the above (JPC) this method is useful for less complex assessments or individual instruments.

Ethical considerations when using experts
When experts are invited to participate in research, their informed consent, written or verbal, should be sought.To be truly informed, all relevant facts must be disclosed and understood by the potential participant and a decision made to voluntarily contribute to the study arrived at without undue coercion.All relevant facts include the purpose, risks, benefits and social im plications o f participation.The latter is of particular importance.Where the results o f the research co n flic t w ith estab lish ed authority or policy and the distinctive influence of the expert can be detected, or his or her identity surmised, then the p rin cip les o f co n fid e n tia lity and anonymity might be jeopardized and the person's good name might be put at risk.This is especially relevant where expertise is limited and only one or two experts are able to be accessed or where experts are well known in a discipline or geographical location.
W here p a rtic ip a tio n w ill involve membership in a group, the invitee has the right to know o f the other members of the group.This is to ensure privacythe control by the individual over the extent, timing and circum stances o f sharing oneself with others (Rischbieth & Blythe, 2005:49).The right to refuse particip atio n , to decline to answ er questions posed or to withdraw at any stage of the process without any penalty or consequence must be assured prior to eliciting participation.
It is important to ensure confidentiality o f data containing opinions, survey results or interviews, which experts will be req u ested to com m ent on.All inform ation which may lead to the identification o f primary subjects and sites or information, which might pose a threat to personal, occupational, legal or ethical rights, must be removed.This is of particular import where the topic under investigation could be considered illegal, immoral or controversial, for example termination o f pregnancy, prostitution, child abuse and illegal immigration.Where the data for expert opinion consist of instruments or scales, which have been modified or adapted, the experts must be aware that the original holders of the copyright -author/s or institutionshave given permission for their property to be used.
The concomitant principle of anonymity is germane here.The expert informant must be aware that, having earned the appellation "expert ", it is possible that th e ir id en tity m ight be presum ed, particularly if expertise in the area is limited or when their views are well known, controversial or contradict the received view.In the case of Nominal Group Technique or a group interaction Delphi the originator o f an opinion, com m ents or ratin g m ust not be identifiable.This may be achieved by summarizing anonymously the experts' opinions and justification given for these statements.If the names of the experts, as individuals or as groups, are to be included in the final report, then their co n sent m ust be given (C olton & Hatchet, 2004:2).
Using experts for consensus building raises sev eral eth ical concerns.C onsensus reached may not be true consensus and may be the result of misleading or manipulated consensus (Yousuf, 2007:4).This leads to data that do not contain best judgments but rather a compromise position.Trustworthiness may be jeopardized when compromise is reached or consensus is manipulated or subtly coerced by powerful or persuasive group m em bers.N ot exploring disagreement or marginalizing dissenting voices may also generate artificial consensus.U sing expert groups to achieve consensus on controversial topics or m orally divisive topics is generally deemed unethical.However, the broad and immediate context from which data are collected must be bom in mind; what is contentious, irreligious or morally divisive in one context might not be so in another.

Conclusion
In clu sio n o f m easu res to enhance objectivity o f experts' judgments and hence the authenticity o f judgmental d ata has becom e m andatory in all scientific research.This implies that researchers must have sound reasons for using experts and follow scientific procedures to make optimal use of expert data.Deciding who is an expert requires careful examination o f the values and assum ptions un d erp in n in g the re s e a rc h e r's co n c ep tu alizatio n s o f know ledge and the ac q u isitio n o f knowledge.Universal ethical principles m ust be o b served w ith p artic u la r referen ce to ethical co m p lex ities associated with privacy, confidentiality, anonymity and the nature o f consensusbuilding.The information obtained from a research study is only as good as the experts who participate in it, and the study design followed; let not poor design and methods adversely influence the q u ality o f ex p erts and th eir judgments.

Figure 1 -
Figure 1 -Example of use of a Visual Analogue Scale

BENNER, P 1984 :
From Novice to Expert: Excellence and Power in Clinical Nursing Practice.Addison-Wesley: California.BENJAMIN, CO; ARCHIBOLD, E & SUAREZ, T 2004: Using expert panels to evaluate new technology.Journal of the A cadem y o f B usiness and Technology, http://findarticles.eom/p/ articles/mi mOOGT/is (Accessed 7 March 2007).BURNS, N & GROVE, SK 2005: The Practice of Nursing Research Conduct, Critique and Utilization.W B Saunders Co: New York.BURNS, N & G RO VE, SK 2007: U n d erstan ding N ursing R esearch, building an evidence-based practice.Saunders Elsevier: St Louis.CHEEK, J & JONES, J 2003: What nurses say they do and need: implications for the educational preparation of nurses.Nurse Education Today.23:40-50.COLTON, S & HATCHET, T 2004: The development o f a research instrument to analyze the application of adult learning principles to online learning, http:// eric.ed.gov/ERICebPortal/Home (Accessed 7 March 2007).CRAWFORD, G & WILLIAMS, C 1985: A note on the an aly sis o f Subjective Judgment Matrices.Journal of Mathematical Psychology.29,387-405.DOW NING , SM; T E K IA N , A & YUDKOWSKY, R 2006: Procedures for E stab lish in g D efen sib le A bsolute P assing Scores on P erform ance Exam inations in Health Professions Education.Teaching and Learning in Medicine.18(l):50-57.p h ical and m ethod o lo g ical foundations of Delphi.Addison-Wesley Publishing Company: Massachusetts.LIST, D 2004: M aximum variation sampling for surveys and consensus groups.Adelaide: Audience Dialogue.Available at www.audiencedialogue.org/maxvar.html(Accessed 3 July 2007).M IR A N D A , E 2001: Im proving S ubjective E stim ates using Paired Comparisons.IEEE Software.Ericsson Research C anada.January/February 20012002: Q u alitativ e Research and Evaluation Methods.Sage: Thousand Oaks, California POLIT, DF & BECK, CT 2004: Nursing R esearch P rin cip les and M ethods.L ippincott W illiam s & W ilkins: Philadelphia.Using Nominal Group Technique to Explore Competence in O ccupational T herapy and Physiotherapy Students during Firstyear Placem ents.B ritish Journal of Occupational Therapy.64(6):298-304.YOUSUF, MI 2007: Using E x p e rts' opin io n through Delphi Technique.Practical A ssessm ent.R esearch and Evaluation 12(4): 1-8.