About the Author(s)

Princess N. Msibi Email symbol
Department of Nursing Science, University of Pretoria, South Africa

Ramadimetja Mogale symbol
Department of Nursing Science, University of Pretoria, South Africa

Maretha de Waal symbol
Department of Nursing Science, University of Pretoria, South Africa

Nomathemba Ngcobo symbol
Department of Nursing Science, University of Pretoria, South Africa


Msibi, P.N., Mogale, R., De Waal, M. & Ngcobo, N., 2018, ‘Using e-Delphi to formulate and appraise the guidelines for women’s health concerns at a coal mine: A case study’, Curationis 41(1), a1934. https://doi.org/10.4102/curationis.v41i1.1934

Original Research

Using e-Delphi to formulate and appraise the guidelines for women’s health concerns at a coal mine: A case study

Princess N. Msibi, Ramadimetja Mogale, Maretha de Waal, Nomathemba Ngcobo

Received: 01 Mar. 2018; Accepted: 08 Aug. 2018; Published: 04 Oct. 2018

Copyright: © 2018. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.


Background: E-Delphi is an online method widely used in health and social research to strengthen decision-making processes and reach consensus on developing guidelines for health services.

Objectives: The e-Delphi technique was designed to formulate and appraise a set of criterion-referenced guidelines for women’s health concerns of mineworkers at a selected coal mine in Mpumalanga, South Africa.

Method: The University Learning Management System’s discussion forum was customised to suit the characteristics of e-Delphi as the second phase of a primary study on the formulation of guidelines for women’s health concerns. Six purposively sampled experts with extensive experience in Occupational Health and in Women’s Health participated. Online engagements on the formulation and appraisal of the guidelines for women’s health concerns took 7 weeks, divided into four phases as preparatory, exploratory, consensus and refinement. From the experts’ inputs, guidelines were drafted. Experts were invited to evaluate the guidelines by using a 7-point Likert scale with AGREE II criteria. Consensus was reached in two e-rounds.

Results: Eight guidelines were formulated, appraised and adopted as: change management, control of hazardous environments, suitable psychosocial working environment, provision of health care service, uphold human dignity and adherence to human rights, effective measures for safety participation compliance, accessible, available and relevant on-site health care services and hope and resilience. Each guideline has rationale, operational strategies and anticipated outcomes.

Conclusion: E-Delphi platform used various tools to deliberate on the process of guidelines formulation and appraisal. The platform was convenient for the experts’ participation in the discussion at any time and anywhere.


Techniques such as traditional Delphi, nominal group discussions and, recently, e-Delphi are used to reach consensus on issues related to nursing and health practices (Graefe & Armstrong 2010; WHO 2010). Unlike the traditional Delphi technique, which is commonly used as a formal consensus method with two or more rounds of face-to-face interactions, the e-Delphi assembles ideas online with experts communicating and engaging with each other at their own time at their vicinities and either in an asynchronous or anonymous manner (Thangaratinam & Redman 2005). The e-Delphi studies create opportunities for researchers, including nurse researchers, to conduct research nationally and internationally, provided the researchers carefully consider such designs and methods as part of data collection. E-research methodologies, such as the e-Delphi technique, have yet to undergo significant critical discussion in respect of possible risk of bias and applicability (Toronto 2017), as not all evidence is the same. Evidence in research is described in accordance with how it is situated in the evidence pyramid (Davidson & Candy 2016). Evidence pyramids represent a hierarchy of internal validity (risk of bias) or both internal and external validity (applicability) (Murad et al. 2016) of knowledge.

This article has four sections. Firstly, the introduction discusses why the lower sources of evidence in the hierarchy are preferred in nursing practice. Secondly, the article presents information about the e-Delphi as a technique in reaching consensus. Thirdly, the article expounds on the entire process that was followed to design an e-Delphi platform through the University Learning Management System (LMS) and the processes that were followed through online engagements to formulate and appraise the guidelines. The article ends with two examples of what the formulated and appraised guidelines look like.

The evidence from expert committee reports or opinions and/or clinical experience of respected authorities falls within Level IV in the evidence pyramid. Most experts agree that the higher up the hierarchy the study design is positioned, the more rigorous the methodology (Davidson & Candy 2016). The study design can minimise the effect of bias on the results of the study. Study design, together with methodological limitations of a study, imprecision, inconsistency and indirectness are additional factors that can affect the quality of evidence (Van der Linde et al. 2005:693). That said, lower sources of evidence in the hierarchy are least preferred in nursing practice because they require extensive expertise and time for identification, appraisal and application. In nursing, evidence-based nursing is a way of providing nursing care that is guided by the integration of the best available scientific knowledge together with nursing expertise.

Nursing researchers interested in harnessing expert knowledge on a broad range of health and well-being topics are increasingly using Delphi as a means of capturing collective opinion (Brüggen 2009). The method has been adopted across a wide range of health, social care and well-being studies related to policy, clinical practice, planning and evaluation (Brüggen 2009). Additionally, Delphi techniques are also used to reach consensus in the development of guidelines (Lindell & Demi 2018). While the use of Delphi is increasing in health and well-being research, less attention has been given to the methodological underpinnings of the Delphi method, especially the e-Delphi technique as a newly emerging e-research method for reaching consensus. Likewise, Internet-based research is growing rapidly in popularity; however, it has yet to undergo critical methodological review (Toronto 2017). Despite efforts to capitalise on the potential of information technology for research and practice in health, inadequate attention has been paid to how knowledge from these new endeavours can be validated. This is because some e-research methods are commonly used to reach a consensus in the absence of research evidence (Toronto 2017).


Delphi is a technique used to achieve a common viewpoint from experts using questionnaires to gather information of interest (Lindell & Demi 2018). This technique has been extensively used within health and social research to strengthen decision-making processes and reach consensus. According to Green (2014), e-Delphi is defined as a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with multiple challenges. The e-Delphi method allows experts to communicate and engage with each other online in their own time in their vicinities to solve problems until consensus is reached. According to Bardhan, Ngeru & Pitts (2012), the e-Delphi technique in this era is important in evidence-based research as it enables participants to post their opinions and accrue their ideas online.

With the use of the e-Delphi technique, experts participate in different periods of time, at their own pace (Meshkat et al. 2014). Other researchers (Mamaqi, Miguel & Olave 2010) have agreed recently that the use of e-Delphi is ideal; for one thing, it is anonymous because neither the researcher nor the experts was physically present, which might influence the communication and lead to prejudice. Such bias may further lead to insufficient and incorrect data collection and lack of evidence, which could in turn lead to inability to reach consensus in the guideline formulation. In the e-Delphi process, reaching consensus by experts cannot be resolved in a once-off discussion. As an iterative process, e-Delphi involves a chance for initial feedback, collation of feedback and distribution of collated feedback to participants for further review. Depending on the number of research questions and available time to reach consensus, the e-Delphi process includes three rounds, to prevent exhaustion and attenuation (Green 2014). Furthermore, e-Delphi as a method strengthens the diversified co-learning through a shifted paradigm in a guideline formulation arena. In addition, some of the common biases that usually occur in a face-to-face group process are removed (Eubank et al. 2016).


Designing the e-Delphi

An e-Delphi platform was designed as the second phase of a primary study on the formulation of guidelines for the health concerns of the women mineworkers at the selected coal mine.

Context of the study

A discussion board from the LMS was customised to meet the requirements of an e-Delphi (Chou 2002). According to Chou (2002) these requirements are the two online interfaces for the project leader and the experts. The interface for the project leader is used for the development and sending of questionnaires to the experts. The interface for the experts is to respond and give inputs to the project leader. Additionally, the platform should allow for other forms of multimedia communication, for example self-recorded podcasts in this case. Of importance, the platform should permit the project leader to totalise and analyse the responses of the experts easily (Brüggen 2009). This e-Delphi had the following features: login page, contact page, landing page, discussion forum, file upload area, communication tools (announcements and email), user guide (for external experts on how to access the LMS and use the system and tools), YouTube link for podcasts (because of the limited space in the LMS to upload large file sizes, podcasts were uploaded to YouTube as unlisted - not searchable - videos, then a link was made available to the LMS) and a Likert scale. See Figure 1 for some of the features of the e-Delphi.

FIGURE 1: Some features of the e-Delphi.

Sampling criteria

E-Delphi employs ‘experts’ as panel members (Taylor et al. 2016:2). The experts who participated in this study were selected through purposive sampling. Only 6 experts out of the initial 12 provided the written informed consent to be part of the online discussions, though. A formal written request in accordance with the university’s policy was sent to the LMS administrator to provide the experts with online access, as they were not affiliated to the university. Not only were the experts from the mining and factory industries, but some were from municipality and government entities such as Transnet and Eskom. All of them had extensive experience in Occupational Health and some in Women’s Health. To enhance the rigour of the guidelines, an external expert was also selected purposively for expertise in guideline development within Occupational Health and Safety. See Table 1 for the characterisation of the experts.

TABLE 1: Characterisation of the experts.

Five of the experts who participated in the e-Delphi were registered nurses, three had a bachelor’s of technology in Occupational Health Nursing, while the other three were PhD prepared scholars. The experts’ years of experience ranged from 15 to 25 years in Occupational Health and Women’s Health and Issues.

Online engagements

The online engagements were collected through the discussion forums. These forums were used asynchronously at the users’ own time and pace, with the duration of rounds clearly highlighted for experts. The forums ran for a period of 7 weeks in succession. The 7 weeks were divided into four phases: preparatory, exploration, consensus, with refinement as the last phase.

Phase 1: Preparatory phase

In this phase both the instructional designer and the system administrator of the LMS played an important role in the preparation and then design of the e-Delphi. As a registered student, the researcher had free access to the LMS. The experts were provided with individual official login details by the system administrator. The user manual on the steps to log in on the e-Delphi platform was developed by an instructional designer. This user manual included the use of particular tools of the LMS. The researcher was trained as the moderator on how to use the platform and self-recording of the podcasts by the department’s instructional designer. The first podcast outlined the processes that the experts were to follow during the entire period of the e-Delphi. The researcher tapped the socio-economic model (SEM) to identify eight provisional statements to be considered in the formulation of the guidelines from the findings of the main study. This phase took 1 week.

Phase 2: Exploratory phase

The aim of this phase was to brainstorm the purpose and scope of the guidelines. The provisional statements, which were optimised with the self-recorded podcasts, were posted on the e-Delphi platform. Hands-on support was provided to the experts through email messages and telephonic support throughout the project period. This phase lasted for 4 weeks. The experts were required to provide inputs and suggestions on the content of the statements, which were to be guidelines for the underground women’s health concerns in the selected coal mine. Following are some of the examples of such inputs and comments on the statements (see Box 1).

Box 1: Expert inputs and suggestions.
Phase 3: Consensus phase

From the inputs and comments of the experts in the exploration phase (Box 1) via the e-Delphi the researcher drafted eight guidelines. A seven-point Likert scale using the AGREE II tool was developed. The AGREE II tool evaluates the methodological rigour and transparency in guideline formulation (MacDermid et al. 2005). The tool has six domains, with Domain 1 as the scope and purpose, Domain 2 being stakeholder involvement, Domain 3 rigour of development, Domain 4 clarity of presentation, Domain 5 applicability and Domain 6 about editorial independence (MacDermid et al. 2005). The overall evaluation required the AGREE II user to make a judgement as to the quality of the guideline, taking into account the appraisal items on a seven-point Likert scale (Brouwers et al. 2010). A score of 1 is given when no information was relevant to the item or if the concept was very poorly reported. A score of 7 was given if the quality of reporting was exceptional and where the full criteria and considerations articulated had been met. A score between 2 and 6 was assigned when the reporting of the item did not meet the full criteria or considerations. In addition, a score was assigned depending on the completeness and quality of reporting (MacDermid et al. 2005). The score increased when more criteria were met, and the aspect or determinant was addressed (MacDermid et al. 2005).

Likewise, the process was followed in this e-Delphi technique and only two e-rounds were used to reach consensus. In the first e-round, the experts evaluated the guidelines with different scorings ranging from 4 to 7 together with the comments and concerns. Those with 7 as a score were not sent again to the experts. Refer to Figure 2 as an example of the rated scale from the e-round. The project leader incorporated the comments in the guidelines and posted them for evaluation for the second round. On the second e-round, the experts evaluated the refined guidelines and gave a score of 7, and consensus was reached.

FIGURE 2: An example of the rated scale from the e-round.

Phase 4: Refinement of the guidelines

With the drafted guidelines from the two e-rounds, the project leader solicited advice from an external expert. The external expert provided constructive comments on how to rearrange and reconstruct the guidelines. From the external expert’s comments, the project leader refined the drafted guidelines, finalised and presented them to the experts, who then adopted the guidelines.

Measures to ensure trustworthiness

The researcher ensured credibility by the ongoing iteration and feedback from and to the experts, which according to Engles and Kennedy (2007) is viewed as member checks. To achieve dependability, the researcher included a range of experts (Cornick 2006) from mining and industry as well as from the municipality. Confirmability was assessed by maintaining a detailed description of the e-Delphi collection and analysis processes. Transferability was established using verification of the applicability of e-Delphi findings.

Ethical considerations

The study obtained ethical clearance from the Faculty of Health Sciences ethics committee. (Ethics Reference number 206/2015). During the selection phase, the information leaflet which had the relevant information about the e-Delphi process and informed consent was emailed to the experts as indicated under the sampling criteria. Participation in the e-Delphi was voluntary, anonymous and confidential (Thangaratinam & Redman 2005).

Results and discussions

Eight guidelines were formulated to address women’s health concerns in the selected coal mine. These were: change management and supportive organisational environment, control of hazardous environments, creation of effective and suitable psychosocial working environment and provision of health care service where women can express their health concerns upholding of human dignity and adherence to a human rights framework, effective measures for safety participation compliance, accessible, available and relevant on-site health care services, hope and resilience. Each guideline has a rationale, operational strategies and anticipated outcomes, as shown in the following two examples (see Box 2 and Box 3).

Box 2: Creating effective measures to enforce change management and a supportive organisational environment.
Box 3: Create effective measures for controlling hazardous environments that affect women mineworkers’ health.

Implications and conclusions

The article highlighted how positive deviance was used to leverage the LMS to design an e-Delphi. The process followed to design the platform may offer scholars unique insights on how to incorporate such techniques into future studies. Nurses are encouraged to utilise their knowledge and skills to creatively generate and develop innovative ways of conducting research as potent technologies are being developed with the potential to transform society.

The e-Delphi technique has proved to be a resourceful, systematic and cost-effective method of accelerating knowledge generation in nursing sciences and research. The platform provided a space to reach consensus on the formulation and appraisal of the guidelines. Various e-tools were used to deliberate on the process of formulation and appraisal of these guidelines. The platform was convenient for the expert as they could participate in the discussion any time and anywhere.


The authors acknowledge the study participants – women mineworkers from the selected coal mine, professional nurses working at the Occupational Health and Safety Centre and mine management working at the selected coal mine in Mpumalanga, South Africa.

Competing interests

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

Authors’ contributions

P.N.M. conducted the research for her thesis and wrote the draft manuscript. R.M. was the supervisor and M.d.W. was the co-supervisor. N.N. provided professional support with the online system and contributed to the draft of the article.


Financial support for the PhD study was received from the Democratic Nursing Organisation of South Africa.


Bardhan, T., Ngeru, J. & Pitts, R., 2012, ‘A Delphi-multi-criteria decision-making approach in the selection of an enterprise-wide integration strategy’, Journal Proceedings of the 2nd International Conference on Information Management and Evaluation, 27 April 2011, Ryerson University, Toronto, Canada, pp. 24–37.

Brouwers, M.C., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G. et al., 2010, ‘AGREE II: Advancing guideline development, reporting and evaluation in health care’, Canadian Medical Association Journal 182(18), E839–E842. https://doi.org/10.1503/cmaj.090449

Brüggen, E., 2009, ‘A critical comparison of offline focus groups, online focus groups and e-Delphi’, International Journal of Market Research 51(3), 363–380. https://doi.org/10.2501/S1470785309200608

Chou, C., 2002, ‘Developing the e-Delphi System: A web-based forecasting tool for educational research’, British Journal of Educational Technology 33(2), 233–236.

Cornick, P., 2006, ‘Nitric oxide education survey – Use of a Delphi survey to produce guidelines for training neonatal nurses to work with inhaled nitric oxide’, Journal of Neonatal Nursing 12(2), 62–68. https://doi.org/10.1016/j.jnn.2006.01.005

Davidson, S.J. & Candy, L., 2016, ‘Teaching EBP using game-based learning: Improving the student experience’, Worldviews on Evidence-Based Nursing, Sigma Theta Tau International 13(4) 285–293.

Engles, T.C.E. & Kennedy, H.P., 2007, ‘Enhancing a Delphi study on family-focused prevention’, Technological Forecasting and Social Change 74(4), 433–451. https://doi.org/10.1016/j.techfore.2005.11.008

Eubank, B.H., Mohtadi, N.G., Lafave, M.R., Wiley, J.P., Bois, A.J., Boorman, R.S. et al., 2016, ‘Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology’, BMC Medical Research Methodology 16:56. https://doi.org/10.1186/s12874-016-0165-8

Graefe, A. & Armstrong, J.S., 2010, ‘Comparing face-to-face meetings, nominal groups, delphi and prediction markets on an estimation task’, International Journal of Forecasting 27(1), 183–195.

Green, R.A., 2014, ‘The Delphi technique in educational research’, viewed n.d., from Journals.sagepub.com.

Lindell, D.F. & Demi, A.S., 2018, ‘Delphi technique’, in J.J. Fitzpatrick (ed.), Encyclopaedia of nursing research, pp. 164–165, 4th edn., Springer, New York.

MacDermid, J.C., Brooks, D., Solway, S., Switzer-McIntyre, S., Brosseau, L. & Graham, I.D., 2005, ‘Reliability and validity of the AGREE instrument used by physical therapists in assessment of clinical practice guidelines’, BMC Health Services Research 5(18), 1–12.

Mamaqi, X., Miguel, J. & Olave, P., 2010, ‘The e-Delphi method to test the importance competence and skills: Case of the lifelong learning spanish trainers’, World Academy of Science, Engineering and Technology 42, 1494–1502.

Meshkat, B., Cowman, S., Gethin, G., Ryan, K., Wiley, M., Brick, A. et al., 2014, ‘Using an e-Delphi technique in achieving consensus across disciplines for developing best practice in day surgery in Ireland’, Journal of Hospital Administration 3(4), 1–8. https://doi.org/10.5430/jha.v3n4p1

Moore, L., De Silva-Sanigorski, A. & Moore, S.N., 2013, ‘A socio-ecological perspective on behavioural interventions to influence food choice in schools: Alternative, complementary or synergistic?’, Public Health Nutrition 16(6), 1000–1005.

Murad, H.M., Asi, N., Alsawas, M. & Alahdab, F., 2016, ‘New evidence pyramid’, BMJ Evidence-Based Medicine 21, 125–127. https://doi.org/10.1136/ebmed-2016-110401

Rantanen, J., Lehtinen, S., Valenti, A. & Iavicoli, S., 2017, ‘A global survey on occupational health services in selected international commission on occupational health (ICOH) member countries’, BMC Public Health 17(1), 787. https://doi.org/10.1186/s12889-017-4800-z

Rijal, S., 2009, ‘Leading the learning organization’, Business Education and Accreditation 1(1), 131–140.

Szamosi, L.T. & Duxbury, L., 2002, ‘Development of a measure to assess organizational change’, Journal of Organizational Change Management 15(2), 198–201.

Taylor, R.M., Feltbower, R.G., Aslam, N., Raine, R., Whelan, J.S. & Gibson, F., 2016, ‘Modified international e-Delphi survey to define healthcare professional competencies for working with teenagers and young adults with cancer’, BMJ Open 6, e011361. https://doi.org/10.1136/bmjopen-2016-011361

Thangaratinam, S. & Redman, C.W.E., 2005, ‘The Delphi technique’, The Obstetrician and Gynaecologist 7(2), 120–125.

Toronto, C., 2017, ‘Considerations when conducting e-Delphi research: A case study’, Nurse Research 25(1), 10–15. https://doi.org/10.1136/bmjopen-2016-011361

Van der Linde, H., Hofstad, C.J., Van Limbeek, J., Postema, K. & Geertzen, J.H.B., 2005, ‘Use of the Delphi Technique for developing national clinical guidelines for prescription of lower-limb prostheses’, Journal of Rehabilitation Research & Development 42(5), 693–704.

WHO, 2010, WHO handbook for guideline development, WHO Press, Geneva, pp. 1–67.


Crossref Citations

1. Expert consensus for a national essential antidote list: E-Delphi method
Dalal Al-Taweel, Samuel Koshy, Sara Al-Ansari, Asmaa Al-Haqan, Bedoor Qabazard, Menno I. Gaakeer
PLOS ONE  vol: 17  issue: 6  first page: e0269456  year: 2022  
doi: 10.1371/journal.pone.0269456