Abstract

PURPOSE Building timely consensus among diverse stakeholders is important in primary health care research. Consensus can be obtained using the nominal group technique which includes 5 steps: (1) introduction and explanation; (2) silent generation of ideas; (3) sharing ideas; (4) discussion; and (5) voting and ranking. The main challenges in using this technique are a lack of representation of different stakeholder opinions and the amount of time taken to reach consensus. In this paper, we demonstrate how to effectively achieve consensus using an adapted nominal group technique that mitigates the challenges.

METHODS This project aimed to reach consensus on the priority care domains for individuals aged 65 or older, using an adapted nominal group technique with 4 strategies: (1) recruit 4 stakeholders groups (older people, clinicians, managers, decision makers) by using maximum variation and snowballing sampling approaches; (2) use remote tools to ensure high participation; (3) add an individual pre-elicitation activity to increase effectiveness; and (4) adapt discussions to the stakeholders’ preferences for meaningful engagement.

RESULTS In total, 28 diverse stakeholders participated. After the pre-elicitation activity and 1 round of group discussion, we reached consensus on a priority domain called symptoms, functioning, and quality of care. Adaptive group discussions and remote tools were the most effective strategies. All participants strongly agreed that they were able to express their views freely. Some perceived a need for emphasizing the alignment between the research objectives and anticipated practice and policy implications.

CONCLUSIONS This adapted nominal group technique is an effective and enriching method when timely consensus is needed among diverse stakeholders. Health care researchers in various fields can benefit from using this research methodology.

Key words:INTRODUCTION

The participatory research approach is an important tool for family medicine and primary health care research.1 By recognizing stakeholders’ expertise and lived experiences,2 this approach increases the applicability of research outputs.3 To achieve this, it is essential to rely on consensus-building methods that enable individuals with diverse backgrounds and opinions to collaboratively arrive at acceptable decisions.4 As this process can be challenging due to time and resource constraints,5 it is important to choose the best method for a given research project. There is a wide variation in the selection, use, and application of consensus-building methods and their reporting in health care, indicating a need for standardization.6

The Delphi technique for consensus-building method is used most often.6 It involves a series of questionnaires that gather opinions from a large number of respondents, usually with the goal to develop guidelines.7 The Delphi technique, however, has some disadvantages: repeated rounds of questionnaire completion is time-consuming, some people who participate in early rounds may drop out,7 and participants often do not engage with each other to discuss their opinions.6

Another well-known and highly utilized method to build consensus is the nominal group technique.4 Originally developed as an organizational planning technique,8 this method can be appropriate when dealing with a well-defined question.5 It can be used for problem solving, idea-generation, or eliciting priorities on a given topic from different groups of stakeholders.9 The classical nominal group technique is a structured process10 encompassing 5 steps: (1) introduction and explanation; (2) silent generation of ideas; (3) round-robin sharing of ideas; (4) group discussion and clarification of ideas; and (5) voting on and ranking the ideas.11 The nominal group technique has advantages as it is an efficient and productive way to reach consensus among a limited number of participants, provides rapid results to researchers, and allows participants to explain their opinions with rich justifications.10 The nominal group technique is also flexible enough to adapt to circumstances5 and can be used in combination with a range of techniques5 and supporting tools for participants.12 The 2 main challenges in implementing the nominal group technique are a lack of representation of opinions from different stakeholders and the amount of time to reach consensus.5 In this methodology article, we demonstrate how to effectively achieve consensus among stakeholders with a wide range of backgrounds and opinions using an adapted nominal group technique (aNGT) that mitigates these challenges.

METHODSOlder Persons’ Health and Social Services Research

This work was part of a project designed to understand the care trajectories of persons aged 65 or older. The Research Ethics Office (Institutional Review Board) of McGill University approved this study (A11-B63-19A). The first phase of this study consisted of adapting the International Consortium for Health Outcomes Measurement (ICHOM) standard set of health and social service indicators for older persons14 to the province of Quebec, Canada. The ICHOM encompasses 6 domains: (1) disutility of care (ie, treatment-related complications); (2) symptoms, functioning, and quality of care (eg, activities of daily living); (3) care (ie, care burden); (4) health care responsiveness (ie, participation and decision making); (5) clinical status (eg, frailty); and (6) quality of death (ie, place of death).14 We aimed to develop of a consensus on the priority domains and identify new indicators, if any, deemed important to the stakeholders.

The Adapted Nominal Group Technique

Given the challenges of the nominal group technique in terms of lack of representation and required time,13 we incorporated the following 4 strategies in the aNGT.

Recruitment of Diverse Participants

We used 2 sampling methods to elicit diverse perspectives. Using a purposeful sampling with maximum variation,15 we targeted participants of different ages, genders, geographic locations, and backgrounds.9 In the nominal group technique, it is recommended that the experts in each group are kept homogeneous in status and limited to a maximum of 7 participants.4 We created 4 groups with varying backgrounds (occupation or roles in care of older persons): (1) persons aged 65 years or older; (2) clinicians (eg, family physicians, geriatricians, nurses); (3) managers (eg, directors and health care professionals working in the management of Quebec regional health organizations); and (4) decision makers (eg, representatives of the Ministry of Health). A letter to potential participants, written by the senior author (I.V.), introducing the study team and objectives was e-mailed to our large network of researchers and collaborators. At this stage, we also employed a snowball sampling method.15 Those who agreed to be contacted (ie, potential participants) were sent an invitation e-mail that outlined the study activities and asked them to confirm their participation. In the e-mail, we also asked them to refer other individuals knowledgeable in the subject matter.15

Use of Remote Tools

The study was conducted during the COVID-19 pandemic, and we needed to reach many stakeholders over a vast territory. We facilitated recruitment by offering potential participants brief individual online meetings to explain the study. We used online consent and questionnaire completion, and a remote aNGT16-18 to facilitate participation, streamline the process, expedite data analysis, and generate timely results.19

Individual Pre-Elicitation Activity Before Discussions

Before the aNGT group discussion, participants individually reviewed materials that explained the study and ranked the ICHOM domains (ie, pre-elicitation).10 This approach allowed us to minimize the time needed for multiple votes.13 The questionnaire results for each group were then used to stimulate reconsideration and interaction among participants of the group during discussion (ie, controlled feedback)10 and lead the group to achieve consensus.

Adapt Discussions to the Stakeholder Group

To promote equity, diversity, inclusion, and active participation in discussions, we tailored group discussions to the needs of each stakeholder group. We gave older persons additional time to allow them to express their ideas (duration), scheduled group discussions with clinicians, managers, and decision makers around lunch hours (time of day), and adjusted the terminology to the group to avoid jargon (vocabulary).

Description of aNGT Activities

We conducted 3 online activities from April through December 2022 (Figure 1).

Figure 1.

Study activities and outputs.

ICHOM = International Consortium for Health Outcomes Measurement.

Individual Activity

The objective of this activity was to introduce the ICHOM set of indicators14 for older persons and pre-elicit20 participants’ opinions while maintaining anonymity.10 We prepared a 20-minute video and PowerPoint presentation introducing the project and French translations of the ICHOM domains and indicators. An e-mail included a link to the consent form which directed participants to these study materials. Participants were invited to complete a 15-minute online questionnaire in Microsoft Forms asking them to rank the ICHOM domains in order of priority based on their perspective (ie, first priority being 1, and last priority being 6). Text boxes were provided to allow participants to explain their thoughts behind the prioritization and propose new aspects not included in the original ICHOM set of 6 domains.14

Group Discussion

The objective of this activity was to allow participants to reflect on their initial ranking at the questionnaire stage, generate additional insights, and suggest modifications through iterative thinking and discussions.10 We conducted 4 group discussions, 1 with each stakeholder group. Before each group discussion, we descriptively analyzed the deidentified questionnaire responses (analysis within each group). We totaled ranking points for each domain (smaller total score indicated higher priority), and listed proposed aspects inportant to include in the ICHOM. We also prepared a short presentation of the results and sent it to the stakeholder groups the week before discussions.4

The specific logistic and organizational details of the group activities were designed to reduce facilitator bias5 and to create a non-judgmental, inclusive atmosphere4. Two co-facilitators (I.V., A.Q-V), who are experts on the topic and had credibility within the group, led discussions using a guide adapting the nominal group technique protocol stages11 (Table 1). In addition, 3 team members took notes (D.C-S., G.A-L., C.F-B), wrote participants’ ideas in a shared document, and ensured that the logistical aspects ran smoothly (eg, Zoom platform management, time keeping).

Table 1.

The Adapted Nominal Group Technique Protocol Checklist

To determine whether we had reached a consensus or needed to hold another round of group discussion, the facilitator summarized main convergences and any discordances among the group members at the end of each group discussion. We defined acceptable levels of consensus as conclusions with minimal discordance that did not require further discussion as confirmed by the group members.

Evaluation of Participant Experience

The objective of this activity was to assess the perceptions of stakeholders regarding our 4 aNGT strategies and their overall satisfaction with participation in this research. At the end of each group discussion, we provided participants with a link to a 5-minute anonymous online questionnaire in Microsoft Forms. We adapted the Public and Patient Engagement Evaluation Tool (version 2.0, August 2018), which promotes a comprehensive assessment of participants’ engagement level and experiences with different types of research activities.21,22 Responses to close-ended questions using a 5-point Likert scale ranging from strongly disagree to strongly agree were tabulated. Comments written for open-ended questions were categorized as either strengths or areas for improvement.

RESULTS

Among the 62 eligible participants who received the study link, 28 (14 women, 14 men) participants consented and completed the questionnaire (45% response rate). Mean (range) age was 58 (33-73) years. Participants were from 6 regions of Quebec. Reasons for not being able to participate in the study included time concerns, travel outside of the country, and personal (eg, health issues, loss of family members). Among those who completed the questionnaire, 20 (14 women, 6 men) participants joined a group discussion (4 persons aged 65 years or older, 7 clinicians, 5 managers, and 4 decision makers). There was a 29% attrition rate at this stage. The main reason for not participating in group discussion was a schedule conflict with the proposed days or times (Table 2).

Table 2.

Participation Results

Both the sum of total scores and overall ranking showed that the priority domain was symptoms, functioning, and quality-of-life; followed by the domains health care responsiveness, care, clinical status, disutility of care, and quality of death. After 1 round of group discussions, participants reached a consensus for the priority domain and identified new aspects of care that are important to include in the revised ICHOM for Quebec. Participants mentioned that the reason for prioritizing these domains was not obvious and suggested doing so according to different contexts (eg, community vs long-term care setting, home care vs end-of-life care). Details of these results will be published separately.

Of the 20 participants who completed the individual questionnaire and participated in discussion, 14 completed the evaluation (30% attrition). The reasons for dropouts were unknown as evaluations were anonymous. The results are summarized in Table 3. Adapting group discussions to stakeholder groups was the most effective strategy. All participants strongly agreed that they were able to express their views freely, that their views were heard, and that they were confident their input in the initiative would be considered. Use of remote tools was the second most effective strategy. Regarding the recruitment of diverse participants, some participants felt that there could have been more representation from community health care providers and older persons. Some participants felt that the pre-elicitation could have been clearer about the research context or questions and anticipated implications.

Table 3.

Results of the Evaluation Questionnaire (N = 14)

DISCUSSION

We presented an aNGT using strategies to alleviate the challenges of achieving representation and reaching timely consensus among diverse stakeholder groups. In addition, we demonstrated how evaluating participants’ opinions about the research activities helped assess consensus methods. The aNGT identified the domain of symptoms, functioning, and quality of care as stakeholders’ number 1 priority for care of older persons. This finding is consistent with an international nominal group technique study in which participants reached consensus on the importance of considering the individual life situation with a holistic perspective, and addressing functioning as a focus of care among community-dwelling older persons.23

Through evaluation of our results and experiences, we developed 6 recommendations for using the aNGT (Table 4).

Table 4.

Recommendations for Future Use of the Adapted Nominal Group Technique

Recruit Diverse Participants From Network

Our recruitment strategy of using our research network helped us successfully engage diverse stakeholders. Although response rates for individual and group activity participation were reasonable, there is likely some nonresponse bias.24 It is possible that participants recruited with snowball sampling had views compatible with those who referred them; however, this method helped reach difficult-to-access populations.25 One-third of participants completing the individual activity were unavailable to participate in discussions, so we potentially missed their complete opinions. Another limitation was that most of the participants were from urban or suburban regions. If the study timeline allows, researchers may spend more time to reach rural areas.

Repeat Study Objectives and Potential Implications

Overall, most participants were satisfied with this engagement initiative, though there was uncertainty about the extent to which the project would effect change. We recommend future researchers reiterate the aim and objectives of the study in a broad context during each activity. We told participants that we would send the summary of the results once the study was completed; however, they wanted to know more about potential implications sooner. Knowledge translation processes can be accelerated for small groups by sending them group-level preliminary results the week after completion of study activities.

Use Effective Communication Tools

For future aNGT studies, we recommend offering a remote approach as it was the second most effective strategy. We used remote technologies and visual representations as much as possible. We did not discuss how to adopt remote nominal group technique as this was previously published.18 We strongly recommend creating a minute-by-minute agenda for each discussion and sending it to participants beforehand as participants appreciated this and it helped us complete the discussions as planned.

Add an Individual Pre-Elicitation Activity Before Discussions

The pre-elicitation method mitigated the need for repeated rounds of questionnaire completion and accelerated the voting process. A single round of group discussion yielded consensus among stakeholders. Our introductory video and PowerPoint presentations were detailed enough, and we gave stakeholders the opportunity to contact us in case they had questions. Although none of the participants opted to do this, the evaluation results revealed a lack of clarity about the objectives of the activity. We recommend future aNGT pre-elicitation methods be more informative on the topic and explicitly state the objectives of the activity.

Adapt Discussions to the Stakeholder Group

Adapting group discussions to the needs and time constraints of each stakeholder group appeared to be most effective strategy. Adaptations may include the proposed duration, time of day, and vocabulary to be used in the discussion. In all 4 groups, participants used all the time allocated for the group discussion, where they were invited to ask questions of each other and discuss among themselves. They expressed that they felt privileged to have had the opportunity to share ideas with their peers, and felt less alone in the challenging process of caring for older persons. We suggest future aNGT researchers allocate sufficient time for discussion.

Hold 1 or 2 Rounds of Group Discussions

Although the group sizes were appropriate for small group discussions4 and we did not feel the need for a second round, participants expected more diversity from community clinicians and older persons. Having a mixed clinician and manager group discussion was suggested to improve exchange of ideas. This suggestion indicates that, although the classical nominal group technique is conducted with relatively homogeneous groups,4 the aNGT might allow for more flexibility and interdisciplinary research.

CONCLUSIONS

The aNGT created an open and engaging platform to reach timely consensus among diverse stakeholders while allowing participants to explain their opinions. To promote participation and reduce the time to reach consensus, adapting discussions to stakeholder groups’ needs and preferences appears to be the most effective strategy, followed by using remote technologies and visual representations. Recruitment using research networks is valuable. The number of group discussion rounds can be determined depending on reaching consensus and time constraints. Pre-elicitation methods accelerate the consensus process, provided that research objectives and potential implications are explicit. Health care researchers in various fields, including primary care and family medicine, can benefit from our experiences with using the aNGT to achieve a shared goal of improving care.

Acknowledgments

The authors thank Gabriela Lopes de Medeiros, who has granted permission to thank her by name, for proofreading the manuscript.

Received for publication February 23, 2024.Revision received June 11, 2024.Accepted for publication June 24, 2024.© 2024 Annals of Family Medicine, Inc.