Quantitative SurveySample

Seventy-six individuals responded to the survey; however, data from six respondents were removed due to missing data in key fields. The 70 individuals in the final sample represented the following organizations: pharmaceutical industry (n = 38), regulatory authority (RA) (n = 15), non-governmental organization (n = 6), academia (n = 4), government other than RA (n = 2), procurement agency (n = 1), other (n = 4).

Survey Results

Box 2 lists the 11 metrics provided in the quantitative survey (Top Five in priority order).

Box 2 Eleven Metrics Considered for the Framework

Respondents were invited to provide additional metrics they thought would be relevant to include in a framework measuring the public health impact or efficacy of reliance-based pathways. In response, 56 entries were generated (See Supplementary Material “Other metrics” to be Included in a Framework to Measure the Public Health Impact of a Reliance Agreement (Open-ended responses)). Of the 56 entries, 51 were novel (i.e., different from the original 11). Participant responses that directly or indirectly reflected metrics from the original list of 11 (e.g., cost savings, ability to meet targeted product assessment timeline) were pulled out and are shown in the supplementary materials as “Responses that reiterated the original 11 metrics.”

The respondents were asked to choose the Top Five most important metrics to be included in a framework measuring the impacts of a reliance-based agreement, selecting from the metrics listed in Box 2 or any metric they provided. The Top Five results shown in Fig. 1 follow: the ability of an agency to meet targeted product assessment timeline; increased access to expertise that is limited or not available in the agency; shortened median number of days to market for medical products; lower morbidity and mortality due to greater access to medical products; and movement toward technical standards harmonization. The disaggregation of survey data by subgroups (e.g., NGOs, academia, industry) resulted in small subsamples that may be useful for generating hypotheses, but will not be presented in this manuscript.

Figure 1

Top Five Metrics Selected by Survey Respondents.

Prior to identifying the Top Five metrics to be included in a framework, respondents were asked to rate the importance of including each metric in a framework that would measure the public health impact or effectiveness of a reliance agreement on a 5-point Likert scale, ranging from “not at all important” to “extremely important.” Below are the results for how they rated the importance of the Top Five metrics (see Fig. 2). Although some participants rated the Top Five metrics as being “not very important” or “not important at all,” taken together, the Likert scale ratings support the “Top Five” results, as the majority of respondents rated each of these metrics as being “important” or “extremely important.” A few respondents skipped some of the Likert scale questions, but still ranked the metrics in the Top Five exercise. It is unclear why this occurred; it is possible that some respondents felt as if they lacked the knowledge to fairly rate all of the Likert scale questions.

Figure 2

Survey Respondents’ Ranking of the Importance of Including each of the Top Five Metrics in a Framework.

Qualitative Individual In-Depth Interviews (IDIs)Sample

Ten individuals participated in IDIs. Of the 10 IDI participants, five represented regulatory authorities, two of which were from low- or middle-income regions (see Box 3). Other sectors represented included industry (i.e., drug company and generic drugs membership organization), patient advocacy, and international public health.

Box 3 IDI Participants’ Locations and Perspective Areas

IDIs focused on participants’ opinions about the Top Five metrics as determined by the quantitative survey results. Respondents were not shown the list of novel metrics provided by individual respondents (See Supplementary Material “Other metrics” to be Included in a Framework to Measure the Public Health Impact of a Reliance Agreement (Open-ended responses)). Qualitative results are summarized below.

Top FiveAbility to Meet the Targeted Product Assessment Timeline and Shortened Median Number of Days to Market for Medical Products

Results for “ability to meet the targeted product assessment timeline” and “shortened median number of days to market for medical products” are presented together, as comments about these metrics overlapped. The IDI participants generally agreed that it would be important to include these two metrics in a framework; however, there were questions as to the usefulness of such indicators for both well-resourced RAs and less well-resourced RAs without additional clarification. For example, shortening a timeline tied to fee regulations was viewed as less meaningful than shortening a timeline that would improve drug lag (i.e., any delay in getting medicines to patients [7]). Others described a need for secondary indicators to understand the local context. Secondary indicators could provide valuable insights into why a product is not being marketed after approval and whether donor programs, not national regulators, are the reasons for delays in patient access.

A sentiment expressed by one RA representative was that every RA considering or engaging in reliance is looking at or collecting data to monitor timelines. Other participants similarly indicated that such data are currently being collected or would be feasible to collect except in instances of RAs with extremely limited resources. Participants representing well-resourced RAs acknowledged that initially, reliance pathways may not save time because protocols must be established; however, they viewed reliance as a long-term commitment, one for which best practices are still being developed. Consequently, these metrics will become more meaningful over time.

Increased Access to Expertise that is Limited or Not Available in the Agency

Participants’ perceptions of the benefits of increased access to expertise varied depending on the resource level of the RA. Well-resourced RAs viewed increased access to expertise as a mechanism for collegial scientific discourse but explained that this sort of access does not determine whether a product will be assessed and subsequently approved. Reliance within GMP was briefly mentioned by one well-resourced RA; however, details were not explored, as the focus of the study was on the approval process. Staff from well-resourced RAs often participate in research “clusters” where products are discussed before the respective approval. Participants categorized these collaborations as “peer exchanges” that do not qualify as reliance or even as work-sharing. Two RA representatives expressed organizational reluctance to embrace reliance pathways; one due to a belief among RA staff that they would not have access to the full dossier, and another raised the need for legal provisions in order to use reliance-based pathways.

Participants’ perceptions were that for less well-resourced RAs, increased access to expertise may have a significant impact on assessment. They explained that access to expertise is one of the main reasons less well-resourced RAs use reliance, indicating that this metric is important for less well-resourced RAs taking a more unilateral approach to reliance. With advanced technology and complicated products, predicting future expertise requirements is difficult, especially within less well-resourced RAs. Instead of trying to build capacity within an RA, it would make more sense to engage experts from other RAs through reliance pathways—a way of gaining needed expertise without publicizing a lack of experts. While participants supported the use of this metric, concerns were raised regarding how the metric would be measured and defined. One public health representative suggested assessing the actual demand for advanced technology product applications in lower-resourced environments bringing into question the pressing need for external expertise.

Lower Morbidity and Mortality Due to Greater Access to Medical Products

All five participants from RAs felt that lowering morbidity and mortality are noble goals and ethically important, but they did not consider lower morbidity and mortality (LMM) to be an appropriate metric for a reliance framework. Instead, they viewed it as an “impact indicator,” meaning that while LMM might be an outcome of a successful reliance partnership, it is not an RA’s goal. It was consistently stated that regulatory assessments determine the quality, safety, and efficacy of medical products. Furthermore, there are numerous factors that influence the availability of medical products to a regional population. There are many additional steps after regulatory authorization before patients have access to products. RAs are not responsible for what happens after approval; consequently, the majority of participants believed that LMM should not be a Top Five metric for a reliance framework nor should it be a central element, as there are too many other non-regulatory factors that impact a metric such as LMM. The two participants with patient and public health foci were more supportive of including LMM as part of the framework.

Participants questioned how LMM would be measured. In situations where a population is dealing with a specific disease on an epidemic or pandemic scale and therapeutic solutions are limited, it may be possible to measure, to some degree, the impact of approving a product and subsequently making it available in the market, but this is not the norm. RA and industry participants believed that LMM might be a more relevant metric for less well-resourced regions, but still thought there would be challenges to capturing the data, as post-market surveillance of the kinds of data needed to determine LMM in many countries is inadequate.

Movement Toward Technical Standards Harmonization

Participants representing RAs of all resource levels directly addressed the issue of alignment and harmonization. The view of one RA representative was that having harmonized standards would facilitate reliance because movement toward technical standards harmonization would set clear guidelines for the critical elements of an inspection. Defining an element through an agreed-upon numerical definition could, the representative suggested, take the ambiguity out of how different RAs interpret the information, thereby minimizing the risk of misunderstandings. Another RA representative agreed saying, “Harmonizing the technical standards would help reliance because it would give me, as Country A, assurance that you in Country B are going to look at and consider what I would have looked at and considered.” To this end, it was suggested that a more important metric may be to understand how each RA interprets and implements the standards. Participants commented that organizations such as ICH, WHO, and PIC/S have already developed technical guidelines for harmonizing to international standards. Much of this effort is being driven by higher-resourced RAs.

Beyond the Top FiveMissing Metrics

After IDI participants were asked to share their opinions regarding the Top Five, they were asked if there were any metrics that they thought should be added to the framework. The most common suggestion was “access.” The industry representatives strongly supported this metric while two of the RA representatives sought more in-depth information, for example, about drug quality and facilitators or barriers outside of the regulatory process that could affect patient access to medicines. Participants made the point that increasing medicine availability by providing approval for medical products that meet quality, safety, and efficacy standards is the main objective for all RAs; the facilitation of this process is the primary driver behind reliance. However, it is recognized that for this to happen, companies must submit the same product and same information in a timely manner to collaborating RAs. This is something over which regulators have no control. Ensuring patient-level access to these approved medicines is beyond the remit of the regulator, but respondents felt it played a significant role in assessing the impact of a reliance framework.

Should There Be One Framework or Multiple Frameworks?

During the interviews, participants often stated that some of the metrics identified in the initial survey were more relevant to less well-resourced RAs or less relevant to well-resourced RAs. Participants who primarily worked with or for well-resourced RAs suggested that most of the metrics were less relevant to their national regulatory practices because these RAs rarely rely on others or rely on others in a more mutual rather than unilateral way. As mentioned elsewhere in this summary and as supported by the quantitative survey data, the importance of LMM as a metric is rated higher for less well-resourced regions. Nevertheless, participants noted that there are still benefits to reliance even for “well-resourced” RAs. Some of the participants suggested that multiple frameworks may be more appropriate. For example, there could be one framework for well-resourced and one for less-well-resourced RAs.

All Top Five metrics were viewed as more important for a framework measuring the public health impact of reliance in less well-resourced RAs. This is due in part to the divide between access to resources for less well-resourced RAs versus well-resourced RAs, which made the selection of metrics difficult and global comparisons inappropriate. A few participants suggested that rather than having a universal framework, regional frameworks and metrics might be more appropriate. One respondent suggested that rather than having a global framework, it may be more useful, especially among less well-resourced RAs to use selected metrics to make comparisons among peers. For example, how does drug lag compare across the region (e.g., neighboring countries) and how has reliance helped to address this? Participants presented one last issue to consider when thinking about the utility of multiple frameworks: the ability of a region or RA to monitor and capture any data necessary for the framework. In some regions, it may be more feasible to focus on collecting only basic data (e.g., “How long did it take for a drug to be assessed?”).