Clinical simulation is primarily used as a training tool. Its utility as an educational methodology and its impact on improving patient safety have been well established. In this paper, we present the experience of a different use case: simulation as a tool for analyzing and improving the design of a clinical trial.

The analysis process through simulation results in a series of observations connected with proposed changes, obtained after reflecting on potential causes and effects. Simulation provides the same powerful type of everyday learning that individuals, teams, and organizations have and apply: the things that happen to us lead to evolution and changes in our actions, decisions, behaviors, and at an organizational level, our protocols. This perspective on simulation is holistic and extensive: simulation is learning from life before it happens.

During the last years, different experiences of using simulation as a strategy for analyzing work systems have been reported, although the published literature on this topic is far from as extensive as its educational use. Within these examples, there are limited published experiences of specifically using simulation to improve the design of a clinical trial [11]. From a methodological standpoint, there is also no evidence that there is a single way to use simulation for this purpose, and this is particularly relevant in key elements, such as how to obtain observations and the structure and strategy of debriefing conversations. Dubé et al. [14] proposed a debriefing framework for system-focused simulations inspired by the PEARLS model [15], which starts with the plus/delta scheme (what went well and what did not go well when applying the tested protocol), combined, if necessary, with direct or focused facilitation feedback. We also used a dichotomous perspective, but with a focus on “difficulties” and “successful adaptations.” The former could be equated with the delta part, while the latter do not share the same focus as the plus part, as it does not capture what went well when applying the protocol but rather the adaptations or spontaneous ideas that participants used to address challenges not foreseen in the protocol. This way of conceiving the types of observations collected aims to look not only at the appropriateness of the tested procedure, but also at how humans work and adapt to challenges in specific contexts, beyond what is covered by the protocols. This approach is fully connected with the Safety II perspective.

Another key aspect of our method is the analysis of results following debriefing, based on several sources: the observations from the debriefing itself, written records from observers, and a review of scenario-recorded videos. Colman et al. [8] propose conducting a Failure Mode and Effects Analysis (FMEA) with observations obtained in simulations for system improvement, especially when focused on space improvement. The team of Barcelona Children’s Hospital conducted some projects using this method in the past, but currently prefer to construct what we call an Observation–Effect–Cause–Solution table (OECS), which is inspired by the FMEA tool but with an expanded view that includes not only risks, but also, as explained, successful adaptations. Furthermore, we believe that FMEA focuses on patient safety, whereas a systems analysis with simulation should consider not only safety, but also other aspects of healthcare quality and the experience of patients, families, and professionals.

Some of the results obtained in SIMTest are quantitative (e.g., time required to perform certain activities or procedures), but many of them are qualitative and most are related to the concept of understanding human work in-depth. If we understand how humans work when trying to apply a clinical trial protocol in a specific context, we can improve the work system to help workers achieve the best results with the best experience and minimal risk. In this analysis process, we attempt to apply Argyris's double-loop view from a systemic perspective [16]. Thus, in SIMTest debriefings and subsequent result analysis, we aim not only to observe what people do and how they do it, but also their internal motivations and mental frames as foundations of their actions. This allows us to bridge work-as-imagined and work-as-done in a much more powerful way.

For example, one of the scenarios of the simulation was dedicated to analyzing the feasibility of providing the service of home nursing for the PK sample collection in the clinical trial. In this specific moment of the clinical trial, three PK sampling options (A: at site; B: home-nurse from site; and C: home-nurse from vendor) were considered. It was clear that the risks increased in the visits performed at home with a site’s nurse but most of all with a vendor’s nurse. To be able to make these types of decisions, it is key to have a multidisciplinary perspective to analyze after each scenario simulated the benefit versus risk, cost, patient, and family experience, etc. Specifically for this scenario, it was agreed that if the sponsor decides to offer the PK sampling process at home, the preference from the family was with a site’s nurse.

SIMTest activities provide an in-depth understanding of human work on the basis of simulation scenarios involving a limited number of people (those involved in that situation in reality). Sometimes, depending on the challenges and specificities of the clinical trial, it may be important to have the perspective of multiple people facing the same element. In these cases, combining simulation with other complementary methods of patient involvement and representation (panels, focus groups) before or after the simulation can provide additional value. Early involvement of patients and/or caregivers is essential to ensure that the clinical trial can have a patient-centered perspective, which is crucial for any clinical trial.

A key point in simulation-based activities is impact evaluation. On the basis of the perspective of multiple SIM Tests conducted in the last 10 years, the team of Barcelona Children’s Hospital is developing an impact evaluation framework for SIMTest activities with five levels of analysis, adapting the Kirkpatrick model used in education:

Level 1 (reactions) would include indicators of the extent to which participants find the experience satisfactory, engaging, and relevant to their work.

Level 2 (learning) would consist of indicators of the number and type of findings and recommendations issued in the SIMTest.

Level 3 (transfer) would evaluate the extent to which SIMTest recommendations translate into actual changes in spaces, equipment, or work procedures.

Level 4 (results) would comprise indicators of real quality improvement, safety, and/or the real experience of patients, families, and professionals.

Level 5: ROI and ROE.

In this work, we have focused on two aspects: A. participant satisfaction (and their perception of usefulness and relevance) and B. conceptualization of ROI and ROE indicators to be analyzed at the end of a clinical trial in which simulation was included as a validation process prior to the clinical trial submission to the regulatory agencies.

People’s perception is that a SIMTest exercise is a useful activity to analyze, understand, and improve, as reflected by the high satisfaction scores of participants and observers. This perception seems to be closely related to the fidelity and immersiveness of the scenarios, but also to the working climate and psychological safety that is created in the working group.

Although it was not possible to perform a quantitative study of the ROI and ROE indicators, it is possible to anticipate the potential value of the simulation if we are aware that many risks associated with the KPIs can be considered of medium or high impact level if they are not adjusted before the implementation of the clinical trial protocol. A second factor of value of simulating before implementing is the cost in terms of investment and time that may be involved in modifying an already established protocol [16].

It is important to define reliable KPIs, which can be compared before and after the SIMTest, for both ROI and ROE evaluation. The pre-SIMTest KPIs could be used as a reference to analyze the impact of the simulation results.

The final ROE analysis would be obtained once the mitigation plans have been implemented and the total expense at the end of the clinical study is available. Depending on the length of the study, this may take years, but it is a good exercise for the sponsor to have a prototype study that can help define decision points for running simulations in future studies.

There is a direct relationship between ROI and ROE analysis. For this project, the main variables identified were “time,” “cost,” and “quality.” This triangulation of data (Fig. 4) has a direct impact on patient and caregiver burden reduction and facility burden, as well as on interest, engagement, and retention rates.

Fig. 4

Relationship between variables connected with ROI and ROE

Finally, the time required to prepare the SIMTest can impact ROI and ROE evaluations. A SIMTest requires a few months of preparation and should be done at the stage where the clinical trial protocol design has a sufficiently deep level of detail, but adaptations are still possible. If this time investment can reduce future amendments to the protocol already submitted, the end result will easily be time and cost savings in the overall trial execution. In any case, the use of clinical simulation to improve clinical trial design is still novel. It stands to reason that increasing the frequency of its use will probably lead to a standardization of the preparation processes and a reduction in the time required.

This work has some limitations. On one hand, the timing of the SIMTest within the clinical trial development process made it difficult to implement some suggested changes, which would have been easier in earlier stages. However, it is essential to consider that to conduct the simulation proposed, which included some complete medical visits, it was necessary to have the most advanced version possible of the clinical trial protocol and the trial materials. It will be very useful to develop experience in using clinical simulation for clinical trial design, so that we can define the optimal timing for conducting analysis simulations in this context.

On the other hand, the scope of this work was not to conduct a complete ROI and ROE analysis, but only to conceptualize indicators. This was influenced by the development stage when the simulations were conducted. There are very limited experiences reported to date using clinical simulation to improve the design of a clinical trial, so we believe the scope of this work is highly valuable, as it provides some insights into the use of simulation in this context and offers an exercise in defining ROI and ROE indicators that can greatly assist in the design of future projects. Studies applying ROI and ROE indicators to assess the impact of SIMTest exercises in the context of clinical trial design are needed to validate the model and gain a deeper understanding of its utility, applicability, and efficiency.