3.1 SMART (Systematic Model adequacy Assessment and Reporting Tool)3.1.1 Steps of Decision-Analytic Model (DAM) Development and DAM Features

We grouped the steps of DAM development from the literature [7,8,9] into three overarching labels: model conceptualisation, model inputs and model analysis. (1) Model conceptualisation involves conceptualising the decision problem by translating the knowledge about the disease and its care pathway into a structured representation [7]. It also includes conceptualising the model structure, where elements of the decision problem are represented in mathematical structure [7], and selecting modelling approaches that align with the conceptualised model [9]. (2) The second step, model inputs, involves identifying and populating all inputs [9]. (3) In the model analysis step, the DAM is run to generate outcomes, and uncertainties within the model are identified and evaluated. Within these steps, we identified 28 DAM features, each with both a simple and a complex modelling choice. For example, in the case of population heterogeneity, if the population is deemed heterogeneous, modelling all subgroups and/or including patient characteristics through individual patient-level modelling can lead to a more complex model compared to one that treats the population as homogeneous. Table 1 summarises all DAM features and their simple and complex modelling choices.

Table 1 Features in decision-analytic models and their simple and complex modelling choices3.1.2 Consequences of Simplifying a DAM Feature

We have identified two key consequences of simplifying a DAM feature when the decision context requires a complex modelling choice: validity and transparency. With validity, we refer to the degree to which a model accurately represents the real-world scenarios, makes relevant predictions and produces reliable outcomes to inform the decision problem [15, 20]. To fully assess a DAM’s validity, several steps are involved, including identifying and correcting errors in model implementation, assessing conceptual validity, validating input data and ensuring that the model’s predictions align sufficiently well with real-world data [15, 27]. In SMART (Systematic Model adequacy Assessment and Reporting Tool), we are focusing on assessing validity as the model’s adequacy and the applicability of its results to the decision context. For example, if the decision context involves a heterogeneous patient population that consists of different subgroups with differences in the disease trajectory, modelling the population as homogeneous may produce misleading results, thus impacting the validity of the model adversely.

With transparency, we refer to the stakeholders’ and reviewers’ ability to understand and evaluate the model structure, parameters and results [15]. Transparency could also influence the replicability of the model results, depending on whether the technical information and details are reported clearly. [15, 28]. In the same example as above, modelling a homogeneous population, when the actual population of the decision problem is heterogeneous, could have both positive and negative impact on the model’s transparency. For example, fewer states or branches in the DAM could make it easier for external reviewers to understand and evaluate the model, while the intended audience might prefer to see more details on treatment effectiveness based on the different characteristics of the population.

3.1.3 SMART

SMART integrates the theoretical framework of DAM features, their simple and complex modelling choices, and a section for evaluating the consequences of choices that deviate from the requirements of the decision context (Fig. 1). The tool includes three sections. Section 1 (Features of a decision analytic model) lists DAM features, with each row in the tool corresponding to a specific feature. Section 2 (Fill in the modelling choices) allow users to select each modelling choice from predefined options, first based on the decision context requirements, then, based on their own choice. Requirements of the decision context can be informed by the decision problem, i.e. PICOTPA (population, intervention, comparator, outcomes, time horizon, perspective and audience of the decision problem), disease management guidelines, relevant reference cases and/or health economic guidelines. If the user deviated from these requirements, they must assess the consequence of their choices in Section 3. If a simplified modelling choice is made that does not fully capture the necessary complexity, a more conservative approach should be adopted by selecting the simpler option from the drop-down list in Section 2 and then assessing its consequences in Section 3. Users may also select “Not applicable” if a feature is irrelevant to their decision context. In Section 3 (Fill in the consequences and provide justifications for your modelling choices), the user must identify the consequences of their deviating choices for validity and transparency and provide justifications for their modelling choice and the rationale used to assess its consequences. The user can also include supporting evidence (i.e., literature or published DAMs for the same disease).

Fig. 1

SMART (systematic model adequacy assessment and reporting tool)

The steps required to complete the tool are explained in the user instructions sheet provided with the tool. A glossary sheet is also integrated into the tool to ensure consistent use among different users (Also available in the ESM). SMART, including the glossary and user instructions sheets, can be found at https://osf.io/4j8qx/?view_only=7c32744f833845a596629e467f7c71ac.

3.1.4 Case Example: Treatment-Resistant Hypertension Early Decision Model

We used SMART in a case example on treatment-resistant hypertension (Table 2). The decision context involved developing an early DAM to assess the potential cost effectiveness of a new combination treatment and the associated digital biomarker for treating patients with treatment-resistant hypertension, compared to standard of care, from the Dutch societal perspective. The DAM needed to be developed within a limited budget and short timeline to support the funding decision for proceeding with the phase II trial, the first-in-human study.

Table 2 SMART as used in the treatment-resistant hypertension case study example

We completed the tool based on the decision context by first filling in the requirements for modelling choices, then selecting our own modelling choices in Sect. 2. In Sect. 3, we assessed the consequences of deviating from standard choices for validity and transparency. The deviating choices were made in capacity constraints, number of outcomes, model perspective, number of health states/events, connections between health states and events, timing of modelled events and consequences, data analysis and decision uncertainty analysis. We also provided justifications for our modelling choices and the consequences assessment. The full decision context and the completed tool are provided in the ESM and can be accessed at https://osf.io/4j8qx/?view_only=7c32744f833845a596629e467f7c71ac.

3.2 Feedback from the Interviews and Expert Workshops

Five online in-depth interviews were conducted with experts in DAMs using Microsoft Teams. Each interview lasted approximately 60 minutes. Three of the participants were DAM developers, while two were DAM reviewers. Three participants worked in academic settings, one in an HTA agency, and one in both an academic setting and an HTA agency.

In addition, two online expert workshops were conducted using Microsoft Teams, each lasting 120 minutes. Five participants attended the first workshop on 28 November, 2024, and seven attended the second workshop on 4 December, 2024. Among all participants, nine were DAM developers, and three were DAM reviewers. Eight participants worked in academic settings, two in HTA agencies, and two in industrial settings.

Feedback and recommendations from in-depth interviews and expert workshops were used to update and improve the tool (the evolution of the framework of SMART based on feedback from interviews and workshops are detailed in the ESM). Table 3 summarises key findings, recommendations and corresponding adaptations from the interviews and workshops.

Table 3 Key feedback and recommendations from the interviews and expert workshops and the adaptation made to address them