IntroductionGeneral Background

Mental well-being is commonly defined as a complex construct that includes a subjective experience (subjective well-being, which is often referred to as “happiness”) [] and a process of self-realization (psychological well-being) [,]. Traditionally, it was thought that mental well-being would arise in the absence of mental illness, as they were considered opposite ends of 1 continuum []. However, the absence of mental illness was found to be insufficient to produce good mental well-being []. The dual-continuum model has identified that mental well-being and mental illness are 2 distinct but related continua instead [], both of which could be considered part of mental health []. It is important to focus exclusively on the effective promotion of mental well-being [], as only a small proportion of the general population has optimal levels of mental well-being [,].

In addition, mental well-being in the general population is crucial for allowing society and the individuals within it to thrive. Improved mental well-being is connected to increased productivity, personal growth, a higher quality of life, stronger social cohesion, and more fulfilling and lasting relationships, as well as a decreased likelihood of developing diseases and mental illnesses and a longer lifespan [,,,]. Promoting mental well-being in the general population is therefore considered a fundamental goal by the World Health Organization (WHO), as described in the Mental Health Action Plan 2013-2030 []. Mental well-being promotion interventions provide “various activities or practices that aim to promote, build on, increase or foster primarily individuals’ strengths, resourcefulness or resiliency” [].

Evidence suggests that a variety of psychological approaches are effective in promoting mental well-being, including acceptance and commitment therapy (ACT), compassion, cognitive behavioral therapy (CBT), mindfulness, positive psychology, and multitheoretical interventions []. These psychological approaches were found to have small to moderate effects on mental well-being in the general population, whereby mindfulness-based interventions (MBIs) and multicomponent positive psychology interventions were particularly efficacious [,]. Further meta-analyses focusing on positive psychology interventions, MBIs, and ACT-based interventions separately also found similar effects on mental well-being [-].

However, these systematic reviews did not focus on fully automated digital interventions. Fully automated digital interventions are interventions that are delivered entirely by the technology itself, not requiring any form of human support (by clinicians or nonclinicians) []. Although fully automated digital interventions might be less effective, as recent research has found that any form of human support enhances the effectiveness of interventions [], fully automated digital interventions allow for great scalability and are highly cost-effective and accessible []. Therefore, fully automated digital interventions provide a particularly pertinent way to promote mental well-being in the general population.

Overall, there is a need to systematically review the evidence of the effectiveness of fully automated digital mental well-being interventions to improve mental well-being (which includes subjective and psychological well-being) in the general population. Furthermore, an understanding of what psychological approaches work when delivered fully automated digitally and for whom (as one approach does not suit all) [] is needed.

Main Objective

This systematic review aims to understand the effectiveness of fully automated digital interventions in promoting mental well-being in the general population.

Secondary Objectives

Furthermore, the systematic review aims to explore the effectiveness of fully automated digital mental well-being interventions across psychological approaches and population subgroups.

MethodsStudy Protocol

The systematic review protocol was registered on PROSPERO (CRD42022310702). The Cochrane handbook was used when designing and conducting the systematic review [], and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines were followed for reporting of the systematic review [].

Eligibility Criteria

Studies were included if they used a fully automated digital intervention that aimed to promote mental well-being in the general population.

The study needed to include adults, meaning that the population needed to be aged ≥18 years. General population was further defined as any adult population subgroup that was not a clinical population and was not specifically recruited by the researchers because of (expected) lower mental well-being baseline scores.

Digital interventions were defined according to the National Institute for Health and Care Excellence [] as interventions that are delivered through hardware and electronic devices (eg, smartwatches and smartphones), software (eg, computer programs and apps), and websites. The intervention needed to be fully automated, which means it should be delivered by the technology itself entirely, independent from health care professionals, and not containing any other form of social support []. For example, a digital web-based intervention in which video content was delivered automatically would have been included, whereas a digital video call intervention in which a health care professional delivered content would have been excluded. Although the content should be delivered entirely by the technology itself, the elements of the study could still have been conducted by the researchers. For example, researchers could have screened, obtained measures, and obtained informed consent (digitally or in person), after which they could have provided the participants with access to the intervention.

Furthermore, the intervention needed to use individual mental well-being promotion, defined by the WHO as “various activities or practices that aim to promote, build on, increase or foster primarily individuals’ strengths, resourcefulness or resiliency” []. This should be a psychological intervention.

Interventions that included physical activity–related or lifestyle-related interventions were excluded. If an intervention contained elements that did not include mental well-being promotion, they would also be excluded, as the detection of the effectiveness of mental well-being promotion strategies would not be possible. For example, an MBI would have been included; however, an MBI that included a yoga session would have been excluded.

The outcome needed to consider a validated measure of mental well-being, including psychological well-being or subjective well-being.

Finally, studies needed to investigate the effectiveness of this digital intervention on mental well-being. Therefore, quantitative randomized and nonrandomized studies of interventions, such as before-after studies, were considered appropriate, as they can provide insights into the effectiveness of interventions []. For further details regarding the inclusion and exclusion criteria, please refer to the protocol [].

Searches

The initial search was conducted in February 2022 and updated using a title and keyword search in October 2022. The databases searched included MEDLINE, Web of Science, Cochrane, PsycINFO, PsycEXTRA, Scopus, and ACM Digital. Combinations of the following key search terms were used: “mental well being,” “mental wellbeing,” “psychological well being,” “psychological wellbeing,” “subjective well being,” and “subjective wellbeing,” in combination with “digital*,” “online,” “internet,” “web-based,” “app,” “apps,” “smartphone application*,” and “mobile application*.” No restrictions were applied. Refer to [-] for the detailed searches conducted in each database.

Study Selection

Each record was double screened, and the reviewers were blinded to each other’s decisions throughout the process. To ensure consistency and quality of the screening process, the lead author (JG) screened all records, and double screening was conducted by MB, ET, and MZ. After screening 10.71% (776/7764) of the records, interreviewer reliability was calculated, which ranged from moderate to substantial agreement (Cohen κ=0.54-0.79) []. Inconsistencies in the screening process were discussed, and conflicts were resolved through discussion. If conflicts remained, an additional discussion with a third, senior reviewer (BA) was conducted. Upon completion of the screening, interreviewer reliability was recalculated (Cohen κ=0.42-0.80), and conflicts were again resolved using the same process. This process was then repeated for full-text screening.

Data Extraction

Before data extraction, the Cochrane data collection form was adapted and prepiloted for this review. Data extraction included information regarding the study population, participant demographics, and setting; details of the intervention and control conditions (such as duration, frequency, timing, and activities); study methodology; recruitment and study completion rates; outcomes, outcome measures, and times of measurement; and information for the assessment of the risk of bias (RoB). Two reviewers (JG and AM) independently extracted all relevant data from the included studies and held meetings to discuss any discrepancies in data extraction. When conflicting views on the data extraction occurred, a third, senior reviewer (BA) advised on how to resolve the issue. Missing data were sought by contacting the lead author of the study via email, which was identified through the journal paper.

RoB Assessment

RoB was assessed independently by 2 reviewers (JG and AM) using the Cochrane RoB 2.0 tool for randomized controlled trials (RCTs) []. No standardized tools were available for noncontrolled before-after studies; therefore, the National Institutes of Health tool, “Quality Assessment Tool for Before-After (Pre-Post) Studies with No Control Group,” was used as a guidance to provide an indication of the RoB in these studies []. However, it was considered that these studies would provide a lower quality of evidence. Following the RoB assessments, discussions were held to discuss conflicts, and any remaining disagreements were resolved through verbal discussion with a third reviewer (BA).

Data Synthesis and Meta-Analysis

Mean, SD, and total number of participants were extracted for each postintervention mental well-being outcome in the study arms that met the inclusion criteria of the digital mental well-being intervention and control group. The effect estimates were averaged, where the studies included multiple study outcomes. This method was also adopted for multiarm studies because it was considered meaningful to combine the intervention effects, as all the included intervention arms were digital mental well-being interventions. In addition, this avoided double counting of participants in the control group. Standardized mean differences (SMDs) were used in a random-effects model.

Initially, both the per-protocol (PP) and intention to treat (ITT) data were extracted. However, only PP data were included in the meta-analysis, as high dropout rates (ranging up to 85%) led to ITT data being less meaningful.

Visual inspection of the forest plot and the chi-square and I2 tests were used to assess heterogeneity. A value of >50% was considered to represent substantial heterogeneity. Heterogeneity was explored, interpreted, and contextualized.

ResultsDescription of Studies

An initial search yielded 12,672 records. Following deduplication, 7764 records were screened in Covidence (Veritas Health Innovation). A total of 7526 records were excluded following title and abstract screening, and 238 records were sought for retrieval for full-text screening. A total of 230 full-text records were screened, leading to the exclusion of another 213 records. The most common reasons for exclusion were the population being a clinical population, intervention not solely using mental well-being promotion, intervention not being fully automated and digital, or that the study was still ongoing. For full details of the study selection process, refer to .

An updated title and keyword search in October 2022 yielded another 525 records. After deduplication, 366 articles were screened in Covidence. A total of 347 articles were excluded, and full texts of 19 articles were obtained. Furthermore, 17 articles were excluded following full-text screening.

‎Figure 1. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart of the search strategy outcomes. Narrative Summary

A total of 18 records containing 19 studies were included in this systematic review, including 17 RCTs and 2 non-RCTs before-after trials.

Setting and Participants

Studies mainly occurred in Western countries; the participants were primarily female and highly educated; and the study populations were students, employees, mothers, and other general population samples ().

Table 1. Characteristics of the included studies.Study, yearPopulationSettingComparatorOutcomeaStudy 3 from Avey et al [], 2022EmployeesUnited States and AustraliaUnknownPWBbBakker et al [], 2018General populationAustraliaWaitlist controlMWBcBrazier et al [], 2022TraineesUnited KingdomWaitlist controlMWBChampion et al [], 2018EmployeesUnited Kingdom and United StatesWaitlist controlSWBdChung et al [], 2021StudentsAustralia and United KingdomWaitlist controlMWBStudy 1 from Di Consiglio et al [], 2021StudentsItalyActive controlPWBStudy 2 from Di Consiglio et al [], 2021StudentsItalyNonePWBEisenstadt et al [], 2021Real-world app usersUnited KingdomNoneMWBGammer et al [], 2020Mothers of infants aged <1 yUnited KingdomWaitlist controlMWBLiu et al [], 2021StudentsChinaPlaceboSWBLy et al [], 2017General populationSwedenWaitlist controlPWB and SWBMak et al [], 2018General populationChinaActive controlMWBManthey et al [], 2016General populationGermanyActive controlSWBMitchell et al [], 2009AdultsAustraliaPlaceboPWBNeumeier et al [], 2017EmployeesGermany and AustraliaWaitlist controlSWBPheh et al [], 2020General populationMalaysiaActive controlMWBSchulte-Frankenfeld and Trautwein [], 2021Students with a part-time jobGermanyWaitlist controlSWBShin et al [], 2020StudentsUnited StatesPlaceboSWBWalsh et al [], 2019StudentsCanadaActive controlPWB

aMental well-being outcomes included 5-item mental well-being index (World Health Organization-5) [] and Warwick-Edinburgh Mental Well-Being Scale (version 1) []. Subjective well-being outcomes included Satisfaction With Life Scale [], Positive And Negative Affect Schedule [], Satisfaction with Life and happiness [], Subjective Happiness Scale [], and single-item life satisfaction and affect measure []. Psychological well-being outcomes included psychological well-being [], Psychological Well-Being Scale [], Psychological Well-Being Index (adult) scale [], and Flourishing Scale [].

bPWB: psychological well-being.

cMWB: mental well-being.

dSWB: subjective well-being.

Psychological Approaches

Several different psychological approaches were used, including the following: (1) mindfulness, ACT, and self-compassion; (2) positive psychology; (3) cognitive behavioral; and (4) integrative (). The most frequently used psychological approach was mindfulness, ACT, and self-compassion. General intervention activities and behavior change techniques, such as well-being tips and behavior change techniques to form habits, were adopted across psychological approaches and in most interventions ().

The intervention content was primarily developed by the study researchers and clinical psychologists (15/19, 79% of studies), some studies collaborated with companies or digital laboratories to develop the intervention (2/19, 11%), and some studies tested a preexisting intervention developed by a company (2/19, 11%).

Table 2. Description of intervention characteristicsa.Psychological approach underpinning the interventionActivities or practicesStudies adopting the approachMindfulness, ACTb, and self-compassionMeditation: awareness of inner experiences, present moment, and acceptance
Overcoming obstacles in mindfulness meditation
Body scan
Increasing awareness through biofeedback
Being mindful in daily life
Loving-kindness meditation
Compassionate journaling and breaks
Self-kindness activities
[,,,,,,]Positive psychologyGratitude (gratitude diary and letter)
Positive future imagination
Best possible self
Counting blessings
Random acts of kindness
Replaying positive experiences
Using strengths
Savoring the moment
Wearing a smile
Brainstorming meaningfulness
[,-,]Cognitive behavioral approachMood-related activities (eg, mood tracker, mood diary, and mood improvement activities)
Challenging thoughts and behaviors
Problem-solving
Goal setting (SMARTc goals and planning)
Committed actions
Journaling
[,]Integrative approachA combination of intervention activities or practices of these psychological approaches
[,,,,]

aFor more detailed intervention description, refer to .

bACT: acceptance and commitment therapy.

cSMART: Specific, Measurable, Achievable, Relevant, and Time-bound.

General psychological intervention components.

General intervention components adopted across interventions

Psychoeducation (eg, on emotions, needs, values, and mental illness)Support-seeking informationWell-being tips

Behavior change techniques adopted across interventions []

Habit formationGoal settingAction planning (eg, implementation intentions)Prompts or cuesSelf-monitoring of behavior or outcome of behaviorSelf-assessment of affective consequencesFeedback on behaviorMaterial or nonspecific rewardTextbox 1. General psychological intervention components.Intervention Delivery

A total of 24 fully automated digital mental well-being interventions were included. The interventions were app based (n=10), web based (n=11), both app and web based (n=2), and SMS text message (n=1) interventions ().

Table 3. Intervention characteristics and dropouta.Study, yearParticipants randomized, NbInterventionDurationFrequencyMode of deliveryDropout, n (%)cStudy 3 from Avey et al [], 2022102Resilience intervention10 wkWeeklyApp based3 (2.9)Bakker et al [], 2018226Moodkit, Moodprism30 dDailyApp based108 (47.8)Brazier et al [], 2022279Dear Doctor10 moFortnightlySMS text message126 (45.2)Champion et al [], 201874Headspace30 dDailyApp based12 (16.2)Chung et al [], 2021427Brief MBId6 wkWeeklyWeb based280 (65.6)Study 1 from Di Consiglio et al [], 202124Noibene3 mo4 timesWeb based0 (0)Study 2 from Di Consiglio et al, [], 2021178NoibeneNoneNoneWeb based119 (66.9)Eisenstadt et al [], 2021115Paradym2 wkDailyApp based81 (70.4)Gammer et al [], 2020206Kindness For Mums Online5 wkWeeklyWeb based80 (38.8)Liu et al [], 20211000Positive psychology intervention1-3 dTwiceWeb based132 (13.2)Ly et al [], 201730Shim2 wkDailyApp based3 (10)Mak et al [], 20182282Mindfulness-based program and self-compassion program28 dDailyApp based and web based1933 (84.7)Manthey et al [], 2016666Best possible self and gratitude8 wkWeeklyWeb-based video112 (16.8)Mitchell et al [], 2009160Strengths intervention and problem-solving intervention3 wkDailyWeb based111 (77.6)Neumeier et al [], 2017431PERMAe program and gratitude program7 dDailyApp based128 (29.7)Pheh et al [], 2020206Brief MBI1 dOnceWeb based100 (48.5)Schulte-Frankenfeld and Trautwein [], 202199Balloon8 wkDailyApp based35 (35.4)Shin et al [], 2020630Gratitude writing20 minOnceWeb based49 (7.8)Walsh et al [], 2019108Wildflowers3 wkDailyApp based22 (20.4)

aThis table represents the general characteristics of the studies included in this systematic review. Only interventions of the studies that met the inclusion criteria are presented in this table.

bN denotes the number of participants randomized in the study, irrespective of whether people conducted baseline and follow-up assessments.

cDropout rates are calculated from randomization to final assessment.

dMBI: mindfulness-based intervention.

ePERMA: Positive emotion, Engagement, Relationships, Meaning, Accomplishment.

Intervention Duration, Frequency, and Timing

The participants were expected to use the intervention for substantially varied duration across interventions, ranging from 1 single session to 10 months, and there did not appear to be a clear end strategy across interventions. Most commonly, intervention use was recommended daily for up to 30 days, weekly for up to 8 weeks, and fortnightly for up to 10 months. Participants were often encouraged to use and access the intervention content for 5 to 15 minutes at a time, irrespective of the duration of the intervention.

Level of Automation of Interventions

Access was generally automated with instant, sequential, or weekly access to content (). Most digital content was delivered in a standard way, and tailoring and dynamic delivery of content occurred in only 2 mental well-being interventions [,].

Table 4. Level of automation and engagement of intervention.Study, yearInterventionFrequency of content releaseHow access to intervention content was providedTailoring of content to improve or maintain engagementOther digital intervention strategies to improve or maintain engagementActual engagement with intervention contenta (%)Study 3 from Avey et al [], 2022Resilience interventionUnknownUnknownNoneNoneUnknownBakker et al [], 2018MoodkitInstant accessN/AbN/ANoneUnknownBakker et al [], 2018MoodprismInstant accessN/AFeedback on mental well-beingNoneUnknownBrazier et al [], 2022Dear DoctorFortnightlyAutomated text messageNoneNoneUnknownChampion et al [], 2018HeadspaceSequential accessAutomated access upon completion of step in the appNoneNone20.7Chung et al [], 2021Brief MBIcFortnightly or weeklyUnknownNoneNotifying of new contentUnknownStudy 1 from Di Consiglio et al [], 2021NoibeneInstant accessN/ANoneNone100Study 2 from Di Consiglio et al [], 2021NoibeneInstant accessN/ANoneNoneUnknownEisenstadt et al [], 2021ParadymUnknownUnknownNonePush notification32.1Gammer et al [], 2020Kindness for Mums OnlineWeeklyUnknownNoneNoneUnknownLiu et al [], 2021Positive psychology interventionSequential accessUnknownNoneNoneUnknownLy et al [], 2017ShimUpon opening of appAutomated by digital conversational agentOn the basis of individual and external factors (eg, time of day)None126.5Mak et al [], 2018Mindfulness-based ProgramWeeklyUnknownNoneSticker earning and alarm feature29.5Mak et al [], 2018Compassion-based programWeeklyUnknownNoneSticker earning and alarm feature32.2Manthey et al [], 2016Best possible selfWeeklyAutomated emailNoneNoneUnknownManthey et al [], 2016GratitudeWeeklyAutomated emailNoneNoneUnknownMitchell et al [], 2009Strengths interventionInstant accessN/ANoneInteractive features and automated email remindersUnknownMitchell et al [], 2009Problem-solving interventionInstant accessN/ANoneInteractive features and automated email remindersUnknownNeumeier et al [], 2017PERMAd programSequential accessAutomated access upon completion of step in programNoneNoneUnknownNeumeier et al [], 2017Gratitude programSequential accessAutomated access upon completion of step in programNoneNoneUnknownPheh et al [], 2020Brief MBIInstant accessN/ANoneNoneUnknownSchulte-Frankenfeld and Trautwein [], 2021BalloonSequential accessAutomated access upon completion of step in the appNoneA reminder was sent if a session was missed.40.2Shin et al [], 2020Gratitude writingInstant accessN/ANoneNone100Walsh et al [], 2019WildflowersSequential accessAutomated access upon completion of step in the appOn the basis of mood and stress levels recommendations were made for meditationsNone77.7

aActual engagement with content is based on the requested frequency of engagement with the intervention (eg, daily for 2 wk=14 d=100%) compared with the actual frequency of engagement in the intervention (eg, on average, participants engaged with the intervention on 5 d=35.7%).

bN/A: not applicable.

cMBI: mindfulness-based intervention.

dPERMA: Positive emotion, Engagement, Relationships, Meaning, Accomplishment.

Intervention Engagement

Overall, intervention engagement was suboptimal, below the required or recommended intervention engagement levels (). On average, participants engaged in 40.2% (median) of the recommended intervention sessions or days. Only few studies (3/19, 16%) contained optimal levels of engagement, engaging in the recommended intervention sessions or days or more [,,].

Studies attempted to improve intervention engagement in a variety of different ways ( and ), including (1) sending automated email reminders or notifications to use the intervention, (2) increasing participant motivation (eg, increasing awareness of potential benefits and using in-app reward earning features), (3) increasing habit formation, and (4) tailoring intervention content based on external factors (such as time of day) or internal factors (such as suggestion of a specific activity based on someone’s mood).

Although caution should be used when interpreting the impact of these strategies on the engagement with the intervention because of the variety and inconsistency in reporting, preliminary results imply that tailored content improves engagement more than interventions that use reminders (habit formation and prompts) or sticker earning features (nonspecific rewards). Furthermore, it seems that interventions that require little engagement—engaging once or 4 times in the intervention in total [,]—also allow for more optimal intervention engagement. This is in line with studies showing that engagement was generally highest at the start of the intervention and decreased with time.

Study Dropout and Attrition

Dropout occurred at any point throughout the study period when a participant failed to complete the research protocol associated with the digital intervention [].

On average, there was a 37% dropout rate (mean), which ranged from 0% to 85% in the studies (). Strategies used to reduce study dropout included monetary incentives, the intervention being a mandatory element of university courses, and follow-up of participants by sending email reminders.

There were a range of findings across studies on the association between participants’ demographic characteristics and dropout. One study found that male participants were more likely to drop out [], whereas others (2/19, 11%) found no difference [,]. Some studies (2/19, 11%) found that participants who remained in the study were older [,], although other studies (2/19, 11%) did not find this effect [,]. One study found that educational level was higher among participants who dropped out [], whereas another study did not find this effect [].

Several studies have compared whether baseline mental well-being was associated with dropout. Most studies (5/19, 26%) did not find any differences in baseline mental well-being levels between participants who did and did not drop out [,,,,]. However, 1 study found that participants with lower mental well-being and higher levels of anxiety, depression, and distress were more likely to drop out [], whereas another study found that participants with higher mental well-being and lower levels of anxiety, depression, and distress were more likely to drop out [].

Few studies (2/19, 11%) excluded participants from their analysis (considered them to have dropped out) if they did not adhere with the intervention content at a minimum required level [,]; most studies (17/19, 89%) included participants with any level of intervention engagement.

Outcomes

A variety of validated standardized questionnaires were used to measure mental well-being across studies, including the WHO 5 item mental well-being index and Warwick-Edinburgh Mental Well-Being Scale for mental well-being, Psychological Well-Being scale and Flourishing Scale for psychological well-being, and Satisfaction with Life Scale and Positive And Negative Affect Schedule for subjective well-being (). Nevertheless, the authors of 1 study created and validated their own mental well-being questionnaires, which included a combination of different measures. Although not included in this systematic review (as it is not considered the primary aim of mental well-being promotion), most studies (17/19, 89%) included additional outcome measures such as distress, depression, anxiety, and stress.

RoB Assessments

Generally, the RoB of the included studies was considered to be high (). High levels of dropout and nonadherence led to a high RoB in domain 2 of Cochrane’s RoB-2.0 tool. This domain assesses RoB because of deviations from the intended interventions (effect of adhering to the intervention) and leads to high RoB, as the included studies did not appropriately account for intervention nonadherence in their analysis. For example, the Cochrane RoB-2.0 tool recommends using an instrumental variable analysis or inverse probability weighting to appropriately account for nonadherence; however, none of the included studies conducted these analyses.

Table 5. Bias assessment using Cochrane’s risk of bias (RoB) 2.0 toola.Study, yearRandomization processDeviation from intended interventionMissing outcome dataMeasurement of outcomeSelection of the reported resultsOverall RoBbStudy 3 from Avey et al [], 2022Some concernscHighdLoweSome concernsHighHighBakker et al [], 2018Some concernsHighLowSome concernsSome concernsHighBrazier et al [], 2022LowHighLowSome concernsSome concernsHighChampion et al [], 2018Some concernsHighLowSome concernsLowHighChung et al [], 2021HighHighHighHighSome concernsHighStudy 1 from Di Consiglio et al [], 2021Some concernsHighLowSome concernsSome concernsHighGammer et al [], 2020LowHighLowSome concernsLowHighLiu et al [], 2021Some concernsHighHighHighHighHighLy et al [], 2017LowHighLowSome concernsSome concernsHighMak et al [], 2018LowHighLowSome concernsSome concernsHighManthey et al [], 2016LowHighSome concernsLowSome concernsHighMitchell et al [], 2009LowHighHighLowSome concernsHighNeumeier et al [], 2017Some concernsHighHighHighSome concernsHighPheh et al [], 2020Some concernsHighSome concernsHighSome concernsHighSchulte-Frankenfeld and Trautwein [], 2021LowHighHighSome concernsSome concernsHighShin et al [], 2020LowLowLowHighSome concernsHighWalsh et al [], 2019LowHighSome concernsHighSome concernsHigh

aThe National Institutes of Health bias assessment tool: before-after studies with no control group was used for study 2 from Di Consiglio et al [], 2021 (overall RoB: high) and Eisenstadt et al [], 2021 (overall RoB: high).

bThe overall RoB judgement for that specific study.

cSome concerns: indicates that the authors considered there to be some concerns with the RoB for that study on that specific domain of the Cochrane RoB-2.0 tool.

dHigh: indicates that the authors considered there to be a high RoB for that study on that specific domain of the Cochrane RoB-2.0 tool.

eLow: indicates that the authors considered there to be a low RoB for that study on that specific domain of the Cochrane RoB-2.0 tool.

Furthermore, domain 4 in the RoB-2.0 tool, assessing RoB in measuring the outcome, led to a high RoB because of the nature of the research being fully automated and digital. Self-report measures were used to digitally assess mental well-being; however, participants were aware of the intervention they received when self-reporting their mental well-being scores, as most studies (11/19, 58%) included a waitlist control group. Although active controls account for this issue, these control interventions also contained high levels of dropout and therefore might not be appropriate as a control group [].

A high RoB was also detected in studies because of the lack of general high-quality research practice. For example, several studies (7/19, 37%) did not provide any information regarding the randomization process, most studies did not preregister (12/19, 63%), and studies that did preregister (2/19, 11%) sometimes did not indicate their preintended analysis plan.

Intervention Effects

All studies included fully automated digital mental well-being interventions in the general population and were therefore considered sufficiently homogeneous for a meta-analysis. Methodological homogeneity was also considered, which led to a comparison across RCTs only, as these were considered sufficiently homogeneous for a meta-analysis. Considering the incredibly high range of missing values, a meta-analysis based on ITT data was considered inappropriate; therefore, we conducted a meta-analysis based on PP data instead. Nevertheless, this increases the risk of underestimating or overestimating the real effect, which should be considered when interpreting the meta-result. Full PP data were available for a subset of 12 studies. A random-effect model was applied, as different measures were used to measure the same multidimensional construct mental well-being. Average effect estimates were computed for each study, with negative affect scores reversed to ensure that a higher score in each study indicated elevated levels of mental well-being. SMDs, 95% CIs, and 2-sided P values were calculated.

Outlier

During data extraction, the negative affect score in the intervention group of 1 study [] was flagged by both reviewers as unexpectedly high, and further information was sought to identify what could potentially explain this unusually large result. Normative data for negative affect was mean 14.8 (SD 5.4) []; however, the negative affect score in the waitlist control group in this study was mean 26.98 (SD 5.19). When exploring this data further, no methodological or clinical differences could reliably explain this result in our opinion. In addition, when included in the meta-analysis, CIs were entirely outside the range of any other study, and heterogeneity was incredibly high (92%; ). Removing this study from the meta-analysis reduced the overall heterogeneity from 92% to 50%. Therefore, the study was considered an outlier and was excluded from the meta-analysis.

Main Effect

The pooled SMD, for the 12 trials, calculated using a random-effects model was 0.19 (95% CI 0.04-0.33; P=.01), indicating a small clinical effect in favor of digital mental well-being interventions (). There was substantial heterogeneity (I2=50%).

‎Figure 2. Per-protocol meta-analysis of fully automated digital interventions compared with control groups on mental well-being in the general population [-,,-,-,]. Sensitivity Analyses

As there was substantial heterogeneity (I2=50%), sensitivity analyses were performed to explore, interpret, and contextualize heterogeneity. First, intervention duration was explored using subgroups of interventions lasting up to 2 weeks (short), 2 to 6 weeks (medium), and >6 weeks (long).

A small significant effect was found for short interventions (SMD 0.24, 95% CI 0.04-0.45; P=.02) and medium interventions (SMD 0.29, 95% CI 0.05-0.52; P=.02); however, no effect was found for long interventions (SMD 0.02, 95% CI −0.22 to 0.26; Figure S1 in ). No significant levels of heterogeneity were found in any of the subgroups (all P>.05), and the subgroups substantially reduced the overall level of heterogeneity (I2=28.6%).

Another sensitivity analysis was performed to explore methodological heterogeneity across studies based on the comparator. We argue that placebo controls are not feasible in psychological interventions, considering the difficulty in isolating intervention components in psychological interventions []. Therefore, we grouped placebo controls under active controls in this review. A small significant effect was found in studies using a waitlist control as a comparator (SMD 0.28, 95% CI 0.07-0.50; P=.008), but no significant effect was found in studies using a placebo or active control as a comparator (SMD 0.05, 95% CI −0.08 to 0.18; P=.49; Figure S2 in ). No significant levels of heterogeneity were present in either of the 2 subgroups (all P>.05), although substantial heterogeneity remained in studies that used a waitlist control comparator (I2=53%).

Finally, a sensitivity analysis was performed based on the outcomes of mental well-being, psychological well-being, and subjective well-being. A small significant effect was found on subjective well-being (SMD 0.23, 95% CI 0.04-0.42; P=.02). However, no significant effect was found on mental well-being (SMD 0.14, 95% CI −0.12 to 0.40; P=.31) or psychological well-being (SMD 0.26, 95% CI −0.08 to 0.59; P=.14; Figure S3 in ). Despite reducing heterogeneity in subjective well-being and psychological well-being, substantial heterogeneity was found in mental well-being (I2=72%).

Reporting Bias