Introduction

Medication adherence presents a significant and pressing challenge worldwide across health care systems []. This problem spans across various medical conditions, from chronic illnesses such as hypertension, diabetes, and cardiovascular disease to acute infections []. Nonadherence to medication regimens can lead to a cascade of problems, including decreased treatment effectiveness, a higher risk of disease progression or complications, medication resistance, increased health care costs, and a reduced overall quality of life for patients []. It also places an additional burden on health care providers, who must constantly navigate the complexities of patient adherence to optimize treatment outcomes. Addressing medication adherence is not only a matter of improving individual health but also a vital step toward more efficient health care systems and improving population health.

One notable area where medication adherence could have a significant effect on public health is with smoking cessation medications. Tobacco smoking is not only extremely addictive [], but it also damages almost every organ in the body [] and is a leading cause of health inequalities []. Despite decades of efforts to reduce the smoking prevalence in Canada [], over 4 million (15%) Canadians smoked in 2020 []. Tobacco use remains the leading cause of preventable death in Canada [], and following smoking cessation, the risk of developing tobacco-related diseases gradually decreases over time [].

Varenicline, a partial agonist of the α-4 β-2 nicotinic receptor, is one of the most effective smoking cessation medications associated with a doubling of long-term quitting with 12 weeks of continuous treatment []; however, adherence is a major barrier to achieving and maintaining abstinence [-]. Among those who take varenicline for smoking cessation, over one-third stop taking the medication by the second week of treatment, and these patients are significantly less likely to quit smoking [-]. In addition, several studies that have examined the correlation between sociodemographics and adherence to varenicline have found male sex [,], older age [,-], White race [,], and higher education to be associated with better adherence []. Conversely, 1 study found that greater age was associated with poorer adherence [].

One of the most effective strategies to improve adherence to medication is tailoring behavioral interventions []. Research has also shown that the use of technological solutions is crucial to providing personalized, cost-effective, and evidence-based interventions []. One such digital health solution is a web-based program integrating artificial intelligence (AI) conversational agents, or “health bots,” that can be helpful in improving adherence to medication []. Health bots have greater functionality compared with applications providing simple text message reminders since they can track medication intake, provide information about side effects, and answer medication-related questions [,].

The “ChatV” health bot is an example of such a digital health solution that we developed to improve adherence to the smoking cessation medication varenicline []. We used a theory-based, patient-centered approach through the Discover Design Build and Test framework [] to design the health bot; we conducted a rapid review to explore the barriers and facilitators to varenicline adherence and interviewed 20 patients and 19 health care providers to understand similar barriers and facilitators as well as features to be included in a health bot that may improve varenicline adherence. The data from these sources were analyzed using the capability, opportunity, motivation-behavior change (COM-B) [] model of behavior change and its associated framework (theoretical domain framework) [] and taxonomy (behavior change techniques taxonomy) [] to identify the features to be included in the health bot. We used the Wizard of Oz methodology [] to build a library of responses that were programmed in the health bot to answer users’ questions. We are currently beta testing the minimal viable product that has the following features: (1) scheduling and receiving reminders for varenicline dosing; (2) setting a smoking quit date; (3) answering questions related to varenicline (eg, guidance for missed doses) and smoking cessation (eg, strategies to reduce cravings); (4) providing information and strategies to manage side effects [,,]; (5) tracking medication use and smoking; and (6) encouraging participants to increase their motivation to maintain medication intake and continue their quit attempts. Throughout the development of the health bot, a health equity lens was adopted to address any disparities in varenicline use and health bot features between different demographic characteristics.

In this protocol, we detail our plans to examine the uptake, usability, acceptability, and appropriateness of ChatV (ie, feasibility) [] and to determine whether progression to a fully powered randomized controlled trial (RCT) is warranted.

MethodsDesign

We will conduct a nonrandomized, single-arm feasibility study.

Participant Recruitment

We will recruit 40 participants. In order to ensure diversity with respect to age, gender, race, and socioeconomic status, we will purposefully recruit participants through a variety of methods, including the research boards of Centre for Addiction and Mental Health (CAMH), where recruitment flyers are posted for patients to explore research opportunities; health care providers at CAMH programs (eg, Smoking Treatment for Ontario Patients [STOP] program, Aboriginal services, and Rainbow services) and partnering organizations (Rainbow Health Ontario); the STOP program’s list of people who are seeking treatment for smoking and who have consented to be contacted for future research studies; and social media and community boards. Research staff members will conduct a prescreening for those who are interested, and if eligible, participants will be enrolled in the study.

Inclusion Criteria

The inclusion criteria include individuals who are seeking treatment for their tobacco use who smoke 10 or more cigarettes a day, are willing to set a quit date in the next 30 days, are willing to take varenicline for 12 weeks, are aged 18 years, speak and read English, have a smartphone with a data plan, report being committed to answering questions during follow-up, and live in Ontario.

Exclusion Criteria

The exclusion criteria include having contraindications to varenicline use, being pregnant, planning to become pregnant, breastfeeding, or participating in the co-design phase.

Ethical Considerations

The CAMH Research Ethics Board (REB #050/2022) has provided approval to conduct the study, and all participants will provide written informed consent before participation. It has been registered at ClinicalTrials.gov (NCT05997901). Any amendments to the protocol are noted, and substantial changes will be reported to the ethics committee. All interested participants will receive an information sheet and a consent form describing the study and providing sufficient information for the participant to make a decision regarding study participation. A research staff member will arrange a personal meeting with the prospective participants and inform them again about all study procedures, benefits, risks, and their rights regarding study participation. Afterward, the consent form is signed by both the participant and the research staff before the participant undertakes any study procedure. The informed consent process is documented in Research Electronic Data Capture (REDCap; Vanderbilt University).

All study-related documents and data will be held in strict confidence and stored on CAMH servers that will follow CAMH policies and procedures to ensure participant privacy and confidentiality. Any data collected as part of the participants’ completion of the electronic consent form will be securely stored in REDCap, a CAMH-approved data collection tool.

In addition to free medication (for 12 weeks), participants will receive an honorarium in the form of a CAD $30 (US $22) gift card after completion of the 4-week follow-up survey and a CAD $40 (US $30) gift card after completion of the 12-week interview.

Procedures

The research team will inform potential participants about the study and evaluate if they meet the inclusion criteria. If they qualify, participants will receive a consent form and arrange a phone-based discussion to obtain consent. After obtaining consent, participants (n=40) will complete an internet-based questionnaire covering sociodemographic characteristics, nicotine dependence [], and an adapted scale to measure varenicline adherence self-efficacy [-]. After completing the questionnaire, participants will have an internet-based or in-person visit with a health care provider for eligibility confirmation and to obtain a prescription for varenicline for the first 4 weeks (1 tablet [0.5 mg] daily for the first 3 days, 1 tablet [0.5 mg] twice daily for the next 4 days, and 1 tablet [1 mg] twice daily for 3 weeks). In addition, research staff will show participants how to use the health bot on their phones. Medication will be provided in person or, in the case of internet-based contacts, mailed to the participants. A follow-up visit will be scheduled at 2 weeks, where health care providers will assess the participant’s tolerance of varenicline and provide a prescription for the remaining 8 weeks (1 tablet [1 mg] twice daily or a different dose depending on the tolerance of each participant). Any unanticipated problems (ie, adverse events that are unexpected in terms of severity, nature, or frequency; are related or possibly related to participation in the research; and suggest that the research places other research participants at greater risk of harm) will be documented and reported to the research ethics board, and adverse drug reactions will be reported to the market authorization holder (Apotex) as applicable.

Participants will be required to complete surveys at 1, 4, 8, and 12 weeks. When participants join the study, they will be able to opt to respond to these questions using either the health bot or an internet-based platform. Those who do not complete the internet-based or health bot follow-up within the designated time frame will receive a phone call from our research staff. Additionally, after 12 weeks of joining the study, participants will be invited to participate in a 1-hour semistructured phone interview.

MeasuresOverview

The taxonomy of implementation outcomes of Proctor et al [] guides our selection of implementation outcomes ().

Table 1. Summary of measures.MeasuresSurvey timelineInterviewAnalytics
BaselineWeek 1Week 4Week 8Week 12Week 12Weeks 1-12Demographics✓





Adoption




✓✓Acceptability
✓a✓a✓a✓a✓
Appropriateness
✓b✓b✓b✓b✓
Fidelity




✓
Usability
✓c✓c✓c✓c✓
Medication Adherence
✓d✓d✓d✓d✓
Smoking Status
✓✓✓✓✓

aAcceptability of Intervention Measure.

bIntervention Appropriateness Measure.

cSystem Usability Scale.

dTimeline Followback.

Acceptability

Acceptability is the extent to which an innovation is agreeable, palatable, or satisfactory to a stakeholder []. In order to measure acceptability, we will use the Acceptability of Intervention Measure (AIM) in the follow-up surveys as well as in the semistructured phone interviews.

Appropriateness

Appropriateness is the perceived fit or compatibility of an innovation with a practice setting or context []. In this study, appropriateness will be assessed on an individual basis, including factors such as alignment with users’ attitudes, needs, and background, as measured in the 12-week survey with the Intervention Appropriateness Measure (IAM), a validated 4-item intervention appropriateness scale [], and explored in depth in the semistructured interviews.

Fidelity

Fidelity is the extent to which an intervention is used as intended []. For this study, in the semistructured interviews, we will investigate the functions of the health bot that participants used; the purposes for which they used it (such as answering questions or seeking support); the frequency of usage; and the quality, specifically whether the health bot provided accurate information.

Adoption

Adoption is the intention, decision, or initiation of use for an evidence-based practice, characterizing it at the level of the provider or organization []. Since the concept of adoption is closely related to the actual use of systems, researchers investigating behavioral intervention technologies, such as the health bot, have extended this level of analysis to include consumer behavior []. In this study, we will measure the adoption of the health bot by analyzing analytics data throughout the 12 weeks that the participant is scheduled to take varenicline. These data will allow us to evaluate the timing (day and time) of user interactions with the health bot, the specific features used, and the duration of engagement. Thus, we will have objective data on the actual adoption of the health bot by each participant. Additionally, during the semistructured interviews, we will probe for participants’ subjective perceptions of their use of the health bot.

Usability

Usability will be assessed using the System Usability Scale (SUS) []. The SUS is a 10-item measure assessing usability and user satisfaction with technology.

The AIM, IAM, and SUS all have items that are rated on a 5-point Likert scale (1=completely or strongly disagree to 5=completely or strongly agree).

Medication Adherence and Smoking Status

Participants will be asked about their smoking status [] and adherence to varenicline using the Timeline Followback (TLFB) [] in the surveys collected at 1, 4, 8, and 12 weeks. Participants will report the number of pills taken since the previous assessment. Participants who log their varenicline use in the health bot will be able to refer to these data when filling out the TLFB. We chose the TLFB method for assessing adherence because the 12-week TLFB measure demonstrates a moderate correlation with saliva varenicline levels and is widely regarded as the most practical measure of varenicline adherence []. In order to assess the feasibility and accuracy of measuring adherence using TLFB, participants will also be asked to send pictures of their varenicline blister packs at 1, 4, 8, and 12 weeks. This additional measure will also help explore which method to assess adherence should be used for any future research studies (eg, RCT). Smoking abstinence will be defined by a negative response to the dichotomous 7-day point prevalence question, “Have you had a cigarette, even a puff, in the last 7 days?”

Analysis of Survey Data

We will calculate descriptive statistics, including means, medians, ranges, and SDs, for the demographic data as well as for the AIM, IAM, and analytics data captured by the health bot internal logs. All scores will be analyzed using the appropriate guidelines provided by Weiner et al [] (acceptability and appropriateness) and Brooke [] (usability). Consistent with previous studies, adherence will be defined as taking ≥80% of the prescribed varenicline [-]. We will not treat adherence as a continuous variable because it is likely to be strongly bimodal and is not necessarily linearly associated with the cessation outcome. We will calculate the percentage of participants who were adherent at week 12 and who reported not having smoked in the last 7 days. These results will serve as progression criteria to determine whether a full-scale RCT is warranted. Although the sample size will be insufficient for meaningful regression analysis, especially given the potentially nonlinear effects of age, we will also test for differences across age and gender groups using 2-tailed t tests and chi-square tests. The power for these comparisons will be low; therefore, this analysis will be an exploratory evaluation of large adherence or outcome differences that may be important to future study design, and interpretation will focus on observed differences and CIs.

Analysis of Semistructured Interviews

We will use framework analysis [] to analyze the data. Specifically, we will use the implementation outcomes of Proctor et al [] with an intersectional lens as the a priori framework. Framework analysis is a method used to draw descriptive and explanatory conclusions from qualitative data and includes the following six steps: (1) verbatim transcription of interviews, (2) familiarization of interviews, (3) coding and labeling the data, (4) developing a framework by grouping and categorizing codes, (5) indexing transcripts using the developed framework, and (6) reducing and charting data into a matrix. Interview findings will help us contextualize the findings from the surveys and the information collected from the use of the health bot. For example, we will understand why the health bot was or was not used (adoption); if it was used as intended (fidelity), what about the health bot the participants found agreeable and what they did not (acceptability); what features, if any, aligned with participants’ attitudes, needs, and background (appropriate); and how other features would need to change in order to align with their needs. In addition, we will be able to understand how social identity (eg, age and gender) and power structures (eg, ageism and sexism) played a role in these results. This will provide us with important information to improve the health bot through an equity lens.

Triangulation of Findings

Once the survey data and interviews are analyzed, we will triangulate the results by listing all the findings on the same page to identify instances where findings from each method align (convergence), provide additional insights on the same topic (complementarity), or seem to conflict with one another (discrepancy or dissonance) [,]. Triangulating the data will allow us to contextualize the quantitative findings and explore “intermethod discrepancy” that might lead to a better understanding of how the health bot works and the modifications needed.

Exploring the Feasibility of the Approaches on Recruitment, Retention, and Data Collection

This study, using “stop, amend, and go” progression criteria for pilot studies [], will help determine if proceeding with an RCT is a logical next step and whether additional adjustments are necessary. We will only conduct a full-scale RCT if none of the criteria outlined in meet the “stop” criterion (unless we see that there are contextual issues that can be modified to overcome the identified problems). If 1 or more of the concepts in meet the “amend” criterion, the trial will proceed to an RCT only if the issues can be addressed. If all concepts in satisfy the “go” criterion, we will proceed with planning an RCT.

Table 2. Criteria to decide if a randomized controlled trial is warranted.CriteriaStopAmendGoPatient recruitmentIf <30 patients are recruited within 12 monthsIf 30-40 patients are recruited within 12 monthsIf 40 patients are recruited within 12 monthsPatient retentionIf <25% of patients answer the 12-week surveyIf 25.1%-79.9% of patients answer the 12-week surveyIf ≥80% of patients answer the 12-week surveyMissing data, medication adherence, or smoking statusIf >60% of adherence data and smoking status are missingIf 21%-59% of adherence and smoking status data are missingIf ≤20% of adherence and smoking status data are missingAcceptability and appropriatenessIf  mean score for the AIMa or IAMb is ≤8If  mean score for the AIM or IAM is 9-12If  mean score for the AIM or IAM is ≥13UsabilityIf mean SUSc score is ≤40If mean SUS score is 41-67If mean SUS score is ≥68AdoptionIf mean use is ≤ 20 timesIf mean use is 21-79 timesIf mean use is ≥80 times

aAIM: Acceptability of Intervention Measure.

bIAM: Intervention Appropriateness Measure.

cSUS: System Usability Scale.

Results

Participant enrollment for the study will begin in January 2024.

Discussion

Despite the proven effectiveness of varenicline in aiding smoking cessation [], a significant number of individuals attempting to quit continue to smoke because they struggle with adhering to their varenicline regimen [-]. While there are proven strategies to help people adhere to their medications, they have been hard to implement. Digital technology for medication adherence has started to emerge [,]; whether these tools can be feasibly implemented and adopted, especially within diverse and disadvantaged populations, remains largely unknown.

In this protocol, we outline the methodology to assess the uptake, usability, acceptability, and appropriateness of ChatV and assist in determining whether it is appropriate to advance to an RCT to gauge the effectiveness of the intervention. This study will address many limitations of previous mobile health trials [,], including using an intervention that clearly outlines the behavior-change theory used and the intervention content. In addition, this study provides well-outlined a priori criteria for progressing to an RCT. This will enhance its replicability. Furthermore, this study will provide insights into potential mechanisms underlying the intervention’s impact, shed light on the effectiveness, or ineffectiveness, of its various components, and offer possible explanations.

In summary, the results of this feasibility study will provide much-needed data and insight into the potential implementation of ChatV to augment adherence to smoking cessation medications. In addition, this project has the potential to increase our understanding of how digital tools can improve adherence to other medications in a cost-effective manner.

This research is funded by a Proof of Concept Intervention Grant in Primary Prevention of Cancer (action grant) of the Canadian Cancer Society and the Canadian Institutes of Health Research-Institute for Cancer Research (grant 707218) and a Canadian Institutes of Health Research Project grant (funding reference PJT 180405; ). The authors would like to thank MEMOTEXT for their contribution to the development of the ChatV health bot.

The data sets generated during this study will be available from the corresponding author upon reasonable request.

NM conceived the study with the support of JR, PS, SV, LZ, and MR. NM (lead), KM, ME, SH, RTAK, and PS have shaped the intervention (health bot). NM (lead), KM, and ME were involved in gaining ethical approval. NM and KM wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

The authors declare that they have no conflicting interests with respect to this manuscript; however, some authors have general disclosures to report. NM reports receipt of funding from the Canadian Institutes of Health Research, the Canadian Cancer Society, and the discovery fund of the Centre for Addiction and Mental Health (CAMH). PS reports receipt of funding from the Canadian Institutes of Health Research, the Canadian Cancer Society, Pfizer Inc, Pfizer Canada, and the Ontario Lung Association. PS also reports that, through an open-tender process, Johnson & Johnson, Novartis, and Pfizer Inc are vendors of record for having provided smoking cessation pharmacotherapy for research studies at free or discounted rates. PS holds the Vice-Chair, Research, and Giblon Professor in Family Medicine Research, a university-named professorship at the University of Toronto. OCM reports receiving grant and salary support from the Centre for Addiction and Mental Health and the Department of Family and Community Medicine at the University of Toronto through the New Investigator Award.

Edited by A Mavragani; The proposal for this study was peer reviewed by the Canadian Institutes of Health Research. See the Multimedia Appendix for the peer-review report; submitted 10.10.23; accepted 23.11.23; published 11.12.23

©Nadia Minian, Kamna Mehra, Mackenzie Earle, Sowsan Hafuth, Ryan Ting-A-Kee, Jonathan Rose, Scott Veldhuizen, Laurie Zawertailo, Matt Ratto, Osnat C Melamed, Peter Selby. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 11.12.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.