This study is one component of a multifaceted project funded by the National Institute of Mental Health (1R34MH110591) that supported the development of the MEYA training platform and the initial evaluation of its success in supporting practitioner fidelity and youth outcomes. MEYA, the expert rating measure of fidelity developed for it (MEYA-FS), and the participants in this study are described in detail in McLeod et al. (2022a) and Wood et al. (2024). These study elements are summarized below, with additional detail provided on the practitioner self-rating measure, MEYA Integrity Scale-Practitioner Version (MEYA-IS-PV), which is the focus of the present study.

Participants

Seven practitioners conducting psychotherapy or counseling with children or teens on the autism spectrum participated in this study. All participants resided in the USA, and their eligibility was determined through an initial phone screening. Eleven practitioners initially expressed interest, but three withdrew due to difficulty identifying eligible child participants, and one chose not to participate, leaving seven who completed the study.

Practitioners were recruited through organizations such as the Autism Speaks Autism Treatment Network, medical centers, parent support groups, and schools. Interested practitioners initiated contact with the study coordinator. Following written consent, practitioners provided study details to eligible families. Informed consent was obtained from both practitioners and parents, while children assented after receiving a full explanation of the study. Participating children scored above a T-score of 68 on the SRS-2, reflecting moderate to extensive autism-related challenges (Constantino & Gruber, 2012), and above the first percentile on the WISC-V Vocabulary subtest. Practitioners received $250 in gift cards for participation, while families received $100 split equally between parents and children. The study was conducted remotely through the university affiliated with the lead author.

Practitioners represented numerous behavioral health care disciplines (e.g., clinical psychology, counseling, social work) and had experience working with autistic youth. Eligibility criteria for youth participants included being 6–17 years old, having a clinical autism diagnosis, and engaging in outpatient intervention with a participating practitioner. Other restrictions were minimized to enhance external validity, although participant characteristics were evaluated and recorded. The principal investigator reviewed eligibility, and the study coordinator informed participants of their status. Participants were assigned code numbers to maintain confidentiality.

Practitioners provided services in various settings, including schools, mental health agencies, private practices, and university-based clinics. Background information was collected through open-ended interviews, though not all participants provided complete data. See Table 1 for the practitioners’ demographic information. Table 2 summarizes demographic information for the youth (client) participants.

Table 1 Practitioners’ demographic characteristics and experienceTable 2 Youth client demographic characteristics

Participant 1 holds an M.A. in Marriage and Family Therapy, is a Board-Certified Behavior Analyst (BCBA), and is a Registered Play Therapist (RPTS). With a primary behavioral orientation, she has worked with 200–300 autistic children, including school consultations. Participant 2 has a Ph.D. in Psychology and extensive experience in CBT, including exposure and response prevention (ERP) and mindfulness-based cognitive therapy. She is licensed and board-certified in CBT. Participant 3 is a pediatric psychologist with a Psy.D. in School Psychology and master’s degrees in clinical counseling and educational psychology. He is a board-certified behavior analyst with a behavioral orientation and 13 years of experience. Participant 4 is a licensed clinical social worker (LCSW) with 6 years of experience and a focus on CBT and Socio-Dramatic, Affective Relational Interventions (SDARI). Participant 5 has a master’s degree in clinical psychology and a background in ABA. She is a licensed marriage and family therapist who has been practicing behavioral and CBT approaches for 5 years. Participant 6 is a clinical psychology graduate student in her third year of doctoral training, with a master’s degree in psychology and experience in CBT and parent management training. Participant 7 is a graduate student with a master’s degree in psychology, experience in ABA, and a theoretical orientation combining CBT and dialectical behavior therapy (DBT).

Procedures and Randomization

After consent was obtained from practitioners and families and an initial assessment was completed, practitioners were randomized to one of four baseline conditions (2, 5, 6, or 8 sessions). This randomization process, conducted by the last author using a computerized table, ensured impartiality, as this investigator had no direct participant interaction. The study coordinator communicated baseline assignments to practitioners.

Baseline Phase

During the baseline phase, practitioners continued their usual psychotherapy or counseling approaches for the designated sessions. The research team recorded sessions using video devices and securely recorded them for later coding.

MEYA Phase

Following the baseline period, practitioners received login credentials for the free/open-access MEYA website (www.meya.ucla). They completed a 2-h training video introducing MEYA and incorporated its guidance into their sessions. The MEYA platform provided just-in-time training, offering brief video tutorials on EBPs tailored to the specific needs of each youth. Practitioners were asked to conduct at least eight MEYA-guided sessions unless therapy concluded earlier. Optional consultation calls with a clinical expert were available to address questions about interventions and/or the MEYA platform.

Throughout the MEYA phase, practitioners retained autonomy in session planning and continued video-recording sessions for research purposes. MEYA facilitated a practical approach to learning EBPs, integrating training directly into clinical workflows. MEYA includes six primary clinical focuses: disruptive behavior, negative affect (e.g., anxiety and depression), repetitive/rigid behavior, peer engagement, conversation and friendship, and self-care skills. Modules incorporate content for school- and clinic-based settings, with adaptations for children with minimal expressive language or varying cognitive levels. Practitioner feedback during development ensured usability and relevance. MEYA offers tools like the Session Selector/Planner and the MEYA Chart to streamline intervention planning. The Planner uses caregiver input from an initial assessment of youth’s goals or challenges, scored on a 10-point scale. Weekly updates guide algorithmic recommendations for the most appropriate modules. Practitioners begin with core sessions (e.g., self-management) and progress through clinical areas based on priority ratings or progress achieved.

MeasuresMEYA Integrity Scale-Practitioner Version (MEYA-IS-PV)

The MEYA-IS-PV items were designed to parallel the content of the MEYA-FS. Steps taken to establish the content validity of the MEYA-FS items included review of treatment protocols and expert review (see McLeod et al., 2022a for details). The 10 MEYA-FS model items were used for the MEYA-IS-PV: Cognitive, Perspective Taking, Self-Management, Peer Skills, Exposure, In-session Reinforcement, Goal Charts, Home-based Rewards, Positive/Preventative, Parent Training. One MEYA-FS Delivery item was used: Role-Playing. Finally, one MEYA-FS item, Homework, was split into two items for the MEYA-IS-PV: Homework Assigned, Homework Reviewed. This resulted in a total of 13 items. The MEYA-IS-PV items were reworded into a rating form to provide information on how much each practice was used in a specific treatment session. The items were scored on a 5-point scale with the following anchors: 1, not at all; 3, some; and 5, a lot. Hence, higher scores reflect more extensive use of relevant EBPs in the session. This scoring strategy has been used in previous self-report treatment fidelity measures (see Hogue et al., 2014; McLeod et al., 2022b).

After each session (both baseline and MEYA phases), practitioners made self-report ratings on the MEYA-IS-PV according to the following instructions: “Indicate the extent to which you used each technique in today’s session. In making your ratings, please consider both the frequency and thoroughness with which you used each technique. For any techniques that you did not use, please circle ‘1’ for ‘not at all.’” Sample items include the following: “Taught or encouraged the child to identify others’ perspectives,” “Encouraged the child to participate in exposure(s) to specific challenging situations to build emotion-related coping skills and reduce target symptoms,” “Taught or encouraged parents to use rewards and privileges at home to achieve treatment goals,” and “Taught or encouraged child to practice positive peer relationship skills such as conversations, joining games, sharing, or hosting playdates in session and/or at home.”

Modular EBPs for Youth on the Autism Spectrum Fidelity Scale (MEYA-FS)

As detailed in McLeod et al. (2022a) and Wood et al. (2024), the MEYA-FS is a 16-item measure designed to evaluate practitioner adherence and competence in EBPs for autistic youth. Parallel adherence and competence items assess universal elements common to most evidence-based programs (e.g., assigning out-of-session tasks) and core practices (e.g., perspective-taking training, exposure) related to CBT and BI practices for autistic youth. Coders review session recordings and rate adherence on a 7-point scale from 1 (“not at all”) to 7 (“extensively”).

The MEYA-FS Change in Adherence and Competence subscales track practitioners’ improvement in applying EBPs within modular interventions. These subscales estimate fidelity regardless of the modules delivered. Scores are derived from four item clusters: common EBP practices (e.g., modeling, reinforcement), social-communication components (e.g., perspective-taking), behavioral procedures (e.g., self-management, exposure), and generalization strategies (e.g., homework, goal charts). The present study used MEYA-FS Change in Adherence scores to compare with practitioner self-reports on the MEYA-IS-PV (below), which focuses on adherence (not competence). Higher subscale scores reflect greater adherence to autism-related EBPs.

Three doctoral students made MEYA-FS ratings after a systematic training phase (see Wood et al., 2024, for details). Training with the last author entailed didactic instruction, a review of the scoring manual, session discussions, and coding exercises. To be certified for independent coding, raters met reliability criteria for each item on 30 training tapes. Each session was double-coded, and raters were naïve to study hypotheses. As Wood et al. (2024) reported, inter-rater reliability was ICC (2,2) = .79.

Youth Characteristics

Baseline assessments were used to evaluate child/youth clinical and developmental profiles. The Social Responsiveness Scale 2 (SRS-2; Constantino & Gruber, 2012), a 65-item parent-reported scale, measured autism-related challenges like social awareness and preoccupations. This tool is reliable and distinguishes children on the autism spectrum from non-autistic children. To assess oral language abilities, youth completed the Vocabulary subtest of the WISC-V, a validated ability test for youth aged 6 to 16 (Wechsler, 2014), over the phone. All participants were native English speakers.

Youth Clinical Needs

The Youth Top Problems scale (YTP), a personalized symptom assessment sensitive to treatment effects in autistic youth (Weisz et al., 2011; Wood et al., 2022), was used at baseline. Parents identified the most pressing issues within six clinical areas during a semi-structured interview and rated severity on a 0–10 scale. Weekly ratings of up to 12 prioritized problems or goals provided data for personalized session planning as well as assessment of treatment outcomes.

Data Analysis

Psychometric analyses of practitioner self-reports on the MEYA-IS-PV addressed inter-item correlations, convergence with expert ratings on the MEYA-FS, and two indices of concurrent validity: responsiveness to training (within the context of the multiple baseline design) and association with youth outcome (on YTP ratings). Linear mixed modeling (LMM), which accounts for the patterns of nesting in this study (e.g., sessions nested within practitioners), was used to test hypotheses pertaining to convergent and concurrent validity using SPSS software (Version 29). LMM is a full-information analytic procedure that estimates the treatment effect in multiple baseline studies using the Satterthwaite method coupled with an autoregressive level 1 error structure (Ferron et al., 2009). Visual analyses and phase-related nonoverlapping frequency tabulations of the multiple baseline practitioner self-report data were also conducted.