Breakout D – May 16, 2013
1. Fidelity Measurement in the Real World: Feasibility of BECCI & MITI for Motivational Interviewing in Child & Youth Mental Health
Presenter: Melanie Barwick, PhD
Authors: Melissa Kimber, MSW, PhD(c), Raluca Barac, MA, PhD, Melanie Barwick, PhD, CPsych, Hospital for Sick Children, Toronto, Canada
Abstract: Treatment fidelity is an essential element of good scientific research and clinical practice (Waltz et al., 1993). Treatment fidelity helps researchers and practitioners draw solid conclusions about treatment effects, inform on implementation of evidence-based practices (EBPs) so that cross-site comparisons can be made, and monitor therapist training and need for additional training and supervision. Despite its undisputed importance, treatment fidelity is often neglected in practice. Schulte et al. (2009) noted that fidelity monitoring is least likely to occur precisely when there is high risk of compromising fidelity, namely during EBP implementation. Fidelity neglect stems from lack of theoretical knowledge and implementation guidelines and the increased costs associated with its use (Perepletchikova et al., 2009). METHOD: We addressed the issue of fidelity in the context of implementing Motivational Interviewing (MI) in four child and youth mental health organizations in Ontario, Canada. Across the four organizations, 20 clinicians audiotaped monthly therapy sessions with their clients to capture practice at several time points: 3 months leading to MI training, while receiving coaching support over 9 months, and 3 months following support. All sessions were coded using the Behavioral Change Counseling Index (BECCI; Lane et al., 2005) and 50% of the sessions were coded by an expert coder using the Motivational Interviewing Treatment Integrity scale (MITI; Moyers et al., 2005). Both instruments measure therapists’ competence in MI and have acceptable psychometric properties (Wallace et al., 2009). However, the two instruments differ in practical and economical aspects: BECCI is a brief tool, to be scored in one pass and requiring minimal amounts of training, whereas MITI is more elaborate, shown to be reliable when used by expert raters, and requires relatively longer training. The brevity of the BECCI makes it a very appealing instrument for practice settings, with great potential for use in mental health settings to ensure MI fidelity. This is important because fidelity assessment will only become a reality if it is simple and practicable. To date, the BECCI has only been tested with data from simulated therapy sessions and how it stands up when tested in real world clinical practice or how it relates to other MI fidelity instruments is less known. Thus, the present study examined (a) the concordance between the MITI and BECCI and (b) the extent to which the two fidelity measures detect change in therapists’ competence following training and coaching in MI. These findings have significant implications for implementing fidelity checks as standard practice in mental health.
2. Comparisons Among Six Methods for Measuring Fidelity: Implications for Research & Practice
Presenter: Kristin Duppong Hurley, PhD
Authors: Kristin Duppong Hurley, PhD,1 & Mark Van Ryzin, PhD2
1University of Nebraska-Lincoln; 2Oregon Social Learning Center
Abstract: Developing valid, reliable, and cost-effective fidelity assessment tools that can be used in practice-settings is a challenge for many evidence-based interventions. One of goals of this pilot study was to compare the psychometrics of six methods for measuring the fidelity of an adaptation of the Teaching Family Model. These six methods of assessing fidelity included external observations, internal-agency observations, supervisor ratings, staff self-ratings, youth ratings, and archival data. The study included 145 youth with disruptive behavior disorders, 120 direct-care staff, and 16 supervisors. Fidelity process data and youth mental health outcome data were collected longitudinally. We will briefly discuss the findings of this pilot study comparing the fidelity measurement approaches. Individually, each fidelity approach had acceptable psychometric properties and their ratings were correlated over time. However, the different assessment approaches were not strongly correlated among each other. Interestingly, the supervisor ratings had some issues with bias in ratings. By comparing the multiple assessments, certain supervisors were identified as likely being stricter in their ratings, causing an unexpected inverse correlation with youth outcomes. Only youth ratings of fidelity and some of the archival data were predictive of positive youth mental health outcomes. Implications and directions for future research will be discussed.
3. An Update on Project BEST (Bringing Evidence-Supported Treatments to South Carolina Children & Families): Challenges to Measuring Provider Fidelity
Presenter: Rochelle F. Hanson, PhD
Authors: Rochelle F. Hanson, PhD,1 Benjamin E. Saunders, PhD,1 Libby Ralston, PhD,2 Michael de Arellano, PhD,1 & Angela Moreland, PhD1
1National Crime Victims Research & Treatment Center, Medical University of South Carolina; 2Dee Norton Low Country Children’s Center
Abstract: This presentation provides an update on Project BEST (Bringing Evidence-Supported Treatments to South Carolina children and families; funded by Duke Endowment), an ongoing statewide initiative designed to support the dissemination and implementation of Trauma-focused Cognitive Behavioral Therapy (TF-CBT). Project BEST utilizes the Community-Based Learning Collaborative (CBLC) dissemination/implementation model to build community capacity to deliver and sustain trauma-informed services to abused children and their families. Since its onset, we have completed three CBLCs, and two are nearing completion. These have involved 477 clinicians, brokers, and senior leaders from 105 different agencies serving 38 of South Carolina’s 46 counties. One of our key goals is to train mental health providers in the delivery of TF-CBT, and an ongoing challenge is to determine the most feasible, cost effective ways to measure therapist fidelity to the model. After providing an update on Project BEST activities to date (e.g., total number of participants who completed all training requirements; pre/post treatment outcome data for TF-CBT training cases), the focus will be to discuss our measure of therapist fidelity to TF-CBT, including factors associated with TF-CBT fidelity; and preliminary findings on the relations among therapist fidelity and child/family treatment outcomes. The presentation will conclude with a discussion of challenges encountered in this statewide initiative, lessons learned, and future plans.