Observational Assessment of Fidelity to a Family-Centered Prevention Program: Differentiation and Predictive Validity

Saturday 2:30 – 3:45 Breakout F5

Presentor: Justin D. Smith

Justin D. Smith, Baylor University; Kimbree Brown, University of Oregon; Karina Ramos, University of Oregon; Nicole Thornton, Prevention Science Institute, University of Oregon; Thomas J. Dishion, REACH Institute, Arizona State University; Elizabeth A. Stormshak, Prevention Science Institute, University of Oregon ; Daniel S. Shaw, University of Pittsburgh; Melvin N. Wilson, University of Virginia


Assessing the fidelity with which evidence-based programs are implemented has critical implications for the validity of the findings  and for translation and scale out of interventions. A breakdown in treatment fidelity inhibits the ability to draw valid inferences regarding the nature of a putative treatment effect. In this study, we conducted a randomized experiment to determine whether ratings of fidelity to the Family Check-Up (FCU) provided on the COACH rating system could differentiate levels of training in the model. Further, we evaluated the predictive validity of fidelity ratings on clinically meaningful outcomes. Trained coders observationally rated 75 intervention sessions for fidelity. Three groups of 25 sessions each were randomly selected from the intervention arms of an efficacy trial, an effectiveness trial, and from treatment as usual provided by licensed therapists with no training in the FCU model. Results indicated declining ratings of fidelity corresponding to the level of the therapists’ training in the FCU. Further, COACH ratings were predictive of child conduct problem ratings at posttreatment, controlling for baseline levels, for only those families who received the FCU; there was no relation between COACH scores and outcomes in the treatment as usual condition. These findings indicate that the COACH system reliably differentiates the level of fidelity to the FCU and that scores are predictive of child outcomes. Post hoc analyses indicates that conceptual accuracy to the FCU was the primary substantive difference between levels of training and also could explain the lack of predictive validity of the COACH score in the treatment as usual group. Future directions of fidelity assessment are discussed.

Strategies and Challenges in Housing First Fidelity: A Multistate Qualitative Analysis

Saturday 2:30 – 3:45 Breakout F5

Presentor: Mimi Choy-Brown and Emmy Tiderington

Mimi Choy-Brown, Silver School of Social Work, New York University; Emmy Tiderington, Silver School of Social Work, New York University; Bikki Tran Smith, Silver School of Social Work, New York University; Deborah K. Padgett, Silver School of Social Work, New York University



Pathways Housing First (PHF), an evidence-based model of permanent housing and supportive services for homeless adults, has been widely disseminated in the United States and internationally. However, model fidelity has varied widely across settings.  Less understood are on-the-ground experiences of providers at different stages of implementation and organizational contextual factors influencing fidelity. Further knowledge of these provider experiences can inform more efficient and effective strategies to support fidelity in multiple contexts. The primary aim of this study was to understand the strategies and challenges providers encounter in the uptake and fidelity of PHF. Data were derived from a NIMH-funded study investigating perspectives of front-line staff and supervisors. Four focus groups were conducted with staff at PHF sites in two East Coast cities (total n=24). Thematic analysis (Boyatzis, 1991) was utilized to code and develop themes regarding challenges to model fidelity. Themes identified were 1) feasibility of the workload; 2) importance of supervision; 3) impact of technology; and 4) availability and types of training. Providers adapted limited resources to respond to barriers to fidelity (e.g. turnover) with some success. Qualitative focus groups were found to be an efficient method for assessment of proximal factors influencing model fidelity.


Developing Treatment Fidelity Rating Systems for Psychotherapy Research: Recommendations and Lessons Learned

Saturday 2:30 – 3:45 Breakout F5

Presentor: Kevin A. Hallgren

Kevin Hallgren, University of Washington; Shirley Crotwell, Boston Veteran Affairs Health Care System; Rosa Muñoz, University of New Mexico; Becky Gius, University of New Mexico; Benjamin Ladd, Reed College; Barbara McCrady, University of New Mexico ; Elizabeth Epstein, University of Massachusetts Medical School in Worcester


Measuring fidelity to evidence-based treatments is a key component of dissemination and implementation research.  However, developing reliable, valid, and clinically-relevant treatment fidelity measures remains a challenge.  Although much of the literature has focused on theoretical and psychometric aspects of measure development, the literature often omits practical considerations for developing and using fidelity measures.  The present study describes the development and testing of a treatment fidelity rating system used in couple-based alcoholism treatment.  Over a three-year period, seven coders received extensive training and rated 74 components of treatment fidelity across 284 psychotherapy sessions.  During this presentation, the theoretical model underlying the instrument and psychometric properties will be briefly described.  However, a considerable portion of the presentation will focus on practical recommendations and lessons learned from our work, which we hope will inform future measure development.  Major themes include challenges in (1) measure development (e.g., adapting existing fidelity measures for new treatments), (2) defining “treatment integrity??? (e.g., conceptual and practical difficulties in rating various therapist behaviors), (3) process improvement (e.g., procedures for improving quality and efficiency of coder training and ongoing monitoring), and (4) inferring information from the ratings (e.g., improving clinical relevance and internal/external validity).