ABSTRACT 3: Research example, BMC format

Shannon Wiltsey Stirman1, Cassidy A. Gutner2, Jennifer Gamarra3, Dawne Vogt2, Michael Suvak4, Jennifer Schuster Wachen2, Katherine Dondanville5, Jeffrey S. Yarvis6, Jim Mintz5,7, Alan L. Peterson5,8, Elisa V. Borah9, Brett T. Litz2, Alma Molino5, Stacey Young McCaughan5, Patricia A. Resick10
1National Center for Posttraumatic Stress Disorder (PTSD), Veterans Affairs (VA) Palo Alto Health Care System and Stanford University
2National Center for PTSD, VA Boston Healthcare System and Boston University
3University of California, Los Angeles
4Suffolk University
5Department of Psychiatry, The University of Texas Health Science Center at San Antonio
6Headquarters, Carl R. Darnall Army Medical Center, Fort Hood
7Department of Epidemiology and Biostatistics, The University of Texas Health Science Center at San Antonio
8Office of Research and Development, South Texas Veterans Health Care System
9The University of Texas at Austin
10Duke University Medical Center
Correspondence: Shannon Wiltsey-Stirman (sws1@stanford.edu)
Title: Leveraging routine clinical materials to assess fidelity to an evidence-based psychotherapy

Background

Fidelity monitoring and support is a central component of many implementation models. A critical barrier to efforts to monitor and support treatment fidelity in routine care settings and large systems is a lack of availability of feasible, scalable, and valid fidelity measurement strategies [1]. Development of reliable, low-burden methods of fidelity assessment is an important step in promoting sustained implementation fidelity for complex interventions in routine care.

Materials and methods

We developed a system to assess fidelity (adherence and competence) in an evidence-based psychotherapy by rating clinical notes and worksheets. External raters assessed clinical notes, along with worksheets that were completed with therapist guidance within sessions. Worksheets completed independently by clients for homework were also rated to differentiate between therapist and clients’ contributions to worksheet quality. We examined feasibility, efficiency, reliability, criterion-related validity (correlation with observer ratings of session video), and predictive validity (whether ratings predicted symptom change) using data from a clinical trial of Cognitive Processing Therapy conducted in a military setting (N = 106).

Results

The rating system required an average of seven minutes per session (versus 50-60 for video observation). Intra-class correlations indicated good to excellent rater agreement. Adherence and competence ratings were highly correlated with observer ratings for worksheet-related items.

Symptoms did not predict subsequent therapist fidelity, but therapist fidelity in certain sessions predicted subsequent symptom change. Client skill on homework worksheets did not predict subsequent symptom change.

Conclusions

This system of assessing fidelity using routine clinical materials has potential as a reliable, valid, efficient, and scalable fidelity monitoring strategy.

References

  1. Schoenwald SK, Garland AF, Chapman JE, Frazier SL, Sheidow AJ, Southam-Gerow MA. Toward the effective and efficient measurement of implementation fidelity. Admn Policy Ment Health. 2011 Jan 1; 38(1):32-43.

Learning Objectives:

  1. Identify barriers to fidelity monitoring in large service systems
  2. Examine predictors of therapist fidelity to evidence-based practices and client outcomes
  3. Discuss feasibility and scalability of assessing fidelity using client charts and psychotherapy materials

Keywords: Fidelity, methodology, implementation barriers, cognitive processing therapy, VA

Data collection phase: Data analyzed

Project implementation phase: CompletedPresentation type: Oral presentation