Breakout 0 – May 17, 2013
1. Implementation of the Program to Encourage Active & Rewarding Lives for Seniors (PEARLS)
Presenter: Lesley Steinman, MSW
Lesley Steinman, MSW, MPH,1 & Mark Snowden, MD, MPH2
1Health Promotion Research Center, Department of Health Services, University of Washington; 2Department of Psychiatry & Behavioral Sciences, University of Washington
Abstract: Background: Depression is often undertreated in older adults. PEARLS is an evidence-based depression care management program for homebound elders. Working with multiple community partners, we studied several approaches for improving implementation and adapting PEARLS to address commonly identified barriers.
Methods: Ten focus groups with 40 staff were analyzed using thematic analysis to identify barriers to implementation. A formal agency plan was developed to improve implementation and the plan was evaluated using process and outcome measures. We developed a 20-item fidelity instrument through key informant interviews and validated the instrument using known-groups method with 12 agencies.
Results: Focus groups revealed strict eligibility criteria interfered with agency’s mission to serve all clients. PEARLS modifications were piloted with interpreters for limited English-speaking clients and for clients with major depression. Depression response and remission rates were similar to the original model (80%). Implementation coaching resulted in modest improvements in referral (9% to 15%) and enrollment rates (4% to 8%). Fidelity instrument testing showed PEARLS programs had higher fidelity scores compared to other types of depression programs (p<.05).
Conclusions: PEARLS is an innovative program for treating late-life depression through community-based social service agencies. Technical assistance may be necessary to help address common early implementation challenges.
2. Common Issues with Assessing Fidelity to Complex Multi-Modal Service Programs: Lessons Learned from Assessing Fidelity to the ACT Model
Presenter: Maria Monroe-DeVita, PhD
Author: Maria Monroe-DeVita, PhD, Department of Psychiatry & Behavioral Sciences, University of Washington School of Medicine
Abstract: Fidelity assessment is a key element to ensuring that treatment programs are adherent to the intended model and can therefore anticipate achieving desired clinical outcomes; however, comprehensive evaluation of fidelity to more complex multi-modal service programs can be difficult to achieve. For some programs, the broad range of biopsychosocial service needs of the population served require clinicians to employ more than one evidence-based practice (EBP) within the context of the larger program; in some cases, service recipients may receive treatments delivered by more than one clinician and/or in a variety of community-based settings outside of the office. This presentation will focus on how these complex program elements may be assessed by using the new Assertive Community Treatment (ACT) fidelity scale – the TMACT – as a case study, focusing on key issues in fidelity assessment such as balancing evaluation of: (1) process and structure; (2) team and individual clinical skills; and (3) other EBPs integrated or blended within the larger service program. While this presentation will use ACT, an EBP for adults with serious mental illness, as an illustration of how these core dilemmas in fidelity assessment can be handled, implications for other service programs for different populations will be discussed.
2. Assessing Implementation Fidelity of the Family Check-Up: Development & Validation of the COACH Rating System
Presenter: Justin D. Smith, PhD
Justin D. Smith, PhD1 Elizabeth A. Stormshak, PhD,1 & Thomas J. Dishion, PhD1,2
1Child & Family Center, University of Oregon; 2Prevention Research Center, Arizona State University
Abstract: We present a series of studies concerning the development and validation of an observation fidelity of implementation rating system for the Family Check-Up (FCU). The FCU is a family-based intervention shown to improve family management practices and reduce problem behaviors in youth ages 2-18 (e.g., Dishion et al, 2008; Stormshak & Dishion, 2009). Method: Therapists treating families of children ages 2–17 from two randomized trials (one efficacy and one effectiveness) were rated for fidelity to the FCU in three separate studies, the final study being an experimental manipulation of the rating procedures in an attempt to improve reliability of the ratings. Results: Study 1: Variations in fidelity were associated with observed positive parenting of toddler-age children one year after receipt of the FCU, which in turn predicted reductions in child problem behaviors the following year. Study 2: Therapists employed at community mental health agencies achieved adequate levels of fidelity, which was associated with family and child level outcomes. Study 3: Two factors of previous fidelity rating studies were identified that likely contributed to less than optimal reliability: Coder training in the FCU and access to family’s assessment data. Observed caregiver engagement, a single item in the rating system that has been found to be an important intervening variable in the relationship between fidelity and family outcomes, was also examined for validity. Conclusions: The FCU is an effective family-based intervention that is feasible for scale-up in multiple community service settings. The accurate and reliable assessment of fidelity of implementation is a crucial factor in training providers to deliver the intervention as intended. These studies demonstrate the validity and reliability of our fidelity rating system.