Application of a Multi-Level Implementation Framework to Evaluate Ontario’s Province-Wide Healthy Babies Healthy Children Program
Saturday 10:30 – 11:45 Breakout D5
Presentor: Heather Manson
Heather Manson, Public Health Ontario; Adrienne Alayli-Goebbels, Public Health Ontario; Anne Philipneri, Public Health Ontario; Helen Cerigo, Public Health Ontario; Lori Webel-Edgar, Public Health Ontario; Sarah Muir, Public Health Ontario,
Introduction: Ontario’s Healthy Babies Healthy Children (HBHC) program aims to provide services to families to help children achieve their full potential. In 2013, program enhancements were introduced to strengthen services to vulnerable families. A process evaluation was conducted to understand how local public health units (PHU) implemented these enhancements during the first six months, and to explore factors contributing to implementation outcomes.
Methods: An adapted multi-level implementation framework by Chaudoir et al. guided our evaluation. Based on the framework, we used an embedded mixed methods design to explore factors at different levels (innovation, client, provider, organization, and system levels) that may have contributed to implementation outcomes (program reach, fidelity and other outcomes related to program enhancements).
Results: Factors that positively impacted implementation included the multi-faceted approach to facilitate program changes at the PHU level, highly committed HBHC staff, positive organizational cultures within PHUs, and staff’s openness to change. Challenges included increased staff workload and ongoing engagement with community partners for service delivery. Underlying population characteristics, health unit size, and local adaptations partially explained variability across PHUs.
Conclusion: The implementation framework was useful in planning our evaluation. Based on the evaluation results, recommendations were provided to improve the HBHC program implementation.
Assessing Implementation fidelity in Ontario’s Healthy Babies Healthy Children Program
Saturday 10:30 – 11:45 Breakout D5
Presentor: Eunice Chong
Eunice Chong, Public Health Ontario; Adrienne Alayli-Goebbels, Public Health Ontario; Anne Philipneri, Public Health Ontario; Helen Cerigo, Public Health Ontario; Lori Webel-Edgar, Public Health Ontario; Sarah Muir, Public Health Ontario,
Introduction: The Healthy Babies Healthy Children (HBHC) program is funded by the Ontario Government and is being implemented in all 36 local public health units (PHUs) across Ontario. The guidance document for the enhanced HBHC program specifies fidelity requirements including HBHC Screen completion and follow-up of postpartum clients screened within 48 hours. In our evaluation, we examine fidelity by assessing the extent to which these and other core components were carried out in all PHUs.
Methods: Client data on screen completion rates, 48-hour contact rates, and other program components were extracted from the HBHC administrative database. Survey and focus groups with PHU staff provided insights related to their experiences in implementing these components.
Results: Overall, 66% of the HBHC screens were complete, with high levels of variation across PHUs in 48-hour contact (9–89%) and other key components. Findings from surveys and focus groups revealed the challenges that PHUs faced in implementing these functions, and different strategies used to address these challenges, including local adaptations.
Conclusion: Results showed that fidelity to core components greatly varied during the first six months of the enhanced HBHC program. PHUs will require additional supports in order to improve the consistency of program delivery.
Contribution of Province-Wide Process Implementation Evaluation to Continuous Quality Improvement through Tailored Knowledge Dissemination to 36 Local Public Health Units
Saturday 10:30 – 11:45 Breakout D5
Presentor: Eunice Chong
Eunice Chong, Public Health Ontario; Lori Webel-Edgar, Public Health Ontario; Sarah Muir, Public Health Ontario; Anne Philipneri, Public Health Ontario; Helen Cerigo, Public Health Ontario
Introduction: Results from Ontario’s Healthy Babies Health Children process implementation evaluation were presented at a provincial level. However, local public health unit (PHU) level findings were anonymized. In this presentation, we describe a tailored approach to broker and disseminate evaluation findings, as guided by Graham et al.’s Knowledge-to-Action framework.
Methods: We customized individual PHU reports that included data related to reach and fidelity specific to each of Ontario’s 36 PHUs and allowed for geographical comparisons. To facilitate knowledge uptake, tailored dissemination meetings were organized with individual PHUs to help program staff understand their evaluation results, and to discuss how results can support continuous quality improvement.
An online survey was developed to measure the awareness and reception of the evaluation results, and to evaluate the usefulness of the individualized report and dissemination meeting to support evidence uptake. The survey was completed by PHU staff who attended the meetings.
Results: All 36 individual PHU dissemination meetings have been completed. Descriptive analysis of the survey results will be shared at the time of the presentation.
Conclusion: Tailoring results and dissemination methods for intended knowledge users can help facilitate knowledge application and can help identify local strategies for continuous program improvement.
The Impacts on Program Reach of Cumulative Declines and Losses to Contact as Clients Traverse the Multi-Step Healthy Babies Healthy Children Program
Saturday 10:30 – 11:45 Breakout D5
Presentor: Heather Manson
Heather Manson, Public Health Ontario; Helen Cerigo, Public Health Ontario; Eunice Chong, Public Health Ontario
Introduction: Ontario’s Healthy Babies Healthy Children (HBHC) program includes screening, contact, in-depth assessment (IDA), and referrals to blended home visiting (BHV) and community services. For an evaluation of the enhanced HBHC program, we assessed program reach as an implementation outcome by examining cumulative declines and losses of clients at each step of the program.
Methods: Client data were extracted from the HBHC administrative database, with reach denominators at each step defined as those who successfully completed the step immediately before. Characteristics of those reached were collected from the initial screening.
Results: In the first six months of the enhanced program, 46% of postpartum clients screened with risk (25,930). Progressive client declines and losses to follow-up meant that only 37% of these went on to IDA and 9% were referred to BHV. Clients with pregnancy, labour and delivery related risk factors were less likely to receive IDA and BHV compared to those with socioeconomic and parenting risks.
Conclusion: Progressive client declines and losses to follow-up impact program reach in multi-step programs. Our evaluation demonstrated advantages to using existing administrative data for program evaluation, and present opportunities to explore reasons for declines/losses and optimize service outcomes.