The Bate-Papo Vacina! Project in Zambézia, Mozambique [Endline Evaluation Report]

"The success suggests that both the Bate-Papo Vacina! approach (combining Community-Based Participatory Research [CBPR] and Human-Centered Design [HCD]) and its interventions can serve as a valuable model for similar future programming."

The Bate-Papo Vacina! (Let's Talk about Vaccines!) project aimed to reduce immunisation dropouts and improve immunisation services for 8,257 children as a target annually in 11 health facilities across two districts in Zambézia, Mozambique. This endline evaluation report provides an assessment of the project's impact on improving routine vaccination coverage, increasing access to immunisation services in hard-to-reach areas, and enhancing community engagement. It offers insights into the project's impact, challenges faced, and recommendations for sustaining the project's positive outcomes.

As detailed at Related Summaries, below, the five-year project, conducted by VillageReach and funded by Wellcome, works with caregivers of children under two, health workers, and ministry of health personnel to understand the barriers caregivers face in fully vaccinating their children. This collaborative approach involves identifying, implementing, and evaluating interventions to reduce routine immunisation dropouts. Guided by the behavioural and social drivers (BeSD) of vaccination framework and the immunisation journey framework, the research undertaken as a first step to the project identified four reasons why children under two missed their vaccinations: (i) concerns about side effects, particularly if the child falls behind schedule; (ii) lack of social support and the prevailing perception that vaccination is solely the "mother's" responsibility; (iii) uneven social dynamics between caregivers and healthcare workers; and (iv) reduced trust in the health system due to repeated negative experiences in accessing vaccinations and other healthcare services.

Based on these findings, the Bate-Papo interventions were developed through community-based participatory research (CBPR) and human-centred design (HCD) workshops. The CBPR/HCD approach led to co-development of an intervention consisting of four components: (i) immunisation education (e.g., pictorial cards with key messages to improve caregiver knowledge and agency), (ii) mobile brigade prioritisation (via a paper-based tool that was developed during the HCD workshop to facilitate planning, equity, and execution of mobile vaccination teams), (iii) collaborative immunisation planning, and (iv) supportive supervision. These interventions engage community stakeholders, healthcare workers, and community health workers to empower caregivers and their families with the knowledge and agency to demand immunisation resources for their children and to improve access to under-two routine immunisation in the hardest-to-reach areas.

Using the RE-AIM [Reach, Effectiveness, Adoption, Implementation, and Maintenance] evaluation framework, the evaluation team at University of the Western Cape (UWC) and University of Cape Town (UCT) focused on:

1. Understanding the mechanisms underlying the implementation process of community-driven interventions;
2. Assessing the impact of the interventions on access to immunisation services and reducing dropout rates; and
3. Identifying any unintended consequences of the implemented interventions.

UWC and UCT conducted a cross-sectional evaluation using both quantitative and qualitative methods. They employed an embedded realist approach to ensure that the evaluation was closely aligned with the project's activities.

Overall, the evaluation found that the Bate-Papo Vacina! project has made significant strides in boosting community involvement in childhood (under two) immunisation programmes. This involvement is linked to a positive rise in immunisation coverage and a decrease in routine immunisation dropouts. Key highlights, as outlined in the Executive Summary, include:

• The Bate-Papo Vacina! project exceeded its set target by contributing to the full vaccination of 9,041 children, surpassing monthly vaccination goals throughout the one-year pilot.
• During the 12 months of implementation, there was a substantial 133% reduction in diphtheria-tetanus-pertussis (DPT) vaccine series dropouts and a 47% increase in the proportion of children under two fully vaccinated in comparison to the 12 months preceding the implementation. In the comparison sites, on the other hand, no significant change occurred during the same time period.
• Comparison of health system-reported vaccinations between the intervention and comparison sites shows a gradual improvement over time in 6 of the 16 antigens, which can be attributed to the implemented solutions.
• Qualitative findings highlight positive shifts in knowledge, attitudes, and practices among caregivers of children under the age of two, leading to increased participation in routine vaccination activities and improved vaccination coverage. The project also fostered community engagement, enhanced immunisation planning processes, empowered caregivers through accessible immunisation education, and intensified community immunisation and primary care mobilisation efforts, thereby expanding the reach.
• The project's alignment with existing government strategies, such as the Reach Every District/Reach Every Community (RED/REC) strategy and its focus on both improving vaccination demand and accesses, yielded positive outcomes and amplified its potential for adoption, ensuring strong stakeholder buy-in.
• The implementation of the Bate-Papo Vacina! project demonstrated a high level of fidelity to the implementation plan across all sites. This fidelity was established through various mechanisms with participants and communities, such as monthly collaborative meetings, communication channels, monitoring structures, supportive tools (prioritisation matrix and educational materials), and feedback-informed adaptations, thus enabling effective monitoring and community involvement. Challenges to implementation included logistics and technological limitations associated with competing campaign priorities and poor communication network availability within some health facilities and communities, impacting the efficiency of information dissemination and community engagement.
• The Bate-Papo Vacina! project shows strong potential for maintenance and scalability, with local structures ready to sustain implementation due to robust community engagement and healthcare facility collaboration. Participants in the evaluation emphasised that because the interventions were developed in partnership with healthcare workers, community representatives, and government health officials to align with local structures, officials were primed to sustain and expand project implementation beyond the pilot phase.

Recommendations for project maintenance included continuing project routines, equitable support for all health facilities, regular visits and ongoing support, integration within the existing structures such as RED/REC, utilisation of digital platforms and educational resources, knowledge sharing of experiences, and funding decentralisation. It is recommended that VillageReach monitor the long-term impact of the intervention beyond the pilot phase, including tracking changes in behaviour, health outcome and organisational practices.

In conclusion, the researchers note that: "Strong communication and monitoring mechanisms, along with effective implementation, ensured community involvement and project adaptation. The project's success stemmed from the interconnectedness of its demand and access components, collaborative capacity building and alignment with existing government strategies. In addition, the project's readiness for integration into existing systems and the positive shifts in caregiver knowledge and attitudes towards vaccination bode well for long-term sustainability."