Bridge Project Mid-Term Evaluation

Executive Summary

"A mid-term evaluation of the Johns Hopkins University Center for Communication Program (JHU/CCP) and Save the Children's (SC/US) BRIDGE Behavior Change Initiative HIV/AIDS Project was conducted in Malawi between June 20 and August 12, 2005 to determine progress towards achieving results as outlined in its work plan. Another purpose of the evaluation was to decide whether to recommend that USAID exercise its option of continuing the project for the two remaining years of its cooperative agreement and, if so, to determine the advisability of modifying project approaches to maximize potential for meeting targets of USAID's SO8. This report addresses these issues by answering three main questions or evaluation objectives:

• Objective 1: Have project activities been appropriate and effective in moving toward prevention of new HIV infections through behavior change initiatives?
• Objective 2: What are the principle strengths and weaknesses of the BRIDGE project in terms of management, coordination, and communication?
• Objective 3: What, if any, changes of an administrative or technical nature should BRIDGE consider to strengthen its performance and maximize impact?

The evaluation team consisted of two consultants from Chemonics International and one from The Manoff Group. After initial briefings from USAID and the BRIDGE Project, the team conducted field visits to Mangochi, Chikwawa, and Mzimba districts. Visits included interviews with youth groups, cultural committees, parents groups, AIDS support groups, District AIDS Coordinating Committees (DACCs), Community AIDS Coordinating Committees (CACs), Village AIDS Coordinating Committees (VACs), radio listening groups, women’s groups, school staffs, and others. In addition, numerous interviews were conducted in Blantyre and Lilongwe with representatives from radio stations, NGOs, CAs, and government ministries. Follow-up in-depth interviews were also conducted with BRIDGE staff and USAID.

Objective 1: Assess progress made in implementing the project and achieving yearly targets and estimate if project is likely to achieve end of project objectives. Review suitability of project design and effectiveness of BRIDGE Project components in helping community and national stakeholders implement NBCI. In general, the team found that the project is making significant progress towards achieving its objectives. Regarding its theoretical assumptions, the project found after completing the baseline survey and reviewing other documentation that the initial assumption of targeting high-risk groups was not appropriate, and changed its strategy to target the general population, with emphasis on youth.

The “four pillars” of the BRIDGE concept outlining intervention levels (national, community, coordination, and capacity building) appropriately comprise the essential elements of an effective health behavior change intervention (BCI) approach. Some of the successful interventions include the Radio Diary, the Hope Kit, and the Nditha campaign. Another big success was the youth listening groups formed around the PSI Youth Alert program. BRIDGE formed 200 clubs and trained 400 leaders. In addition, 936 teachers were trained and provided with a facilitators guide developed by the project. The project has also successfully worked to disseminate information and build capacity through district and national Public Affairs Committee (PAC) member FBOs motivating strong interfaith collaboration and support.

At this point it is difficult to ascertain quantitatively whether the project is on track in meeting its targets, though from a qualitative perspective it appears to be so. However, the relationship between the project indicators, which seem to be taken from the PEPFAR program and the mission PMP, and the project’s design model is unclear. The M&E plan does not specify how the project is going to gauge the impact of activities on behavioral outcomes to be measured at the end of the project. To date there has been no regular collection of monitoring data to record the effects of project activities (other than quarterly reports from grantees that do not include quantitative data). Because there is no coverage data available, it is difficult to measure the public health impact of activities. The project intends to conduct a quantitative survey using baseline indicators in Year 3, but the evaluation team recommends that the large survey be postponed until more programmatic activity has occurred, and instead conduct specific monitoring actions to better measure the effects of ongoing program interventions.

The "four pillars" of the BRIDGE strategic framework are an implementation approach of the Structural-Environmental model and include: national, community, capacity building, and coordination. BRIDGE's multilevel approach is reasonable and appropriate, and facilitates the creation of social norms and interpersonal support for adhering to protective behaviors. The evaluators found evidence that dissemination of consistent messages to the different program levels is creating synergy that is reinforcing the intention to adopt HIV protective behaviors. This was particularly evident in the radio listening groups, the youth clubs, the interfaith groups, and in the community committees. There was reliable evidence that messages have been harmonized among the various groups working in HIV/AIDS prevention and control. There is a need for ongoing capacity building for efficient production of communication materials and dissemination of messages. In addition, sustainable expansion of activities will require capacity building (including monitoring skills) of district and community level actors, continued training of faith-based organizations (FBOs) at local levels, and expansion of ABC messages consistent with local realities. BRIDGE has engaged and provided technical assistance to multiple national and community partners, including PAC, NYCOM (National Youth Council of Malawi), MANASO (Malawi Network of AIDS Service Organizations), DACCs, CACs, radio stations, youth congresses and other groups.

The "unifying themes" of hope and self-efficacy (Nditha) are effectively reaching targeted audiences. In accord with the strategy to date, greater emphasis has been given to youth. The quality of the radio broadcasts is high. Radio Diaries and HIV/AIDS content in Youth Alert in particular are reaching audience segments and are a critical link to the community, offering messages of hope and empowerment. These are being cited by community members as essential in changing their attitudes and behaviors at the community and household level. Integration of the Youth Alert Facilitator’s Guide was also well done. Technical capacity building will continue to be needed at the radio stations, in teacher training, and in managerial and programmatic areas at the district level during the remainder of the project.

All evidence gathered by the evaluation team indicates that BRIDGE's technical assistance for implementation of the NBCI strategy, especially involving capacity building of the district assemblies, DACCs, CACs, VACs, FBOs, CBOs, traditional leaders, and youth organizations, is working extremely well. In districts where BRIDGE has been able to build on SC/US’s STEPs HIV/AIDS program, this has been more easily achieved than in areas where STEP is not located. According to processes and inputs, BRIDGE is somewhat behind schedule in terms of planned inputs, and the order of rollout as defined by the work plan has been rearranged. Unanticipated capacity building needs (technical and administrative) of implementing partners has resulted in the delay. The BRIDGE Project is directly addressing harmful cultural practices, especially related to gender, family, and sexual relations. Improved relationships between the genders at the household level have been reported, and communities have detected decreases in domestic violence as a result of improved family communication.

Objective 2: Assess management and coordination of the BRIDGE Project.
The project is organized appropriately for the tasks outlined in the work plan. However, the project may need either more time or more staff to accomplish all the planned tasks, particularly since the budget is one-third less for years 3 and 4. As presently organized the project can manage two major tasks (campaign, Hope Kit, Radio Diaries, youth congress, etc.) at one time. Furthermore, it should be noted that because of their successes there is an increasing demand for BRIDGE's skills and assistance with BCI tasks being conducted by partners, CAs, and NGOs.

The non-CA sub-award activities are progressing, although they were delayed in starting and some still have difficulties with financial and technical reporting. Since building BCC programming capacity is part of the objective, it is appropriate that time be taken for this. Unfortunately extra time was not programmed for it either technically or financially. Systems and procedures are in place and being followed for personnel, sub-grants, sub-awards, and technical activities. Unfortunately, the fact that many administrative and financial transactions need approval or are managed through Baltimore, and that USAID requires approval of many activities, has contributed to some delays and programming inflexibility. A good complementary relationship exists between JHU/CCP and SC/US, and the organizations clearly function as a team. BRIDGE has brought important technical assistance and capacity to partners through project staff and headquarters support.

Although the progress reports do a good job of documenting activities conducted and discussing steps to be taken during the follow-on period, project monitoring information regarding effectiveness in reaching target groups is not included.

BRIDGE effectively operates at the national level through working groups and other networking activities to develop a coordinated and "harmonized" BCI approach. The project assures that consistent messages are used and that all agencies are promoting the same HIV prevention concepts. In addition, the core BCI concepts (hope, youth) have been very successful.

The "Advances" workshop was very well received and the workshop tools are being used by participants to develop BCC activities in their own institutions and settings. There has been no follow-up to document the kinds of actions taken by participants following the workshop or provide further direction for the activities.

BRIDGE has also done a good job of bringing its baseline research findings to the attention of other agencies working with HIV/AIDS prevention in Malawi and drawing attention to the underlying norms and values affecting behavior. The missing piece to operationalize a coordinated National BCI strategy is a clearly laid-out operational plan (OP) for implementing the strategy. This operational plan should include yearly targets, specifying roles and responsibilities assigned to agencies and personnel. The plan should include materials, agreed-upon timelines, and budgets for projected activities.

Objective 3: What, if any changes of an administrative or technical nature should BRIDGE consider to strengthen its performance and maximize impact?

The evaluation team makes the following recommendations:

Funding for the BRIDGE Project should be continued for option years 3 and 4.

Monitoring and Evaluation:

• The quantitative survey planned for August 2005 should be postponed until all project elements, including those targeting men’s behaviors have been implemented for ample time to produce impact. A monitoring system to measure the effect of all major program elements should be established, along with feedback mechanisms to each level where data is collected. Project reports, including those from partners should include data and more specificity as to the "who, what, where, when, and why" activities were undertaken and their effect.
• BRIDGE should strengthen community monitoring and evaluation with technical assistance from SC HQ, CORE group or Child Survival Technical Support Project (CSTS), etc. The project should consider Lot Quality Assurance Samples (LQAS) used by child survival projects as a quantitative monitoring tool.
• BRIDGE should call a partner and stakeholder meeting after the evaluation to examine activities and determine:
  1. Which activities have yielded the best results thus far?
  2. Which activities can be turned over to other partners, e.g. NAC, MOH, MOE?
  3. Which activities can be supported by other donors?
  4. Given limited resources, which activities can be dropped or consolidated for better focus and impact?
  5. How can activities be optimally positioned to achieve maximum synergy of the pillars?

Documentation and Dissemination:

• The project should collaborate with the national BCI Working Group to develop a specific plan to maximize the dissemination of project findings and lessons learned across sectors.
• Success stories, of which the evaluation team heard many, should be thoroughly documented, especially the project elements that lead to success. Success stories should be included in reports to USAID and shared through the national BCI Working Group.

Scale-up:

BRIDGE should determine existing program coverage and develop plans for comprehensive coverage within project districts. Community organizations already have scale-up strategies but are in need of additional resources. BRIDGE's national impact can be significantly enhanced by:

1. Collaborating with SO8 partners (particularly FHI) to build DACC capacity to conduct mapping exercises and plan coverage for district activities.
2. Documenting the "critical pathway" of BCC capacity building at the district and community levels, including resource requirements and dissemination through the NAC's Operational Plan.
3. Linking national radio programs and communities by increasing regional radio capacities to include community members (including youth, traditional and religious leaders, pregnant women, etc.) in broadcasts. Expanding programming to include more stories from rural and regional communities.
4. Strengthening the radio stations' ability to produce their own HIV/AIDS programs.
5. Complementing the NBCI Strategy and the National BCI Social Mobilization Plan with technical assistance to the National BCI Working Group organized by NAC. Special efforts should be made to include the Ministries of Education, Gender, Youth, Agriculture and Rural Development and to provide specific recommendations on how these ministries can support community groups (especially to youth groups, CACs and PLWHA groups) to facilitate the implementation of a BCC "scale up".
6. Developing the community mobilization arm of the men's campaign as soon as possible while media component is being developed.

Management:

The evaluation team recommends that BRIDGE either increases staff or reduces planned tasks. Given reduced funding for years 3 and 4, the project may want to consider reducing the number of districts to concentrate inputs in selected areas. If more funding becomes available, the project then needs to carefully program its activities for the remaining period to maximize impact. Planning should include realistic timelines.

• M&E programming capacity should be added to project staff (FTE or consultant).
• In order to speed the proposal and reporting process for sub-awardees, mentoring and/or training should be provided for required inputs (budgets, work plans etc.). BRIDGE might consider “conditional” proposals that can be corrected as activity begins.
• A realistic work plan should be developed that considers available staff and financial resources and time needed to complete tasks that will demonstrate measurable impact.
• Project monitoring information on progress in reaching target audiences and groups should be included in reports. It would also be useful to include lessons learned and interesting “success stories” that are surfacing from the project.
• BRIDGE should work with NAC to develop an operational plan for implementing the NBCI Strategy.
• BRIDGE should build on its successful "Advances" workshop by providing a refresher course for participants."