Discussion Paper on the Monitoring and Evaluation of UN-Assisted Communication for Development Programmes: Recommendations for Best Practice Methodologies and Indicators

"Experience has shown that development interventions are much more effective when they employ the strategic use of communications. In addition, a fully democratic society cannot function without clear channels of communication available to all. Because of this, communication as a means for development has been adopted by the United Nations as a central strategy in the global bid to achieve the Millennium Development Goals..."

This 51-page paper from the United Nations Development Programme (UNDP) and the World Bank was prepared in advance of the 11th UN Inter-Agency Round Table on Communication for Development (Washington, DC, United States), March 11-13 2009. It was designed to serve as a discussion paper for the first theme of the roundtable: "Assessing and Demonstrating the Impact of Communication for Development" (hereafter, C4D). This paper was designed to help roundtable participants explore key C4D issues, consider case studies, and identify best practice methodology, in order to glean key questions and indicators, some of which are proposed at the end of this paper. These indicators are designed to focus on the effectiveness of communication as a tool and process for development rather than on the impacts of specific C4D programmes.

According to the paper's authors, despite a greater appreciation of C4D within the UN, there is still some conceptual confusion regarding its practical interpretation and application. The first section addresses this confusion in the context of monitoring and evaluation (M&E), locating C4D programmes under two broadly defined, human rights-based frameworks: i) diffusion, or behaviour change communication (BCC), and ii) participatory communication, or communication for social change (CFSC). The common denominator throughout these approaches is the use of a wide range of information and communication technologies (ICTs) to deliver information and promote discussion and participation. The authors present evidence suggesting that combining elements of both approaches can be the most effective way to ensure the success of a C4D process. However, initiatives that simultaneously employ many types and levels of communication necessitate a more complex and sensitive approach to M&E. To increase and demonstrate the effectiveness of C4D, planning, monitoring, and evaluation must play a critical role and be accorded the appropriate consideration and resources.

"There is a strong case for integrating C4D into all U.N. programme development and for U.N. communicators therefore to be at the table from the outset, working alongside results professionals, programme managers, M&E specialists and statisticians to ensure that C4D indicators are included in the wider picture of results-based management." Building on the categories outlined in the first section, the paper moves on to discuss principal M&E tools and approaches, from conventional, quantitative research techniques to more intricate, participatory processes.

• Here is a list of the "diffusion-based tools and approaches to M&E" that are summarised in the paper: behaviour change comparisons/behavioural surveillance surveys; cost benefit/cost effectiveness analysis; experimental impact evaluation studies; knowledge, attitudes, behaviours, and practices (KABP) surveys; logical framework ("LogFrame"); media coverage analysis; policy change analysis; propensity score matching; public expenditure tracking surveys (PETS), rapid rural appraisal (RRA), regression analysis, and tracking surveys.
• Here is a list of the "participatory tools and approaches to M&E" that are summarised in the paper: ethnographic action research (EAR), most significant change (MSC), outcome mapping (OM), participatory rural communication appraisal (PRCA), and social network analysis (SNA).

"In order to drive C4D into the U.N. development framework at both theoretical and operational levels, the above methodologies must be considered in the context of achieving results and more specifically managing for development results (MfDR)." This management strategy focuses on development performance and on sustainable improvements in country outcomes. It provides a framework in which performance information is used for improved decision-making, and it includes practical tools for strategic planning, risk management, progress monitoring, and outcome evaluation. The authors also explore the importance of ownership in the context of M&E, stressing that "stakeholder communication, involvement and ownership should not be limited to methodologies that are more participatory in nature: efforts should be made to build this element into any evaluation practice."

The third section considers the principal challenges associated with measuring C4D that discussions over best practice must take into account. Here is a list of the challenges that are summarised in the paper: causality/attribution, resources, nature of outcomes, lack of consensus on the tools of measurement, time frame, defining the so-called "target audience", gaining access (to ICTs), reporting failure, reporting for the longer term, creating sustainability and capacity, proactive and reactive initiatives (assessing the impact when something did not happen), from the individual to the whole, globalisation, and organisational challenges.

An examination of the practical application of C4D and strategies to monitor and evaluate progress and impact follows. A theoretical model is used to illustrate how programme goals, results, and activities link in to research, monitoring, and evaluation. Various case studies are used to illustrate how initiatives have used a variety of approaches to gather informative results and feed these back into programme design. Others highlight some of the practical challenges to M&E even within a highly successful programme, such as limited understanding at senior stakeholder level of the implications and requirements of participatory M&E.

A combination of research and information gathered from interviews with practitioners at the policy and field levels is used in the next section to make recommendations for best practice methodology. The section discusses the issues of causality/attribution; working within the resources available for analysing an initiative (which are often very poor); using M&E to establish all the outcomes of an initiative; combining approaches to provide enhanced feedback and avoid a conceptual divide; reporting for and aligning results with donors and funding bodies while nonetheless maintaining ultimate objectivity; working through country-led systems; identifying the strategic intent in order to give an initiative a clearer direction; making planning, monitoring, and evaluation key parts of any C4D initiative; and ensuring that all the above considerations feed into real country ownership of results. The section proposes that a tailored toolkit approach to M&E of C4D is necessary in order to deal with the many complexities of this field. It emphasises the need for flexibility within any evaluation and discusses the strengths of MSC, OM, and the LogFrame as techniques that attempt to meet the above challenges. The section concludes by reiterating that qualitative and quantitative tools are not mutually exclusive but should be used as complementary strategies for providing the most comprehensive, clear, and pragmatic assessment and reporting of C4D.

Section 6 moves on to consider the use of indicators for assessing impact across the broad remit of U.N. agency activities in the field of C4D. Five principal C4D components are identified: the level of local awareness about the development programme and the issues covered by the initiative; evidence of direct impact as a result of the programme; participation and empowerment; the level of media coverage; and country capacity. Within each component, key questions, proposed indicators, and guidance on methodology to verify the indicators are suggested. This framework for thinking about indicators has been designed to have practical application in the field and is proposed to guide discussion at the roundtable over how such an approach can be used across the UN system.

The paper concludes by proposing questions for roundtable participants to discuss in order to refine the suggested indicators and consider strategies for their adoption. "A critical part of such strategies and the successful adoption of indicators will be strong communication and coordination at all levels within the U.N. agencies in terms of continuing to enhance and cultivate modes of monitoring and evaluation for communication programmes."