A Guide to Monitoring and Evaluating Policy Influence

This Overseas Development Institute (ODI) Background Notes paper from the Research and Policy in Development (RAPID) Programme gives an overview of approaches to monitoring and evaluating (M&E) policy influence and is intended as a guide, outlining challenges and approaches and suggested further reading. It was developed with the United Kingdom (UK) Department for International Development (DFID).

From the document: "The paper begins by defining policy and policy change, before outlining a typology of approaches to influencing policy. It then sets out the main challenges of monitoring and evaluating this influence on policy, together with the ways in which M&E frameworks respond to these challenges - often by constructing a theory of change (ToC). The next section outlines key considerations for developing a ToC. The paper then uses the typology of influencing activities presented earlier, discussing for each the issue of what to measure, and when and how it should be measured. The paper concludes with key recommendations for developing M&E frameworks for policy influencing activities."

An illustration (Figure 1) on  page 2 shows policy influencing approaches including: lobbying, advising, activism, and advocacy with communication-related examples of direct action, petitioning, policy briefings, and company lobbying. A chart on typologies of influencing activities further details both channels and means of influencing, for example:

• Evidence and advice as typologies use policy discourse/debates and meetings employing the following means: research and analysis and good practice; evidence-based argument; providing advisory support; and developing and piloting new policy approaches.
• Public campaigns and advocacy use: public and political debates; public meetings, speeches, and presentations; television, newspapers, radio, and other media employing the following means: public communications and campaigns; "public education"; messaging; and advocacy.
• Lobbying and negotiation use formal meetings, semi-formal and informal channels, and participation on boards and committees employing the following means: face-to-face meetings and discussions; relationships and trust; and direct incentives and diplomacy.

The discussion that follows includes developing a TOC and measuring policy influence results. For example, on public campaigns and advocacy, the document emphasises the importance of monitoring the audience of campaigns to ascertain outcomes such as: awareness of an issue or campaign, perception of saliency or importance of an issue, attitudes, norms and standards of behaviour, and actual behaviour. Suggested monitoring methods include surveys, focus group discussions, and direct responses. Monitoring the media for increased coverage of an issue is another possible method using a media tracking log or "tracking column inches in newspapers, air time on television or radio, or monitoring hits on a website", as well as calculating the estimated audience figures for a programme that features the campaign. Tracking the effect on the audience is more difficult. "The possible approaches include: 

• Exposure: measuring exposure by looking at the degree to which the audience has encountered a campaign, how many times they were exposed, and whether they paid attention. Interviews and surveys could be used to see whether people recall a particular message or campaign, and simple figures about readership of papers, and ownership of televisions/radios can be a useful guide.
• Framing analysis: looking at how issues are presented or discussed, by reviewing the key themes, metaphors, arguments, and descriptions in a given media (newspaper, websites, etc.)."

The document contains a table on page 10 that lists outcomes to measure and tools to use for the three typologies of influencing approaches explored in the document. It then recommends the following:

• "Collecting information, monitoring target audiences, making judgements about level of influence (and so on) are time-consuming and tricky activities.... Any systems developed should ensure that information collected can have multiple uses (e.g. both for decision-making and, later, reporting), and that it is integrated with, and draws on, any information or knowledge produced during the planning stage of a project.
• It is important to develop some kind of theory of change (ToC) as early as possible in the planning stage of an influencing project. This sets the overall framework for M&E, giving teams a way to categorize and make sense of available information throughout the project, and a basis for more in-depth studies by external evaluators during or after the intervention.
• A number of... tools can be used to collect relevant data opportunistically or at periodic intervals throughout the policy influencing work. If these can be selected and integrated into programme management from the outset, they will be useful for decision-making throughout the work, and become a useful resource to be visited after the end of a project. The tools listed in Table 2 are used for each influencing approach."