Blog – Informing policies with causal impact evaluations: Co-creation and trust matter

Tracking policy outcomes and costs over time helps stakeholders monitor progress and budgets. However, that approach does not measure the impacts that a program or policy has on the intended beneficiaries—for that, you need a “counterfactual.” For example, what if people in some districts get a program and people in other districts do not? People and circumstances vary across districts, so we would not expect that the program would have the same impacts in both places; it’s a classic apples-to-oranges comparison. Thus, assessing a program by comparing its impacts in one district to conditions in a “control” district without it may not yield meaningful results. What we really want to know is, what if the same people who did get the program did not get the program—what would have happened to them? Obviously, we can never know that. But we can still learn what would have happened using a causal impact evaluation approach. This involves identifying a group of people with demographic characteristics that are almost exactly the same as those who did participate in the program (a “counterfactual” group).

Figure 1 below shows how causal impact evaluation works. The vertical axis is the measurement of what you want the program to do: Say, improve family income. At the start of the program, we want the two groups to have virtually identical incomes, as well as other characteristics like family size, age of the head of the household, etc.; they start at the same point on the graph (Y₀). The horizontal axis marks time from the start to the end of the program. We can see that those without the program experienced an increase in income (Y₁*)—maybe there were good rains that year. However, those participating in the program experienced a greater increase in income (Y₁). That difference is the true, causal effect of the program. Causal impact evaluations show if programs and policies actually work, demonstrating that changes are because of the program and not other factors that could effect the same outcomes.

Figure 1

^IFPRI

How can causal impact evaluations help?

Impact evaluations can give us two main types of evidence. The first type helps assess a program that a government, nongovernmental organization (NGO), or other stakeholder may want to expand (ex ante evaluation). Here, researchers team up with the stakeholder to figure out what kind of information they need and then set up an evaluation to see how well the program works for the outcomes they want to achieve. The results tell us how effective and cost-effective the program is and give suggestions on how to make it better. Often, these suggestions are taken into account, and programs get tweaked before they are expanded.

The second type looks at a program that’s already been expanded, to see if there are any changes that could make it work even better (ex post evaluation). Here, researchers team up with stakeholders to try out different ways of running the program and see if there is a more effective approach. The evidence from this kind of collaborative research is usually considered and can lead to improvements in the program.

Providing the proof

The collaboration between the government of Bangladesh and the IFPRI is a great example of an ex ante evaluation. IFPRI researchers based in Dhaka designed the Agriculture, Nutrition and Gender Linkages (ANGeL) pilot project, the first government initiative in Bangladesh to use evidence from a causal impact evaluation to design and implement a national program. It was a randomized controlled trial that aimed at improving nutrition-sensitive agriculture. IFPRI’s evaluation found that the program improved farmers’ incomes, agricultural diversity and diet quality, and empowered women. The Ministry of Agriculture subsequently approved, and the Ministry of Finance endorsed ANGeL’s national implementation. It has been integrated into the Country Investment Plan (CIP-2) and the 8th Five-Year Plan.

Another example of successful collaboration is the research partnership between the government of Egypt and IFPRI focusing on the country’s social protection strategy, which includes a range of programs, including ration cards (the Tamween program) for purchasing food and direct financial support (Takaful and Karama program) for the poorest families to meet their basic needs. IFPRI has evaluated these programs and found that they have lifted millions of people out of poverty. These findings have informed important policy changes. After initially subsidizing less healthy foods like rice and sugar, Tamween was adjusted in 2017 to include healthier foods after an evaluation showed an increased risk of children becoming overweight. Another evaluation showed that Takaful and Karama reduced poverty by 11%, but missed some of the poorest households. Targeting procedures were subsequently updated.

IFPRI’s causal impact evaluation of Ethiopia’s Productive Safety Net Program (PSNP), the largest social protection program in Ethiopia, ran for 15 years (2006-2021). The four rounds of impact evaluations centered around resolving operational problems to improve the reach and efficiency of the program. Evaluation results directly influenced the reform of three aspects of the PSNP that the government adopted: 1) the appropriate transfer amounts for people involved in the public works component of the program, 2) the criteria for being able to support oneself without the PSNP and 3) how to ensure the timely payment of transfers to beneficiaries. The government describes the critical role these causal evaluations have played in measuring proven success and coming up with properly tested solutions to problems.

Co-creation and trust matter

As these examples show, causal impact evaluations can be extremely helpful for stakeholders to maximize the potential of their policies and programs. To ensure the uptake of evaluation-based recommendations, strong partnerships between all stakeholders are invaluable. Decision-makers should be part of the whole evaluation process, from defining the questions to the launch of the study results. Preliminary findings and questions emerging from the incoming data should be continuously exchanged among all evaluation partners. Such a process of co-creation builds trust, helps researchers to better understand the data and supports implementing partners realizing the early implications of their programs. It can also help decision-makers to make real-time adjustments to policy designs and implementation. For such a continuous, mutual learning exchange to happen, evaluation teams should be based in the countries where the evaluation takes place, examples of which are IFPRI’s country strategy support programs and the CGIAR Initiative on National Policies and Strategies (NPS). Such longer-term country presence not only helps with informing individual policies, it can also help with promoting science-based policymaking more broadly and sharing institutional capacity more sustainably.

Kibrom Abay is a Senior Research Fellow with IFPRI's Development Strategies and Governance (DSG) Unit; Akhter Ahmed is the Country Representative for IFPRI-Bangladesh and Senior Research Fellow with IFPRI's Poverty, Gender, and Inclusion (PGI) Unit; Clemens Breisinger is a DSG Senior Research Fellow and IFPRI Kenya Country Program Leader;  Naureen Karachiwalla is a PGI Research Fellow; Sikandra Kurdi is a DSG Research Fellow; Alemayehu Seyoum Taffesse is a DSG Senior Research Fellow and Ethiopia Program Leader. This post first appeared on Agrilinks. Cited research is peer-reviewed.

This work is part of the CGIAR Research Initiative on National Policies and Strategies (NPS). CGIAR launched NPS with national and international partners to build policy coherence, respond to policy demands and crises, and integrate policy tools at national and subnational levels in countries in Africa, Asia and Latin America. CGIAR centers participating in NPS are the Alliance of Bioversity International and the International Center for Tropical Agriculture (Alliance Bioversity-CIAT), IFPRI, International Livestock Research Institute (ILRI), International Water Management Institute (IWMI), International Potato Center (CIP), International Institute of Tropical Agriculture (IITA) and WorldFish. We would like to thank all funders who supported this research through their contributions to the CGIAR Trust Fund