Who determines what counts as sufficient or accurate evidence? What values are allowed to shape the choices that are made? What are the implications for everyday development work? The political push and pull around ‘evidence’ will be debated in our conference in April 2013 in Brighton, UK. As part of our conference planning, clarity around participation and purpose is paramount. Here are our thoughts so far.
Who is it for?
This conference provides an opportunity to sharing and strategise for people working on transformational development, be it on the ground, in head offices, in consultancies or think tanks. You, like us, are trying to reconcile your understanding of messy, unpredictable and risky pathways of societal transformation with bureaucracy driven protocols. And have struggled to make sense of the shifting sands of the results agenda – seeing the wisdom in some aspects while actively questioning its less useful, sometimes damaging, manifestations and consequences.
The conference aims to make the most of the experiences and ideas of more seasoned practitioners and thinkers. It is not a training workshop and, therefore, is not intended for those who are new to these debates or issues. However, the outputs of the conference will be documented to have wide practical value for all those working on and supporting transformational development.
What are the outputs?
Conference outputs will range from the visible to the personal. Some outputs will be very tangible such as analytical frameworks to understand organisational discourses around ‘results’ – and case studies that illustrate how the results agenda works in practice in diverse organisations. What’s behind a particular protocol or a reporting format in terms of how results are understood? What alternative processes might be adopted in working within organisational systems or stretching them? The conference will not be a marketplace to exchange specific evaluation methods or approaches. Rather we will focus on sharing experiences, strategies and tactics that expand the institutional spaces for those approaches that give useful and accurate insight into development as transformation.
Being engaged in an interactive conference process will give participants greater capacity to analyse power dynamics in their contexts, to understand the politics of evidence, identify manoeuvring space, and generate appropriate options. Seeing examples of how others have undertaken this will give people courage and confidence to start or continue to measure and value transformation in ways that are respectful, fair and useful. Participants will leave with specific questions answered and information shared, as well as potential strategies and tactics to try out.to make possible evaluative practice that fitting for transformative development.
Why this is important
It’s the transformation that matters; the evidence emerging from evaluation must serve rather than constrain it. Tapping into the collective experience gives us a stronger basis for creative ways forward.
So if these are our ideas, what do you want the conference to do for you? Please let us know!
Power dynamics shape whose and what knowledge counts and which results matter in development policy and practice. The Big Push Forward is organising a conference in April 2013 for those concerned with transformative development and the political challenges of assessing what works. Our aim is making participants more conscious of how power plays out in evaluation processes; strengthen their capacity to deal with it; and gain the courage and confidence to navigate political space, maintaining or increasing options and putting pressure on the system to shift demands.
Knowing what works for whom, how, when, why and in which contexts is crucial for all working in development, be it through the older institutions of international aid or the newer forms of partnership, South-South collaboration, and business-oriented efforts. Many organisations are testing and using more systematic and robust approaches to assess performance and to learn to be more effective. Discussions and protocols focus on ‘hard’ attributable evidence, rigorous data, conclusive proof, value for money, and evidence-based decision-making. These tantalising terms are often accompanied by statements that promise clarity once and for all about what works and what should be funded in international development. read more…
Owen Barder has written a typically crisp blog on the Results Agenda. He provides a useful and succinct delineation of ‘Seven Worries about Focusing on Results’, most of which the Big Push Forward along with many others, have canvassed in the past year or so.
First, focusing on results may add to bureaucratic overload. Second, it may make aid less strategic and short-termist. Third, it may impose the wrong priorities. Fourth, it may ignore equity. Fifth, it may create perverse incentives. Sixth, it may inhibit partnership. Seventh, the results information is all bogus anyway as claims about results must rely on assumptions about the counterfactual which are usually flawed or incomplete.
The weapons of mass destruction never existed yet unleashed a human tragedy that continues today. What can we learn for the results agenda from the evidence base surrounding the WMD myth?
Last week, Dr. Michael Patton was with us in Wageningen, the Netherlands. On March 23rd over 50 people in international development working on evaluation debated the politics of evaluation and the battlefield of ‘rigour’. Two issues struck me in relation to rigour as a contested concept.
First, what is rigour all about? Patton shared the example from the intelligence community in the USA in relation to the non-existent weapons of mass destruction for which the
re was supposedly so much evidence. The intelligence community was seriously discredited. In response, a community of practice emerged to redefine ‘rigour’. This resulted in the Rigour Attribute Model by Zelik, Patterson and Woods (2010). This model challenges the focus on prescribed standards and tight adherence to this, linking this practice to a failure of ‘intelligence’. The authors look at different sources of risk around evidence, and offer eight attributes of analytical rigour.
A couple of weeks ago, someone phoned me from a large INGO to explain that his organisation was under increasing pressure from its government donor to ‘do a ToC’ in its funding proposal. ToC had become an additional mandatory results-based management tool constraining the INGO’s partners from designing projects based on their own strategic decisions. This is a real shame because a dialogue about the change desired and a theory of how to achieve it can be useful provided we recognize that any explanation is partial, contingent on context and needing to be regularly checked against reality, as experienced from diverse perspectives. read more…
In his classic article back in 1959, Charles Lindblom describes the technical ideal of evidence-based policy use:
“[A policy-maker] might start by trying to list all related values in order of importance … Then all possible policy outcomes could be rated as more or less efficient in attaining a maximum of these values. This would of course require a prodigious inquiry into values held by members of society and an equally prodigious set of calculations on how much of each value is equal to how much of each other value. He could then proceed to outline all possible policy alternatives. In a third step, he would undertake systematic comparison of his multitude of alternatives to determine which attains the greatest amount of values.” Public Administration Review, Vol. 19, No. 2. (Spring, 1959), p 79.
The conclusion that Lindblom comes to is that this approach is ‘of course’ impossible. It “assumes intellectual capacities and sources of information that [people] simply do not possess”.
One possible interpretation of ‘value for money’ management processes (VfM) resembles Lindblom’s task. Such an interpretation understands VfM as requiring the monetization of an intervention, and thus the comparison of different interventions using a common denominator. In a guidance note on VfM and governance indicators, DfID rejects this approach. It emphasises the need for a common sense approach which does not ask impossible questions like ‘what is the value for money of an election?’ This moderate approach has been replicated as considerable ingenuity is being dedicated to develop methods for capturing value for money (see, e.g. BOND, ICAI, ITAD).
However, recent evidence sessions with the Public Accounts Committee evidence sessions, suggests that its expectations of DfID’s VfM process may be closer to Lindblom’s Herculean process:
“The Department must be able to demonstrate unequivocally that it allocates resources on the basis of value for money… The Department should develop clear and auditable mechanisms which ensure that staff in both Headquarters and country offices have value for money criteria at the heart of their decision making, and that they reallocate funding to the best possible alternative when projects are delivering weaker value for money than expected.” (At p. 6, para 7.)
The requirement for unequivocal demonstration of value for money suggests that DfID must show how it has analysed different programming choices against others when making allocation decisions. It hints at a quiet, underwater, trial of strength between DfID and its domestic stakeholders for the definition of the VfM agenda, with the PAC seeking to lever open the black box of DfID’s decisions and subject them to technical review and audit.
The process of making resource allocation decisions between countries constitutes a particularly interesting arena for this struggle. Consider the PAC’s demand in the evidence session that DfID assign a number to the fiduciary risk of giving aid per country – a demand which DfID suggested (rightly, in my view) that this approach didn’t make a lot of sense, since risk depends on the recipient institution rather than the country. However, interpreted as a desire to have DfID articulate technical and auditable criteria for country allocations, it makes a great deal more sense. Such an interpretation may cast light on other data requirements: the demand for comparable sector-level data across countries and the Bilateral Aid Review’s requirement for evidence on security for each country. All constitute comparable indicators that might instruct resource-allocation decisions.
However, as Lindblom observes, there is a limit to the capacity of evidence-based policy-making. Some decisions cannot easily be boiled down to technical criteria and pressure to the contrary carries with it considerable challenges, as has been illustrated by the recent Africa All-Party Parliamentary Group’s report on DfID’s Bilateral Aid Review:
“Our concerns relate to the lack of objective criteria used to select focus countries… The Needs-Effectiveness Index (NEI) appears to have been used to justify the subjective decisions of officials, rather than to make objective decisions. DFID did not provide a clear and convincing explanation of why its Needs-Effectiveness Index was constructed in the way it was…”
While the AAPPG disputed the grounds DfID chose (resulting in closure of the Burundi office, amongst others), in fact any assumptions would have been open to challenge. An insistence on common indicators, representing complex ideas of value and impact, will lead to reductionism. This risks submerging the true decisions somewhere in the generation of the numbers and the assumptions made to do so. Because these reasons are hidden, they are less susceptible to challenge and scrutiny, and therefore may, paradoxically, be less accountable. The consensus is increasingly that it is important not to throw the evidence baby out with the reductionist bathwater: that the limitations of evidence do not mean that evidence should not be collected and used. Equally, however, moderation is required so that processes of evidence collection inform rather than replace decisions. Resource allocation on the basis of reductionist indicators is unlikely to give of its promise, either in terms of accountability or effectiveness.
Improving value for money is not about delivering activities at the lowest cost or creating the outcomes that are easiest to measure and monetise, we conclude in a new paper by Bond, the membership body for UK development NGOs.
The paper moves away from looking at value for money as a donor imposition and instead frames it as an opportunity for NGOs to improve the consistency and robustness of how they assess, demonstrate and strengthen the effectiveness of their work.
Developing a new framework for value for money
Building on examples drawn from the current practice of UK development NGOs, the paper presents a framework to help NGOs navigate the value for money agenda and identify which approaches and methods are best suited to their needs. The framework identifies three areas that UK NGOs should focus on to deliver and improve value for money:
- the systems and processes in place for managing value for money
- the approaches they use to compare value for money between activities
- the use of evaluations to demonstrate value for money
There is no one size fits all approach, as the range of different methodologies being used across the sector shows.
While the language of value for money may be new, many of the methods and approaches used to realise it are not. Transparent procurement, monitoring expenditure, avoiding corruption, should already be very familiar to many NGOs.
Do we really need econometric modelling?
Perhaps the newest element of the value for money agenda (and the one that most NGOs are most wary of) is the use and promotion of econometric modelling and methods such as Cost Benefit Analysis or Social Return on Investment studies as a way of demonstrating value for money. In the paper we are clear that while these methods may work in some situations, they are not suited to every context and there are other similarly robust ways of demonstrating value for money which do not require monetisation of outcomes.
For example, an evaluation of an NGO’s programmes which combines high quality quantitative and qualitative evidence of outcomes and impact with strong evidence of economy and efficiency on the cost side can also provide a powerful evidence base for demonstrating value for money. It is important that we do not overcomplicate what it means to demonstrate value for money; ultimately it should be about NGOs explaining their cost, robustly describing their results and justifying the relationship between the two. You don’t need an economic model to do this well.
Putting beneficiaries at the heart of value for money
The paper also argues that listening to the voices and focussing on the priorities of beneficiaries is key to achieving value for money. The practical resource on incorporating value for money into the programme cycle which accompanies the Bond paper looks at how beneficiary participation throughout the programme cycle is crucial for building a strong case for value for money. This is in line with the Independent Commission for Aid Impact’s new report on value for money which argues for value for money to be a balance between economy, efficiency, effectiveness and equity. Equity, assuring that benefits are distributed fairly, speaks to the central place of poor and marginalised beneficiaries in the value for money debate.
Join the debate
We hope that this paper generates discussion and is useful to NGOs and others in shaping the way they approach value for money. We welcome your feedback and experiences of using the framework and tool.
Last week BOND published some useful VfM resources for NGOs. In this post Rob LLoyd and Mischa Foxell highlight some key messages
Improving value for money is not about delivering activities at the lowest cost or creating the outcomes that are easiest to measure and monetise, we conclude in a new paper by Bond, the membership body for UK development NGOs.
The paper moves away from looking at value for money as a donor imposition and instead frames it as an opportunity for NGOs to improve the consistency and robustness of how they assess, demonstrate and strengthen the effectiveness of their work.
Developing a new framework for value for money
Building on examples drawn from the current practice of UK development NGOs, the paper presents a framework to help NGOs navigate the value for money agenda and identify which approaches and methods are best suited to their needs. The framework identifies three areas that UK NGOs should focus on to deliver and improve value for money: read more…
Towards the end of 2011 three thousand people travelled to Busan, South Korea, to discuss the future of international development assistance. All eyes were on that group of donors, who are themselves still aid recipients – such as India, Brazil, Indonesia and China – who were seen to be challenging the paradigm in which the ‘West/North’ determined what was development and how it should take place. However, the ‘BRICs’ them was not the only show in town. Various coalitions of donors/recipients/civil society were pushing their favourite themes. In a joint statement released before Busan, the ‘blue club’ of bilateral donors (Britain, Canada, Denmark, Germany, Netherlands, Sweden, and the United States) issued a joint statement in which ‘a drive for better development results’ headed the list of their priorities. 
They successfully established a post-Busan ‘ coalition of the willing’ to pursue a ‘results and accountability’ agenda but their persistent emphasis on results (and everything it entails in tyrannical practice, if not in rhetoric) irritated some conference participants and some spoke out about it. It was implicitly criticised by Rwanda’s President Kagame in his speech at Busan when he said,
“Developing countries spend more time and energy agreeing on procedures and accounting to donors and an ever-increasing number of related non-state actors than in actual development work, often responding to endless questioning that no answers can fully satisfy.”
Richard Ssewakiryanga, the Chair of Uganda’s NGO Forum was more explicit and trenchant in his blog about Busan an extract from which he has given BPF permission to reproduce here.
The word that kept lingering in my mind after all 100 side events, knowledge and information events and plenary sessions is “results.” Every speaker both at the opening and closing ceremonies and in the two thematic sessions and one plenary focusing on results spoke about the importance of showing results – to tax payers, to voters, to poor people and donors as well as partner governments and parliaments in both the north and south. But three issues stand out around the concept and idea of results that we may need to pay more attention to read more…
In February 2009 Alnoor Ebrahim, Associate Professor at the Harvard Business School, gave a presentation on ‘Measuring and Aggregating Civil Society Results’ to a Donor Roundtable Meeting on the “Aggregation of Donor-funded Civil Society Organizations’ Development Results” . See here for the full report, and section 3.3. for a summary of his presentation.
He suggested four types of results that could be captured.
- Added-Up Results (1+1=2): i.e. the whole equals the sum of the parts based on comparable projects/programs. This approach assumes scale is achieved through replication and the implication for donors is efficient, scalable actors.
- Synergistic Results (1+1=3): i.e. the whole is more than the sum of the parts, scale is achieved through complementarity and the implication is fund networks and collaborations that have clear targets.
- Tranformational Results: (1+1= LOVE): i.e. the whole is more than the sum of the parts + there are ‘threshold’ effects or ‘tipping points’ above which more transformational change is achieved. Implications – fund based on evidence-based linkages between activities and attitudinal shifts.
- Harmful results: i.e. by implementing a process of aggregating results, you invest energy in something else than achieving development results.
He also proposed three principles to guide any framework for aggregation and arguably therefore for reporting: read more…
