Fear, Innovation and the Results Agenda
This post suggests that the politics of the results agenda is playing out in ways that make a frank debate about its consequences hard. The Big Push Forward is seeking to surface this issue.
There have been a number of interesting papers published recently dealing with the ins and outs of evaluating development impacts. These include a paper by Howard White and Daniel Phillips and a paper by Elliot Stern and colleagues both of which seek to broaden the range of methods deemed suitable for impact evaluation.
A paper I co-authored with Linda Kelly for the Developmental Leadership Programme called the Evaluation of Politics and the Politics of Evaluation also explores some of these issues, illness arguing strongly for a ‘mixed method approach’, site but in which we also suggest that it is political demands and interests which often shape the choice of methods as much as concerns over rigour or appropriateness.
Some commentators, pharmacy like Owen Barder, have acknowledged the risk that the results agenda might create perverse incentives, but argue there is little evidence that this is occurring. Others feel that the bigger risk is missing the important opportunity that the results agenda provides to build a new agenda based on the voices of those the aid system seeks to benefit.
The organizers of the Big Push Forward conference – planned for April 2013 – are picking up from some NGO staff and their partners, and from some within bilateral and UN agencies, that there is a view that the way results and value for money agenda is being pursued is distorting practice for the worse in a number of places. Other observers are making similar points.
We believe this is a critically important debate. However one of the problems that we are experiencing is that very few people in agencies are prepared to say much about this in public. They fear that being seen to be ‘against’ the results agenda is potentially damaging for their individual careers or for their organisational standing, and this risks aggravating donors and therefore their puts their funding at risk.
There seems therefore to be a kind of subterranean struggle going on. Individuals within NGOs and other grantees, as well as in donor agencies, are trying to create space to pursue alternative means of addressing what they see as the ‘legitimate’ or meaningful parts of the results and value for money agenda, but without it undermining what they feel to be good development practice.
It is our view that these strategies are not being properly discussed or shared and that doing so would be highly beneficial. Learning from how adaptation is happening in practice is often a key to learning and improvement. As Dennis Rondinelli argued many years ago understanding how front line development workers behave in practice provides important clues to how best support their efforts, rather than relying on ‘perfectionist planners’.
We also believe there is perhaps more space within bilateral and UN agencies, as well as in NGOs, to promote alternative and innovative approaches than is currently suspected. It is important to exploit the opportunities that this space provides in order to influence an agenda that is still somewhat fluid.
The BPF conference is a critical opportunity to explore these possibilities as well as the potential for cross-agency collaboration and research in this area i.e. on developing approaches to value for money which are appropriate for rights based approaches; how to assess gender justice programs which seek to change power relations; or how governance and policy dialogue programs can be monitored and evaluated.
We will be soliciting your stories shortly. Do join us in the debate!
Comments are closed.
Thanks for this Tina. The Big Push Forward Conference certainly hopes to explore some of the issues you raise including what ‘counts’ or is valued as evidence. We also hope to be able to collect more ‘evidence’ about what people are experiencing, and we are currently working out how best to do that. Watch this Space.
It would be really interesting to know more from your work about what strategies and innovations people have adopted in the light of these concerns. It strikes me that many of the issues that the results agenda seeks to address are absolutely legitimate, notably that taxpayers deserve a transparent and honest account of what is done with taxes spent by governments in their name, that governments and aid agencies need to be held to account for how they perform; and that the process by which these things are done should reinforce an improvement in practice. If the results agenda is not delivering these things, as you suggest, then what are the innovations and adaptations that would?
Thanks Chris. These questions that NGOs are trying to answer are legitimate questions, but then are so many other questions about NGO work.
The issues of accountability to tax payers and being held to account for performance are important (though government probably has more responsibility to tax payers than development NGOs) and being held to account has to go way beyond obligations to funders and politicians in UK. In terms of honesty and transparency how to get honest reporting and evaluations when continued funding (often huge amounts for many NGOs) depends on proving success against present targets and agendas.
What about other issues NGOs are legitimately concerned with or being asked to address (i.e. by Hulme’s recent review paper)? For example, ‘do no harm’ is not even a yardstick for many at present, and it was a very low one already! What about being answerable to those in whose lives we intervene? How well equipped are we, how well do we understand the problems and what might enable poor people to better engage and increase their options given current ways of working and an assumption that the questions raised here are the most important? What problems do the influx and then cessation of aid, new ideas, different ways of working from outside cause for social movements, local struggles, women’s organisations etc? How do choices made here skew and distort actions far away? Where do NGOs see their legitimacy deriving from? How much is from getting funds and accounting for these adequately to donors and how much from the quality of their work in building partnerships, contributing to local capacity, supporting people in defining their agendas…and much more.
For me a bigger question is why issues of value added, efficiency, effectiveness, accountability to funders etc have become defined as THE questions, questions perhaps more relevant to Government and the private sector. Are these the key questions facing ‘the third sector’. Or is the concept of a third sector, a civil society driven sector, now irrelevant or dead?
My concern is less to find tools that will meet these current concerns and more to explore what roles NGOs are/could be playing , where they fit in the politics of aid at a time when the role of the north in development is far outweighing attention to the agendas, needs, ownership etc of those in whose name aid is provided . Should NGOs be openly challenging some of the assumptions and demands being made on them in return for funding and challenging the demands for targets, results based approaches etc as teachers, health staff and even the army have done in recent years? Do they have alternative agendas and questions to raise, based on long experience of development work, and if so (and many do because I hear them discussed in small groups etc) where and how can they raise them? This to me has to be part of the big push forward (or push back) , pushing to change conversations and some of the norms being entrenched and looking at past and present experience to draw out other ways of conceptualising what is critical to development and so contributing in more collegiate ways to generating social transformation. We do not deliver results (except in the narrowest of service delivery programmes), we are working with people who are citizens of other Governments and regimes who have other accountabilities, it is not simply ‘our gig’ to shape and control and account for, as so much of the current debate implies.
Tina thanks again.
Lots of great questions of precisely the type we hope the Big Push Forward conference can have a frank and honest debate about.
What we are also hoping is that this debate is not just about NGOs, but also about the challenges faced by those in government agencies, and civil society organisations more broadly defined – and of course their relations to the citizens and communities they either represent or seek to work with . Indeed, as you rightly point out, bringing in experiences from other sectors who are facing similar issues may be useful too.
Arguably sharing these different perspectives will be important if we want to be able to explore the political economy across the sector as a whole and how different interests and power relations interact. This is also liable to assist in helping us, as you suggest, reimagine and reconceptualise ways to support the promotion of transformational development.
Thanks. Sounds good.
This is a critical debate and one that has been surfaced in DSA study groups in recent times, for example in some seminar workshops in the gender policy and practice study group.It is so good to see others raising these issues more broadly and for wider debate.
There is a lot of evidence out there about the way the growing bureaucratisation, streamlining etc of aid, RBM etc are affecting development work in NGOs. ‘The aid chain’ (which I was involved in) looked at how top down tools alter and reshape relationships with partners in v problematic ways; this was based on first hand research. There is evidence in reviews like the one recently put out by David Hulme, looking at written evidence over several years, although Duncan Green at Oxfam dismissed this as ‘annoying’, thus rather shutting down debate. Reviews exist to show the problems caused by these approaches, such as BEYOND SUCCESS STORIES: MONITORING & EVALUATION FOR FOREIGN ASSISTANCE RESULTS
EVALUATOR VIEWS OF CURRENT PRACTICE AND RECOMMENDATIONS FOR CHANGE
MAY 2009
This paper for US government was produced independently by Richard Blue, Cynthia Clapp-Wincek, and Holly Benner showed that
– The role of monitoring in the USG foreign assistance community has shifted dramatically in the last 15 years. The role of USG staff has shifted to primarily monitoring contractors and grantees. Because this distances USG staff from implementation of programs, it has resulted within the Agencies in the loss of dialogue, debate and learning from monitoring.
• The myriad of foreign assistance objectives has led to onerous reporting requirements that try to cover all bases.
• There is an over reliance on quantitative indicators and outputs, deliverables over which the project implementers have control (number of people trained) rather than qualitative indicators and outcomes, expected changes in attitudes, knowledge, and behaviors that would be the consequence of the outputs.
A UN report a few years ago (ref to be found!) said that results based monitoring had been time wasting, onerous and did not add anything to knowledge or understanding of development…and there are many more reports dotted around.
For me then the question becomes more why is this evidence, supported by a great deal of commentary heard in NGOs and evaluations carried out by those who want to learn about what happened and what changed rather than assessing grant compliance and proving success etc, is still not seen as evidence and consistently dismissed.
There is an understandable a fear amoung staff in many NGOs about speaking out publicly for all kinds of reasons, but people do talk internally and share examples and these can become a body of valid evidence over time. Significant evidence is already out there and I hope the BPF can serve as a catalyst for bringing together different ‘people speaking out’ and the existing evidence to demonstrate what is happening under current results focused regimes, where proving success is required to get the next tranche of often considerable funding. We are bringing out a book on these issues in 2013….which adds a range of evidence and first hand experiences to the debate, or so we hope!