On April 23 and 24, 2013, one hundred development professionals debated ‘the politics of evidence’, the report of which is now available (BPF PoE conference report). The conference provided an opportunity to share and strategize for people working on transformative development, and who are trying to reconcile their understanding of messy, unpredictable and risky pathways of societal transformation with bureaucracy driven protocols. We distinguished between the big ‘E’ (evidence of what works or not) and small ‘e’ (evidence about performance) and the interaction between these.
Participants discussed four questions, using their own cases of generating and using evidence:
- What is ‘the politics of evidence’ – factors, actors, artefacts? And why is it important?
- What are the effects on transformative intentions and impacts of potentially useful approaches on evidence of and for change?
- Under what conditions do these practices retain their utility rather than undermine transformational development efforts? What factors and relationships drive the less useful practices and protocols?
- How are people strategizing to make the most of what the results and evidence agendas have to offer transformational development?
Participants noted the positive effect of encouraging more critical reflection in planning and programming. Negative effects included the questionable ethics of certain demands, the unclear utility of some artefacts, wasted resources, and ‘sausage numbers’.
Participants shared strategies for reducing the perverse effects of evidence artefacts and for enhancing their use for more transformative effect. Recognising one’s own power to make a difference, through either resistance or creative compliance, was considered a critical first step. Understanding the contexts that generate the promotion and use of evidence artefacts helps influence their effective use and critical reflection. Building collaborative relationships and stronger organisational capacities to engage meaningfully with evidence and results artefacts were also areas where participants had usefully invested efforts.
More evidence is needed about the ‘politics of evidence’, in particular how it is being experienced by grassroots workers and mid-level government staff. More examples about the utility of certain artefacts are also needed, as are ways to hold organisations to account about the utility and relevance of required protocols.
We have been silent for much too long, but it doesn’t mean there is nothing happening! I have spent the British summer shivering and trying to avoid catching colds in Australia. My (laboured) efforts to write a chapter for the Big Forward Politics of Evidence (POE) book have been enriched by participation in a number of conversations with NGO and AusAID staff. In late July I joined a well-attended and lively Politics of Evidence type workshop in Melbourne supported by ACFID, the Australian NGO network, the Australian National University and La Trobe. (I am going to leave it to one of the fellow convenors Irene Guijt, Chris Roche, Gillian Fletcher, Patrick Kilby or Megan Cooper to report!)
During my visit, I have also participated in several discussions about what lessons the results and value for money in UK might have for practitioners here. It’s difficult to tell with an imminent election and I don’t want to get bogged down in the details of conversations, but I am struck by how much donor (huge turn out from AusAID staff at one event) and NGO staff seem to appreciate having a more political analysis of the value for money agenda informed by the crowd sourcing study and experiences shared at the UK conference. Several, whom I think must have been expecting a dull methodological discussion have enjoyed honest sharing about how confusing we (official aid agencies, NGOs and consultants) initially found the agenda in the UK and the contingent effects it has had on different actors situated in different parts of the aid organisations and relationships. It’s great to be able to share examples that get away from a simple donor recipient dichotomy and provide a more nuanced view of the efforts people in different locations are making to enhance the positive effects of the results and evidence agenda and mitigate risks. I have certainly learned a great deal and sharpened my thinking about the complementarities and potential tensions between the Big E drivers of VFM measurement (economist policy makers) and the small E results (management accountants) that I hope to be able to unpack in the book chapter (groan).
The Australian community are starting their VfM journey a bit later than we did in the UK and therefore enjoy the benefit of learning from some of our ‘mistakes’. There are lots of innovative approaches being developed to help NGOs enhance and demonstrate value for money in ways that are consistent with BPF values. These include enabling citizens to define what is valued and rate the relative efficiency of different NGOs projects as a means to enhance NGO accountability to the people they work with. I won’t say more as I don’t want to steal the thunder of those involved more directly. Hopefully they will soon be able to share experiences here soon. My reason for posting a very short blog is not only to distract myself from the pain of trying to write something coherent, but to let others know how useful folks here are finding our sharing of UK and European experience. I tend to be a bit sceptical about the value of such exercises, but on this occasion it seems to have been particularly worthwhile. Concrete examples such as Christian Aid’s value based value for money framework seem to have really resonated with people here.
This is going to be short as I am sure many are enjoying well-deserved holidays, but I guess the message is that there is merit in trying to keep the BPF POE conversation going. Only this morning I received an email from Gillian Fletcher asking whether it might be possible to do a POE gig in Burma where practitioners are struggling with results and evidence issues. What a great idea!
At the Politics of Evidence conference it was clear that for many practitioners the problem with inappropriately imposed results frameworks or approaches was at least as much to do with poor internal dialogue and power relations within their own organisations, as it was with external agencies insisting upon them. This is something we have explored before in terms of how the politico-managerial environment plays out in development agencies.
Whilst the conference included many examples of how staff in agencies acted successfully as brokers and intermediaries between donors and partners, there were also examples of where this was not the case. This post focuses on some of the discussion around these examples.
Four propositions about the ‘squeezed middle’
In part, some people felt that problems were linked to a poor understanding by what was called ‘the squeezed middle’ of the realities and contexts of the programs which they were managing, or ‘representing’. The squeezed middle being those staff in middle management positions who were often the interlocutors between ‘front line staff’ or partners working on the ground, and senior managers or donors.
Secondly was discussed whether, compared with field staff or partners, incentives were greater for the ‘squeezed middle’ to respond to the demands of senior managers and donors. Combined with low knowledge of the context and programs they were responsible for, this tended to a reluctance or inability to ‘push back’ on inappropriate demands.
Thirdly, it was noted there has been a recent tendency to recruit middle managers who either may lack long term field experience of working on transformational development processes, and/or may have skills or experience from forms of project management that tend towards the contractual and linear. Thus they might be happy to insist upon what others might consider inappropriate approaches because they believed them to be the ‘right thing to do’, not because they were told to do so.
Finally there was a view that the competition between agencies and the lack of strategic collaboration between them means there is little solidarity and collective effort designed to resist inappropriate demands. More prosaically it was suggested that this was compounded by busy staff not having the time or the space – or indeed incentive – to pursue such collaboration. If these four propositions have some validity it is perhaps not surprising that compliance with certain forms of results based management was seen as having as much to do with internal organisational dynamics as it does with a simple form of power relationship between ‘donor’ and ‘recipient’.
This is also why examples of how such a situation might be disrupted are of particular interest. One powerful illustration of this was presented at the conference.
Front-line persuades Minister to change results framework
This case explored a project with sex workers, funded by a bilateral agency. This agency in order to placate domestic interests decided to approach two national organisations to support the project – one was an HIV & AIDS activist organisation and the other was a religious development organisation. One of the main ‘success indicators’ was defined as in effect ‘stopping sex workers being sex workers’. Two years after the project started the nine partner organisations were brought together in the country of the funding agency, to meet each other but also to meet with domestic organisations working with sex workers. Despite the organisations being very diverse, spanning nuns working in Bolivia to male sex workers in Macedonia, all agreed that the idea of attempting to stop sex workers working as sex workers did not make any sense, and that what was needed was to support them to be safer in their work.
At the end of the conference the group went to the capital city and got an audience with the Minister whose agency was funding the work. Collectively the group told the Minister that they would be happy to continue spending his government’s money as long as he would agree on different indicators of success, and chosen with respect to their local understandings of change, and what the groups they worked with wanted. Faced with this unanimity of diverse groups speaking directly from their experience the Minister was persuaded to agree with their proposition.
In this case an insistence by those working on the front line to represent their concerns collectively, and directly with the politician responsible led to a heightened understanding of the realities and complexities of the context. Followed by a revision of the results framework in ways that the organisations involved in this project felt comfortable with. This certainly got me thinking that making more opportunities for collapsing hierarchy and letting front line staff and partners represent and speak for themselves is critically important.
During the convenors’ reflection on the recent Politics of Evidence conference we wondered whether more nuanced power analysis might help us break out of unhelpful linear aid chain mentalities related to results and evidence. The case studies presented at the conference suggest that if we are to make the results and evidence agenda more supportive of transformational social change we need to move away from the idea that the politics of evidence is all about visible power. Images of monolithic all-powerful donors placing unreasonable evidence and results demands on well-intentioned, powerless recipients are not very helpful. Many of the experiences shared suggest we need to get more adept at identifying how hidden and invisible power influence the use of results and evidence artefacts in different contexts by individuals from different cultural backgrounds and who possess varying capacity and confidence. Such an understanding might enable us to develop more politically savvy strategies and tactics to harness useful aspects of the results and evidence agenda whilst mitigating the risks of it being used in ways that could contradict transformational development aims.
Conference convenors were delighted that on the 23rd and 24th April we were able to bring together so many thoughtful and engaged development professionals. They came from across the globe, including those working on the ground, in head offices, in consultancies and research institutes.
The Politics of Evidence conference provided an opportunity to share and strategise for people working on transformative development, and who are trying to reconcile their understanding of messy, unpredictable and risky pathways of societal transformation with bureaucracy driven protocols. They have struggled to make sense of the shifting sands of the results agenda – seeing the wisdom in some aspects while actively questioning its less useful, sometimes damaging, manifestations and consequences.
We designed it to make the most of participants’ experiences and ideas and everyone had the chance to share these in the conference break out groups, including documented case studies from about a third of the participants. As Lawrence Haddad comments in his blog yesterday on the conference, power pervaded these stories. We hope that their engagement in such an interactive conference process will have given participants courage and confidence to adopt and develop further the potential strategies and tactics (developed in the break out groups and shared in the final plenary session) to make possible programming and evaluative practice fitting for transformative development.
Over the next month or so – while the conference report is being finalised – the convenors will be blogging about some of the key issues and challenges that the conference threw into relief. Then, we plan to start work on a book that will explore these issues further, including contributions from some of the conference participants.
The Big Push Forward convenors aimed to throw a stone into a pond to make ripples. We hope these ripples will continue to expand outwards. Meanwhile, by September the current group of convenors will be stepping down in the hope that others come along to throw in more stones – either as the BPF or in some other form. Contact us if you are interested!
Day One of the Big Push Forward’s Conference on the Politics of Evidence reflected, I suppose, much of what might have been expected from it. There was a great deal of pushing back and forth about ideas and philosophies, rich discussions, a soupcon of frustration, some positivity and a lot of interest in taking some of the ideas into the second day to talk about strategies.
Since a blog cannot hope to convey the discussion, I’ll restrain myself to some threads which I felt stood out:
- Measurement processes, tools, artefacts, can all be positive or negative: it’s not about the artefact – although discourses can grow around them, and some discourses can push in one direction or another – but about the interpretation of the artefact. It’s about the detail of their implementation, and the people and the relationships involved in bringing them in and communicating them.
- Agency, tai-chi and ju-jitsu: people reflecting on their own positions are not just automatons within a relentless machine. There is agency, and there are possibilities to shape the directions of organisations and the way organisations – or the people they work within – understand the world through measurement and evaluation processes. It’s just that sometimes a little tai-chi – or possibly ju-jitsu – is needed to turn people around.
- Disjunctures in scale: that some of the measurement techniques, when used to evaluate interventions and to convince at the level of general policies, do not necessarily work at the level of individual projects. RCTs, for example, are purpose designed for scale-up, but that may not be the case for many of the project evaluations at a local scale.
- Ownership: fundamentally, one of the biggest concerns articulated was about ownership of the evaluations, and who are they for. It’s about programme staff whose projects become strangers to them, or those we are seeking to support who felt themselves robbed of voice in the face of evaluations, experimental design, and the power of evidence.
- There is no Big Bad Wolf proposing mindless tools to do people down: there are repeated, deep, systemic issues in play, coming from a fragmented and highly political environment, dealing with difficult problems. Everyone in the room had their own philosophies and their own ways of pursuing development aims within that system.
These are of course just some personal reflections and take-aways. Tomorrow, the sessions are focusing on whether we can come up with strategies for opening the space for fair assessments for development in this complex system.
The Big Push Forward convenors welcome our hundred and ten participants and student volunteers. For all those unable to attend, this evening, Brendan Whitty will be blogging about that day’s highlights and don’t forget tomorrow we are streaming live our final session.
Two papers have been prepared for the Conference and these are now on line. Rosalind Eyben’s Uncovering the Politics of Evidence and Results disentangles the historical threads and origins of results-based management and evidence-based policy/programming discourses. She discovers a strong ‘family resemblance’. Both assume that evidence pertains largely to verifiable and quantifiable facts, and that other types of knowledge have less or little value; both have a particular understanding of causality, efficiency and accountability. The paper looks at how and why these discourses have entered and influenced the development sector and who is promoting them in which contexts What has been the effect on the sector’s priorities and practices, and particularly its capacity to support transformative development?
Arguing the importance of being critically aware of how power sustains and reinforces the results-and-evidence discourses, Rosalind examines how these discourses generate artefacts (tools and protocols) such as log frames and theories of change that shape our working practices. When hierarchical ways of working block communications and dialogue, the artefacts trigger perverse consequences but their power is neither uniform nor constant. Analysing the politics of accountability and the sector’s internal dynamics, Rosalind suggests there is room for manoeuvre to expand and enable more transformative approaches to results and evidence within the sector.
Brendan Whitty’s paper, Experiences of the Results Agenda, paper analyses the data from an online survey, which invited visitors to the Big Push Forward website to give their perceptions of the impact of the results agenda on their working lives. Brendan analyses the very different experiences and interpretations of the respondents as revealed through both 153 responses to the quantitative survey and 109 qualitative stories. The study discusses the day-to-day practice of small-e evidence –results and targets in management of specific projects – rather than large-E evidence of establishing broader development policies. The stories are about the nuts and bolts of the development processes and artefacts – the theories of change, results frameworks, reporting requirements and value for money rubrics. It is about what ‘e’ is being collected, how it is used, and to what effect.
Respondents disagreed about the effects of these artefacts. The contradictory perceptions seem to be often in tension. Thus learning is often seen to be(?) in tension with accountability; capturing the complexity in evaluation with harmonisation and reductionism; coordination of partners with constraining their freedom to adapt. How these tensions are resolved and the perceptions play out seems to be dependent on how the artefact is communicated, managed and tailored to its context. The fit appears to be important: the fit of the artefact to the existing systems and capacity of the organisation, and also the fit of the artefact to the specifics of the intervention (e.g. its complexity, the number of partners). Finally, perceptions of an artefact seem to be affected both by staff’s’s own circumstances and their relationship with others. The survey data suggests that those in M&E and management roles, who benefit from better data and more resources for their priorities, tended to be more positive than those in project implementation and mid-level roles.
During the conference we will be exploring these ideas and testing these intepretations. Come back tomorrow for the deliberations of Day 1,
This week our Politics of Evidence conference gets underway – and we celebrate our second birthday. We are delighted there has been so much interest and sorry that we have inadequate space for all those keen to attend. For those unable to attend, we are live streaming the last session of the conference (15.45- 17.00 UK time on April 24). We hope to post the conference report on the website at the end of May and have longer term plans for a book.
The conference coincides with the second anniversary of the birth of the Big Push Forward. We posted our first blog on the 26th April 2011. Our aims have stayed consistent – helping create the political space to ensure appropriate approaches in the design, monitoring and evaluation of projects with transformative aims. This is necessary for the international development sector to continue to seize opportunities to support transformation for greater social justice. read more…
In my experience, the results agenda is not only emotional in the sense of controversial, but also confusing to many people, NGO staff I work with in Africa, Asia and Germany have difficulties with the concept of results, and much goes wrong. Arguably a lot of the trouble stems from a strong utilitarian influence on the results agenda that does not fit well with other cultural traditions involved in development aid.
Utilitarianism is a philosophical tradition that started in Britain in the 18th century. It deals with the question of how to act morally, and what government action is morally best. Put simply, in a utilitarian view, human behaviour is the more moral, the more it creates happiness. In the words of Bentham, “it is the greatest happiness of the greatest number that is the measure of right and wrong”. The utilitarian idea has been very influential in Britain, and more widely in the Anglo-Saxon world. The push for effectiveness builds on this tradition. “Happiness” is now made to be understood as “results”. Governments are “effective” (read: moral) if they produce lots of “results” (read: happiness). To make effectiveness measurable, results should be pre-defined. I am not sure if the architects of the results agenda are aware of their utilitarian background, but we are all heavily influenced by our traditions, and the forerunners of the results agenda (New Public Management, micro-economics and the logical framework concept) are dominated by North American thinking building on utilitarianism. People from other traditions just do not understand the underlying assumptions and are confused. Being German myself, I have observed that German development agencies found it rather difficult to introduce results frameworks. They experienced a lot of resistance from staff, and people were confused for a long time. They disliked the added bureaucracy that comes with the current results concepts. But, I believe, underlying is that the Anglo-Saxon results concept does not fit into German culture.
Many Germans, particularly in the social and cultural sciences, are brought up in very different philosophical traditions than Anglo-Saxons. Two philosophies of German origin are particularly relevant to the effectiveness debate. read more…
At the Politics of Evidence conference we will be discussing how certain approaches to accountability may undermine the sector’s potential to support transformative development. Payment by Results (PBR) is one to watch out for. But we have been here before.
As Europe and North America industrialised and proceeded to colonize the rest of the world, the positivist power of numbers appeared to tame uncertainty in an era of such rapid change. In Britain, the fondness for measurable facts led the introduction of ‘payment by results’ (PBR) into elementary schools in the middle of the 19th Century. PBR (aka Cash on Delivery) is when commissioners of services (e.g. a government) pay the service providers only after a pre-determined result has been achieved and independently verified. The logic of PBR is that there is a manageable level of risk in achieving the result and that service providers must be incentivised to play a more active role. 150 years ago – like today – the buzzwords were efficiency, value for money (VfM), competition and a balanced budget. At the time PBR was criticised for its mechanistic approach that impeded children’s educational development and sacrificed long term benefits for short term achievements. By the end of 19th Century PBR was abolished partly due to the increased bureaucracy and administration costs of verifying the results. PBR had been proven to be inefficient! Fast forward to 2013 when ‘Public bodies seem to be pursuing the use of payment by results with the vigour of a drunk in search of the next bottle of alcohol’, Jon Tizard writes. read more…