3 faces of power:

• Visible power:  formal decision-making mechanisms, e.g. donors insisting on the use of certain results artefacts, such as logical frameworks and reporting formats
• Hidden power:  a mobilisation of bias, e.g. not inviting small NGOs to attend meetings about results frameworks
• Invisible power:  social conditioning through cultural traditions, ideology, etc that shapes the psychological and ideological boundaries of what seems possible, e.g. practitioners being conditioned to accept the results agenda and the use of certain artefacts, e.g. indicators as the norm; recipients feeling they can’t challenge donor decisions;  INGO programme managers feeling they can’t challenge normative organisational norms and beliefs relating to the production and use of results and evidence.

Firstly, a brief recap for those who weren’t there. Case studies presented on the first day indicated that there is nothing inherently good or bad about evidence and results tools. We explored both positive and negative effects of ‘artefacts’/tools such as logical frameworks, cost benefit analysis, theories of change etc (green in diagram below). Some examples suggest that theories of change are being applied to evaluative practice in ways that empower practitioners to usefully interrogate their assumptions about how social change happens. However, a case presented by an organisation working on rights for women with disabilities illustrated how such tools can be used in ways that, perhaps partly as a result of hidden power –their lack of involvement in discussions about results frameworks, undermine a sense of ownership.

We also heard stories of how efforts to demonstrate effectiveness through impact evaluations can fall short when it comes to encouraging organisational learning for social change. I wish we had kept a count of the examples of expensive evaluative processes that failed to meet participants’ expectations that they would produce incontrovertible evidence to inform decision-making. They raised questions about whether expectations that they can are an operation of invisible power? Furthermore, discussions on the second day confirmed that even if it were possible to produce irrefutable evidence, it would be impossible to apply it objectively to learning and decision making given the complex politics at play in most development organisations.

Revelations about the role that senior managers of INGOs play in incentivising the reporting of ambiguous results – such as numbers reached by campaigns – exploded myths about institutional donors being the sole agents imposing artefacts in unhelpful ways. A nuanced power analysis can help us better understand the political factors and conditions that influence the use of results and evidence tools (blue in diagram above) e.g. senior management’s role in shaping organisational cultures and norms Such a power analysis might also inform assessments of which practices are benign and which potentially dangerous and thus what kind of strategies (in red above) are appropriate responses.

Several empowered practitioners I spoke to argue that responding/complying to demands for large number results e.g. number of people reached can be a strategy consistent with transformational agendas. They are of the opinion that it is easy for practitioners to satisfy political requirements and come up with what Brendan Whitty in his crowd sourcing report (page 16) refers to as ‘sausage numbers’. For them resistance – arguing about the meaningfulness of such figures is not a good use of energy they would rather devote to learning about how change happens. Yet some evidence collected through the crowd sourcing report suggests that enormous  effort can go into generating such numbers at the expense of learning. Moreover, case studies, like the example of the INGOs above hinted that if practitioners are asked to measure effectiveness in terms of ‘numbers reached’, compliance could affect the ways they view the aims and success of their practice. I certainly saw evidence of  local NGO frontline staff starting to see development success in terms of numbers of people reached by interventions during research in Cambodia.  Several conference participants felt that these interpretations of results tools privilege upward accountability and need to be challenged. One group suggested that by articulating accountability to the citizens development organisations work for as a measurable result it would be possible to shift power relations and make a focus on results more supportive of transformational change.

Many strategies proposed by conference participants in the last session suggested greater awareness of how power influences the conditions (above) can in itself be empowering. Strategies shared demonstrated a heightened sense of personal agency, confidence and ‘power to’ resist inappropriate use of some artefacts or negotiate organisational politics to frame results in ways supportive of transformational development. Some conference participants that might have previously felt isolated identified ways to collaborate with kindred spirits in donor organisations, INGOs and practitioners working on the frontline. During our post conference musings we wondered what more we might learn from feminist and gender experts who have years of experience using power analysis to inform their efforts to mainstream gender issues both in programmes and in the ways their organisations are managed.

The Politics of Evidence conference provided an opportunity to share and strategise for people working on transformative development, and who are trying to reconcile their understanding of messy, unpredictable and risky pathways of societal transformation with bureaucracy driven protocols. They have struggled to make sense of the shifting sands of the results agenda – seeing the wisdom in some aspects while actively questioning its less useful, sometimes damaging, manifestations and consequences.

We designed it to make the most of participants’ experiences and ideas and everyone had the chance to share these in the conference break out groups, including documented case studies from about a third of the participants. As Lawrence Haddad comments in his blog yesterday on the conference, power pervaded these stories.  We hope that their engagement in such an interactive conference process will have given participants courage and confidence to adopt and develop further the potential strategies and tactics (developed in the break out groups and shared in the final plenary session) to make possible programming and evaluative practice fitting for transformative development.

Over the next month or so – while the conference report is being finalised – the convenors will be blogging about some of the key issues and challenges that the conference threw into relief.  Then, we plan to start work on a book that will explore these issues further, including contributions from some of the conference participants.

The Big Push Forward convenors aimed to throw a stone into a pond to make ripples. We hope these ripples will continue to expand outwards. Meanwhile, by September the current group of convenors will be stepping down in the hope that others come along to throw in more stones – either as the BPF or in some other form.  Contact us if you are interested!

Since a blog cannot hope to convey the discussion, I’ll restrain myself to some threads which I felt stood out:

• Measurement processes, tools, artefacts, can all be positive or negative: it’s not about the artefact – although discourses can grow around them, and some discourses can push in one direction or another – but about the interpretation of the artefact. It’s about the detail of their implementation, and the people and the relationships involved in bringing them in and communicating them.
• Agency, tai-chi and ju-jitsu: people reflecting on their own positions are not just automatons within a relentless machine. There is agency, and there are possibilities to shape the directions of organisations and the way organisations – or the people they work within – understand the world through measurement and evaluation processes. It’s just that sometimes a little tai-chi – or possibly ju-jitsu – is needed to turn people around.
• Disjunctures in scale: that some of the measurement techniques, when used to evaluate interventions and to convince at the level of general policies, do not necessarily work at the level of individual projects. RCTs, for example, are purpose designed for scale-up, but that may not be the case for many of the project evaluations at a local scale.
• Ownership: fundamentally, one of the biggest concerns articulated was about ownership of the evaluations, and who are they for. It’s about programme staff whose projects become strangers to them, or those we are seeking to support who felt themselves robbed of voice in the face of evaluations, experimental design, and the power of evidence.
• There is no Big Bad Wolf proposing mindless tools to do people down: there are repeated, deep, systemic issues in play, coming from a fragmented and highly political environment, dealing with difficult problems. Everyone in the room had their own philosophies and their own ways of pursuing development aims within that system.

These are of course just some personal reflections and take-aways. Tomorrow, the sessions are focusing on whether we can come up with strategies for opening the space for fair assessments for development in this complex system.

Two papers have been prepared for the Conference and these are now on line. Rosalind Eyben’s Uncovering the Politics of Evidence and Results disentangles the historical threads and origins of results-based management and evidence-based policy/programming discourses. She discovers a strong ‘family resemblance’. Both assume that evidence pertains largely  to verifiable and quantifiable facts, and that other types of knowledge have less or little value; both have a particular understanding of causality, efficiency and accountability. The paper looks at how and why these discourses have entered and influenced the development sector and who is promoting them in which contexts  What has been the effect on the sector’s priorities and practices, and particularly its capacity to support transformative development?

Arguing the importance of being critically aware of how power sustains and reinforces the results-and-evidence discourses, Rosalind examines how these discourses generate artefacts (tools and protocols) such as log frames and theories of change that shape our working practices.   When hierarchical ways of working block communications and dialogue, the artefacts trigger perverse consequences but their power is neither uniform nor constant.  Analysing the politics of accountability and the sector’s internal dynamics, Rosalind suggests there is room for manoeuvre to expand and enable more transformative approaches to results and evidence within the sector.

Brendan Whitty’s paper, Experiences of the Results Agenda, paper analyses the data from an online survey, which invited visitors to the Big Push Forward website to give their perceptions of the impact of the results agenda on their working lives. Brendan analyses the very different experiences and interpretations of the respondents as revealed through both 153 responses to the quantitative survey and 109 qualitative stories. The study discusses the day-to-day practice of small-e evidence –results and targets in management of specific projects – rather than large-E evidence of establishing broader development policies. The stories are about the nuts and bolts of the development processes and artefacts – the theories of change, results frameworks, reporting requirements and value for money rubrics. It is about what ‘e’ is being collected, how it is used, and to what effect.

Respondents disagreed about the effects of these artefacts.   The contradictory perceptions seem to be often in tension. Thus learning is often seen to be(?) in tension with accountability; capturing the complexity in evaluation with harmonisation and reductionism; coordination of partners with constraining their freedom to adapt. How these tensions are resolved and the perceptions play out seems to be dependent on how the artefact is communicated, managed and tailored to its context. The fit appears to be important: the fit of the artefact to the existing systems and capacity of the organisation, and also the fit of the artefact to the specifics of the intervention (e.g. its complexity, the number of partners). Finally, perceptions of an artefact seem to be affected both by staff’s’s own circumstances and their relationship with others. The survey data suggests that those in M&E and management roles, who benefit from better data and more resources for their priorities, tended to be more positive than those in project implementation and mid-level roles.

During the conference we will be exploring these ideas and testing these intepretations. Come back tomorrow for the deliberations of Day 1,

The conference coincides with the second anniversary of the birth of the Big Push Forward. We posted our first blog on the 26th April 2011. Our aims have stayed consistent – helping create the political space to ensure appropriate approaches in the design, monitoring and evaluation of projects with transformative aims. This is necessary for the international development sector to continue to seize opportunities to support transformation for greater social justice.

Where we have shifted direction is in how we have implemented our strategy. The Big Push Forward’s core strategy of collective thought and action has been adapted since the initial conception in September 2010 when we held a one day meeting at IDS.   At that meeting, the seventy or so participants identified seven key themes for our agenda and we used these to shape the structure of the BPF initiative, seeking to establish practitioner networks around thematic ‘clusters’. One of them, Value for Money, got much interest, two others gained some traction and the remaining four never got going.  Even members of the VfM network found it difficult to find sufficient time to engage in e-discussions and sharing.

We refashioned our strategy to focus on a single task – an international conference on the central theme of the Big Push Forward that the politics of the results and evidence agenda is playing out in ways that make a frank debate about its consequences hard. The conference preparations achieved what we needed, giving us a clear target to focus our energies, simplifying the structure for the BPF, and focusing the interest of others.  With Cathy, Chris and Brendan, we have had fun as a reflective and supportive steering group.

In the process of challenging ourselves and each other, more nuance has crept into our thinking and messaging. The challenge of thinking through the conference has helped clarify the key questions, sharpened by the wonkwar debates (that proved to be one of  ’greatest hits’ on From Poverty to Power), presentations in places including New York and Geneva, crowdsourced experiences, and thinking through how to discuss sensitive aspects. By working through topics such as Value for Money, reporting, and rigorous thinking, we’ve discussed the positives of the results agenda alongside the problems. All these issues – and more – have found a home in the framing and structure of the conference.

Thank you, Big Push Forward supporters – including our six hundred subscribers – for your interest and encouragement! We hope that making the politics of evidence discussable can prove inspiring for development practitioners.

Utilitarianism is a philosophical tradition that started in Britain in the 18th century. It deals with the question of how to act morally, and what government action is morally best. Put simply, in a utilitarian view, human behaviour is the more moral, the more it creates happiness. In the words of Bentham, “it is the greatest happiness of the greatest number that is the measure of right and wrong”. The utilitarian idea has been very influential in Britain, and more widely in the Anglo-Saxon world. The push for effectiveness builds on this tradition. “Happiness” is now made to be understood as “results”. Governments are “effective” (read: moral) if they produce lots of “results” (read: happiness). To make effectiveness measurable, results should be pre-defined. I am not sure if the architects of the results agenda are aware of their utilitarian background, but we are all heavily influenced by our traditions, and the forerunners of the results agenda (New Public Management, micro-economics and the logical framework concept) are dominated by North American thinking building on utilitarianism. People from other traditions just do not understand the underlying assumptions and are confused. Being German myself, I have observed that German development agencies found it rather difficult to introduce results frameworks. They experienced a lot of resistance from staff, and people were confused for a long time. They disliked the added bureaucracy that comes with the current results concepts. But, I believe, underlying is that the Anglo-Saxon results concept does not fit into German culture.

Different Paradigms

Many Germans, particularly in the social and cultural sciences, are brought up in very different philosophical traditions than Anglo-Saxons. Two philosophies of German origin are particularly relevant to the effectiveness debate.

1. Dialectics as developed by Hegel. The idea is that history develops in leaps and bounces because it is based on contradictions. There are always opposing forces, developments, strands in society, and over time they come to a resolution on a higher level. Nothing is predictable. There are no linear pathways in history – or, in today’s terminology, in social and political development. Power matters, and so do hidden potentials that come into force at crucial moments of development.

Anyone thinking in these terms will find it difficult to fill in a logical framework in a meaningful way. The form gives no space for the surprise that we expect. So meeting the result agenda requirements becomes a tedious or cynical exercise. The results approach intuitively feels wrong.

2. Hermeneutics. Wilhelm Dilthey introduced the idea that there are two traditions of science: the natural sciences (like physics, chemistry, biology, etc.) and the human or cultural sciences (German: Geisteswissenschaften, like history, philosophy, arts, etc.). The core of natural sciences is that they look at their object from the outside (“objectively”) and “explain” what happens. In cultural sciences, we are part of the subject that we look at. There is no “objectivity”, no natural laws. All understanding depends on us, and on context. Subjectivity is a necessary source of insight, and a limitation at the same time. Hermeneutics in cultural sciences is about understanding the meaning, and the hidden meaning, of phenomena. For someone brought up in the hermeneutic tradition, the current results agenda tries to capture development in the field of culture with an approach appropriate to natural sciences. What we identify with the results agenda (linear causal models, deified indicators and a reduction of social change to what can easily be quantified) just do not fit with the understanding of social change that many people have developed during their academic training in Germany. Dialectics and hermeneutics all play out in the fact that out of the four purposes of impact assessment that German NGOs define (see “Quality before Proof”), learning from experience comes first and empowerment of target groups comes before accountability. They also experience that it is difficult to implement such a concept under the current results framework.

Traditions and Results

I am not trying to juxtapose German and Anglo-Saxon philosophy. There is more to both of them. Anywhere, our traditions influence how we interpret the results agenda, and often in a way that defeats the agenda’s purpose.

• Someone who has been brought up in a tradition of strictly following authority will interpret the results framework as an instrument of control
  – no wonder people cook up figures.
• If I grew up in a society where social relations matter most because they give security and meaning, I will have difficulties to focus on material results in my plans
  – no wonder people often focus on immediate needs of those they work with.
• In a cultural tradition that believes in circular processes, linear explanations have little meaning for the development efforts
  – no wonder people are not guided by the plans sent to donors but use them just for reporting, if at all.
• Similarly, if I believe that everything is connected, I will find it challenging to focus on just a few results
  – no wonder that systemic thinking is so much more easily accepted by many African consultants than by Northern consultants.
• In societies where people have learned that there is one way to do things, and tend to follow rules, they will focus on what is written
  – no wonder many people mistake indicators (that are meant to show progress for a larger goal) for objectives (that are to be achieved exactly).

If we want to enhance a positive impact of development interventions, that is: if we want to increase the impact of development efforts, we need to overcome the cultural dominance of one tradition and find culturally appropriate forms of focus on change. People from different traditions need to reflect on their beliefs and bring their views into the discussion. That has been true for a long time. The results agenda has shown that we still have a long way to go.

PBR in the 19th Century

As Europe and North America industrialised and proceeded to colonize the rest of the world, the positivist power of numbers appeared to tame uncertainty in an era of such rapid change. In Britain, the fondness  for measurable facts led the introduction of ‘payment by results’ (PBR) into elementary schools in the middle of the 19^th Century. PBR (aka Cash on Delivery) is when commissioners of services (e.g. a government) pay the service providers only after a pre-determined result has been achieved and independently verified.  The logic of PBR is that there is a manageable level of risk in achieving the result and that service providers must be incentivised to play a more active role.  150 years ago – like today –   the buzzwords were efficiency, value for money (VfM), competition and a balanced budget. At the time PBR was criticised for its mechanistic approach that impeded children’s educational development and sacrificed long term benefits for short term achievements.  By the end of 19^th Century PBR was abolished partly due to the increased bureaucracy and administration costs of verifying the results.  PBR had been proven to be inefficient! Fast forward to 2013 when ‘Public bodies seem to be pursuing the use of payment by results with the vigour of a drunk in search of the next bottle of alcohol’, Jon Tizard writes.. 

Results and New Public Management

‘Results’ discourse reappeared in the mid- 1990’s as part of a broader shift in public-sector management approaches, commonly known as ‘New Public Management’ (NPM). Increasingly dominant since the late 1970s, today NPM in various forms prevails in most OECD countries and is exported through their development agencies to aid-receiving countries. In addition to a strong focus on delivering pre-defined results, NPM includes linking resource allocation to performance, competition between providers of services, greater discipline and parsimony in resource use, and adoption of what is represented as private-sector management practices. Critics argue that performance indicators, when used for control are unreliable: they do not measure performance itself, distort what is measured, influence practice towards what is being measured and cause unmeasured parts to get neglected.

Accountability for results or performance against pre-established objectives is a response to the ‘principal-agent problem’. Because individuals are assumed to be always in pursuit of their selfish interests, policy intentions are likely to be subverted by those designated to implement them.  PBR is becoming increasingly popular in those OECD countries where NPM was first adopted and where less rigorous systems to combat the principal-agent problem have been found wanting. PBR in education has returned. Reporting on one such scheme in the USA the Economist commented ‘ You are transferring from a system where the agents are (to a degree) public-spirited individuals to one that motivates agents to be self-interested’.

Interpreting accountability as ‘the delivery of results’ has also been challenged.  The methods demanded of us to be more accountable can make us less responsible for seriously learning how we can most usefully contribute to transformative social change and be held accountable for our commitment in that respect.

The British Prime Minister has stressed that PBR can tackle intractable social problems, used in the UK and USA to find jobs for the long-term unemployed, break the cycle of re-offending; tackle drug addiction etc.  PBR is used with the front line working with citizens with the least voice and the greatest social distance from those in power. PBR ignores the root causes of their marginalisation. This may explain why PBR is now entering the development sector where people in poverty risk being treated similarly.  In both instances, people are silenced from challenging the technocrats’ framing of ‘the problem’ and from offering alternative ways of thinking about the causes of their poverty.

PBR enters the development sector

In 2008 the World Bank’s Health Results Innovations Trust Fund (HRITF) was established, supported by the governments of Norway and the United Kingdom.  USAID is rolling PBR out across the world in health and family planning programming; the Global Partnership for Output-Based Aid, set up by the World Bank and DFID, is financing PBR projects in a variety of sectors and countries and is also is supporting AusAid and the Asian Development Bank in developing such projects. DFID sees PBR as a ‘promising new instrument’.

Meanwhile, in the UK, PBR is running into difficulties. Tackling long term unemployment is not like ensuring efficient refuse collection. It’s the difference between what Jake Chapman calls unbounded and bounded problems.  The likely effect, notes Tizard, should there be a high risk of not ‘delivering the results’ not for profit organizations will decline bidding for public contracts –  nervous about borrowing the capital for the upfront implementation costs.  I wonder what will happen in the development sector should PBR really take off?   Politicians appear to have forgotten the lessons of history.

 Given shifts in the global political economy and increasing inequality worldwide, Piccotto maintains it is time for new evaluation strategies driven by values and concerns about equity and human well being. A partial and self serving development evaluation agenda determined by aid agencies’ concerns with demonstrating discrete projects increase GNI and achieve short term results is no longer fit for purpose. It is time for approaches informed by economics, psychology and sociology; new metrics of human well-being; and new units of account, e.g. networks. Evaluators are challenged to ‘get off the fence’ and make sure evaluation strategies are driven by evidence of inequality and concerns with improving equity.

Piccotto briefly rehearsed the pros and cons of randomized experimental approaches to producing evidence of ‘what works’ through establishing causality and attribution. He went on to argue there is consensus about their limitations. Aside from costs and ethics, there is no guarantee that evidence of an intervention working (or not working) in one place at a particular point in time has relevance there or anyplace else in the future. Evaluation designs are required that include qualitative approaches to aid understanding of how contextual factors affect the contributions of development interventions to development outcomes. They will also need to produce data that can be quantified. Much of the launch was spent considering if and how complexity science and systems thinking could improve methodologies as development evaluation strives to remain relevant in the new aid architecture.

Not everyone at the CDI launch, like those who followed the wonkwar on the politics of evidence debate, agreed with Piccotto’s assertion that methodological tussles are over. Citing some ‘back of the envelope’ analysis on the political economy of evaluation methodologies several participants commented on the huge investments by the World Bank and other influential donors in institutions like JPAL whose key aim is to build capacity for experimental evaluation designs. Examples provided by presenters resonate with perspectives I have heard from members of African evaluation societies:  incentives created through such training crowd out space for alternative approaches and methods. It is rumoured that there has been an increase in the commissioning of experimental designs some of which may be totally inappropriate for programme attributes and contexts to which they are applied. Those I spoke to were calling for a paradigm in which citizens in aid recipient countries can define their own evaluation paradigm and frameworks to enable their learning and social change initiatives.

Comments from African evaluation society members and presentations at the CDI launch suggest complex operations of power need to be addressed to allow a new politics of evidence driven by principles of equity. Evaluation questions and learning agendas are currently shaped by political imperatives of aid agencies that have to demonstrate they are contributing to short -term results. Some practitioners reported biases in the types of programmes that tend to be chosen for impact evaluation – those that are easy to measure. Econometrics has tended to dominate and often made assertions about its neutrality and objectivity that Richard Palmer-Jones argues do not hold up to scrutiny.

Although academics and practitioners at the conference are working to address these issues, all agreed evaluation is an interested business. Commissioners of evaluations and those supplying services have stakes in producing evaluations that respond to political demands that are linked with funding and existing relations of power within the aid system. The barriers to new actors entering ‘the evaluation market’ can be significant One donor pointed out that his agency’s evaluation procurement processes and scoring systems are biased in favour of mature, white male evaluators with significant evaluation experience; younger potentially able evaluators could never compete.

Biases related to assessments of evaluation methodologies also exist. Some participants cited evidence that the quality of evidence produced by evaluations with critical findings are likely to be subject to more scrutiny and thus are less likely to be published. A point made several times was that if evaluation is really going to contribute to a fairer, more equitable world the evaluation community would need to acknowledge its stakes in evaluation processes, learn to be more reflexive about these biases that reduce ‘objectivity’ and ‘speak truth to power.’   To me it suggested the need for a new kind of politics in evaluation that is more honest about the partial nature of knowledge produced by evaluation practice and strives to better reflect the stakes of the most marginalized citizens in society.

It would seem that most people believe that ‘mixed methods’, amongst other things, are essential in order to make sensible judgements about the effectiveness of development interventions. This would include agencies such as the World Bank, as well as evaluation specialists. DFID has recently commissioned an important study entitled ‘Broadening the Range of Designs and Methods for Impact Evaluations’, which sought to ‘establish and promote a credible and robust expanded set of designs and methods that are suitable to assess the impact of complex development programs’.

Now whilst there is still a great deal of debate about whether there is really a commitment to mixed methods in practice, there are also a number of challenges with implementing ‘mixed methods’ approaches. This post focuses on these issues.

In 2010 the Developmental Leadership Program brought together a number of agencies and programs included the Asia Foundation, the Oxfam International Youth Partnership, Leadership PNG, the Pacific Leadership Programs, as well as a number of staff from AusAID departments. The aim of this collaboration was to explore how best to monitor and evaluate programs that sought to strengthen local individual and collective leadership and associated reform. A series of workshops over 18 months promoted an exchange of experiences about attempts to develop approaches that were tailored to the needs and contexts of these programs, and which sought to develop an appropriate mix of methods and approaches.

In 2011 as part of this process the Developmental Leadership Program assembled a range of evaluations specialists from across the methodological spectrum. This resulted in a paper which I co-authored with Linda Kelly which described the different approaches discussed. We also suggested that whilst evaluators might be largely concerned about the merits and limitations of different methods, managers and decision-makers in development agencies also assessed methods on the basis of whether they met the information demands from their political masters.

It was with this understanding that we then explored with the agencies and programs involved the practical challenges they faced in terms of juggling these methodological and political demands with programs that are necessarily engaged in the messy realm of ‘working politically’, and what strategies they used in doing so.

Some of the conclusions we drew from the experiences of these agencies were:

• Having a theory of change that can identify the complexity of the operating environment and provide a good rationale for program strategy was helpful. This involved developing an understanding of what and who drives change based upon an analysis of political and social relations and processes, including the role of influential stakeholders and the relationships between them. These were not prescriptive theories of action.
•  While all the organisations that participated are working towards long-term, substantial social change, they were also able to identify short- and medium-term change: often in relationships and levels of trust between individuals and organisations. These changes clearly do not tell the whole story, but systematic collection of data about them was seen to be important as this could provide a useful basis for understanding the causes and processes of longer-term change, and might at the same time provide some information to keep donors and their political masters happy. However the precise nature of these changes were not predictable in advance.
•  The participating organisations adopted a mix of methods and strategies to infer their contribution to change within the complex political processes. These included: direct observation; asking participants and observers; using databases and software to assist analysis and pattern detection; supporting organisational and coalition capacity self-assessments; undertaking creative comparisons of the costs and benefits with different ways of working; using social network analysis; and developing case study narratives. An important element in many of these processes were different attempts to elicit feedback from allies and partners, using formal and informal methods, about the quality of the relationship that had been established with them. Many of these approaches are consistent with rigorous ways of assessing causation.
•  Building from current ‘practice knowledge’ to identify several features that are associated with effective social change processes, and to test for their presence. Some of these included: having a solid basis of political and social analysis of the context; investment in the emergence of local developmental leaders and coalitions who are able to act for change; having the flexibility and capacity to act quickly when critical junctures or opportunities arise; supporting locally led processes and development solutions; and working for change over the longer term.
•  The ability to quickly utilise opportunities for change, particularly when there are ’tipping points’ or ‘critical junctures’ requires monitoring that supports nimbleness and agility.  At the same time it is important that M&E continues to collect information in a systematic way. Creating space for regular reflection on the changing context and program-wide analysis seems to have been a critical feature in enabling this balance to be achieved.
•  Most of these programs are seeking to understand their ‘contribution’ to the broader changes associated with their interventions, as much as more directly attributable outcomes. This usually involves verifying their theory or hypothesis about how change happens, which includes an analysis of other influencing factors, and thus is as much about seeking to reduce uncertainty about the contribution being made as it is about ‘proving’ impact.
•  Many of the programs undertaken by the organisations represented at the DLP workshops are complex, ‘messy’ and difficult to communicate to stakeholders and external audiences. These programs are  rarely able to present short, sharp, quantifiable outcomes, and do not wish to ‘claim’ the successes of others (as attributable results measures often demand), even if they have contributed to them. There is often a political requirement to keep successes ‘under the radar’ to safeguard long-term achievements. On the other hand the communication of achievements can often be important, not least to members of the local networks and coalitions involved, and their supporters, as this can help to mobilise further action and broaden coalitions.  If monitoring and evaluation and associated research is going to meet the demands of multiple stakeholders, and actually lead to program and policy adaptation, then the effective communication of what are often complex processes needs to be a central consideration.

 Often supporting partners and networks to ‘tell their own story’ can not only provide some concrete and verifiable examples of achievements, but can also allow the primary actors to determine which of these they chose to make public. In this sense such an approach can simultaneously strengthen domestic actors in their ability to promote change and providesome of the evidence of change that other might need to satisfy their constituents.

•  Given the complex, non-linear nature of the change processes involved, in a number of cases more of a research oriented approach to tracking and explaining change over time is often required. As a result a number of the agencies are seeking to partner with sympathetic academic researchers. Separating out some of the longer term research or evaluation work from the more immediate monitoring can protect the  ‘important’ from the more ‘urgent’ demands of some stakeholders, as well as ensuring that hard-pressed program staff are not overwhelmed by expectations they cannot meet.

Whether these approaches are really ‘playing the game to change the rules’ or simply legitimising current approaches will be very much part of our discussions later this month.

I’ve been taking a fresh look at my own work with a ‘politics’ lens and see it in small and larger forms in many nooks and crannies. Below are some recent work situations from the past five months in which I am directly involved. They illustrate how I see the ‘the politics of evidence’ playing out in designs and decisions around what Brendan has described as the (small and big) E of evidence– the results and target oriented issues, and those that relate to evidence of what works.

• Method before question. Working with an aid agency to think through an impact evaluation, the words ‘really rigorous’ and ‘scientific’ are used in relation to assessing the impact of new service delivery model.  An RCT design is scoped that assumes one can identify comparable clusters. Yet the service delivery being evaluated is highly diverse in implementation, within a very dynamic sector with multiple parallel reforms. The agency involved recognises the need to step back from assuming a particular method is the only rigorous option and to first ask ‘what do we want to know’ and then figure out what’s appropriate.  Can we collectively generate rigorous and scientific alternatives that satisfy accountability needs and also fuel the critical learning they want?

• Sampling. One organization is testing out a participatory impact assessment approach (for large E questions). A good sampling approach is essential, with the assumption being that a counterfactual comparison of a representative sample is required. But other sampling options exist that allow for sufficient scale to generalize, without diluting the in-depth analysis and participatory reflection and action that make possible rigorous findings. We need to meet the needs of statistical rigour, as well as utility rigour. But if we can’t manage this, which version of rigour will be the non-negotiable? And will we be allowed to ‘get away’ with contribution claims or must we comply with narrower attribution norms?

• Grounded theory of change versus impossible goals. In Africa, an NGO has just been given a grant to work with social media to strengthening democracy. Project staff elaborated a more detailed version of their theory of change for which the grant had been given. They located their own efforts within a sea of diverse kinds of change that are needed for democracy to be possible. Doing this, however, left them feeling rather morose as it made it clear that the grant is asking them to achieve the impossible. The funding agency and the implementing NGO both know that the contractually agreed goals are totally unrealistic yet the funder will not allow these to be changed – the ‘small e’ of evidence of results and targets. This insistence is making it impossible to really understand how social media work contributes to political change  (large E). And staff already know they will fail to deliver.

• Alternatives for organizational accountability. In Vanuatu, the Pacific Institute for Public Policy has the mandate to trigger national conversations that serve democracy. Tracking this via a logframe would contribute little – the change process is neither predictable nor linear, which is recognized by funders and implementers. So they have agreed to an approach that tracks anecdotes of shifts and compares these with the evolving understand of desired behavioural changes of key political players. A case where the politics of evidence was not a battleground and commonsense has so far prevailed.

• And then there is the daily politics of evidence around who misplaced the car keys but that is another story altogether!

In wading through the politics, I find it helps to be clear about the underlying worldviews of those involved. I reassure myself that policy is never evidence-based, but hopefully can be evidence informed – and that we all value evidence. None of this is particularly shocking or unique. Yet tensions and tussles around evidence and its politics persist. This and much more will be debated at the Politics of Evidence conference at the end of April in Brighton, the UK.  Stay tuned…