Reporting and Aggregating Results
In February 2009 Alnoor Ebrahim, unhealthy Associate Professor at the Harvard Business School, gave a presentation on ‘Measuring and Aggregating Civil Society Results’ to a Donor Roundtable Meeting on the “Aggregation of Donor-funded Civil Society Organizations’ Development Results” . See here for the full report, and section 3.3. for a summary of his presentation.
He suggested four types of results that could be captured.
- Added-Up Results (1+1=2): i.e. the whole equals the sum of the parts based on comparable projects/programs. This approach assumes scale is achieved through replication and the implication for donors is efficient, scalable actors.
- Synergistic Results (1+1=3): i.e. the whole is more than the sum of the parts, scale is achieved through complementarity and the implication is fund networks and collaborations that have clear targets.
- Tranformational Results: (1+1= LOVE): i.e. the whole is more than the sum of the parts + there are ‘threshold’ effects or ‘tipping points’ above which more transformational change is achieved. Implications – fund based on evidence-based linkages between activities and attitudinal shifts.
- Harmful results: i.e. by implementing a process of aggregating results, you invest energy in something else than achieving development results.
He also proposed three principles to guide any framework for aggregation and arguably therefore for reporting:
i) Aggregate in order to enable development, not simply to report about it;
ii) Build a process that is legitimate
iii) Keep any framework or guidelines simple and focused
He then went on to suggest that the process of aggregating results requires the development not simply of indicators, but more importantly of suitable measurement methods which may be different for measuring transformational rather than additive results. He saw the crucial challenge as identifying what donors’ collective goals for civil society interventions were, and how to achieve them. “How to achieve them” means developing a theory of change for civil society in order to articulate how diverse activities can have a collective impact on complex social problems.
This he argues is the strategic challenge; it is harder than the measurement challenge. A key challenge therefore for donor agencies, he noted, is whether they want to use results aggregation purely for functional accountability purposes (i.e., for activities and results already funded) or also for strategic accountability (for results they desire to achieve in the future).
What do you think about this? What other papers, or examples of approaches to aggregation would you like to share?
Comments are closed.
Thanks for this post, I think it highlights generic problems associated with attempting to aggregate within agencies and indeed within complex programmes that have implications for approaches to assessing results and value for money. In the past few years I have seen examples of efforts to aggregate results to be functionally accountable within individual agencies or programmes having harmful results that have undermined and/or ignored more strategic accountability concerns and thus the achievement of synergistic or transformational results. The learning components of programmes that have been designed to enhance transformational potential have been put on a back burner while staff have struggled to push partners in many different countries to articulate ‘results’ in metrics that can be aggregated. When I was doing ethnographic research with local NGOs in Cambodia in 2006, long before the current results agenda I was horrified by how much everyday office time was dedicated to thinking about how to meet donor reporting requirements. What I saw was so consistent with Ebrahim’s earlier arguments about NGOs learning and change. Local staff began to perceive the success of what they did in terms of their ability to conform with donor reporting requirements rather than listening to and learning from poor people. Opportunities for double or triple loop learning were scarce.
I fear this tendency may be exacerbated as pressure to demonstrate aggregate results especially in terms of value for money grows. Probably the most important question post Busan is what drives a desire to perform such aggregation? Surely it is functional accountability to back donors and or market positioning which reinforce inequitable power relations. Such approaches seem to forget that organisations pursuing transformational programmes may be using short-term donor projects with some immediate MDG outcomes for a limited number of people to contribute to longer terms synergistic or transformational changes in political cultures and relations between states and citizens. I was recently involved in a VFM study for a short term donor project that was part of a CSO’s longer term theory of change and the very exercise which was driven by the need for 1+1 aggregation pushed us towards developing an artificial construct we could measure and aggregate at the risk of ignoring all sorts of other value being contributed to by the recipient and partners.
If one is serious about recipient citizens and governments owning aid then surely the relevant levels of aggregation have to lie there? Why can’t we admit that if we need to aggregate for donors, then the aggregates are going to have to be articulated in quite simplistic forms and involve a great deal of estimation. I wonder whether it is really possible to do better at aggregate level than estimate the number of people affected by programmes in particular sectors? If we could admit that is as good as aggregation for functional accountability can get, then development practitioners could focus more effort on developing high quality fair evaluation approaches that prioritise learning as an indicator of performing strategic accountability, driven by the need to learn to be accountable to citizens and governments participating in aid projects. Then I suspect development aid could be more effective and accommodate love and sums that don’t add up. Wouldn’t it be better if demands for results for aggregation were driven by partners in recipient countries? Surely in the long run this would lead to evaluation approaches that offer tax payers better value for money?