Getting on the impact bandwagon

Impact is all

In recent years, focussing on ‘impact’ has become de rigeur in the charity world. As funding gets harder to come by, demonstrating impact has moved to centre stage. Everyone is obsessing about it, worrying about it, trying to prove it, producing toolkits about it. Passion and good intentions are no longer enough, says Dan Corry, the CEO of New Philanthropy Capital: ‘In the charity sector, impact is everything’[1]. As charities try to adapt to ‘payment by results’ and come to terms with ‘social return on investment’, advising voluntary organisations on how to measure impact is a growing industry.

‘At a time when extra financial resources to achieve more outcomes are unlikely to be forthcoming, making the resources we have work harder and more effectively is a must.’ NPC, 2012

You know it makes sense

It is hard not to go along with this. It is not only funders (including individual donors) who want to know they have made a difference. So do we, as voluntary organisation staff, volunteers and trustees. Finding an effective way of demonstrating impact is the holy grail when it comes to attracting supporters as well as funders, and maybe reaching potential beneficiaries too. We need to know what works and what doesn’t. We want to be accountable to those who have invested time and resources into our organisation or project.

Or does it?

So, off we go on an internet search to find the best toolkit. And there is plenty of choice. A quick google search throws up countless guides – from funders, consultancies, specific programmes (it seems everyone wants to invent their own) – and several dedicated websites. But in amongst these, there are quite a few health warnings: ‘Let’s be realistic about measuring impact[2]’, says one; ‘Three reasons why I hate impact measurement[3]’, says another, ‘Talk of charities ‘proving their impact’ is dangerous and misleading[4]’ says a third. Is ‘impact’, just another term in the management lexicon – another process to compete for time alongside performance management, quality assurance and evaluation? Is it different from evaluation? And there is a lot of talk about theory of change – where does that fit in?

What are the pitfalls?

Measuring impact is not easy – and it is certainly not a quick fix.

First, there is the question of how to define impact. What do we mean by it – and do we all mean the same thing? Are we looking for something we can measure in the short-term? If we want to convince funders we probably are – funding cycles are often short – but in the complex worlds in which most of us operate it’s going to take a while to make a difference and there will be hiccups on the way. If we focus on early deliverables, this could well get in the way of the longer-term outcomes we really want.

Then we have to think about methods. How are we going to measure impact? There are any number of challenges to demonstrating that what we are doing is making a difference.

First there is the question of what can be measured. Not everything that ‘counts’ can be counted and few exercises can measure impact directly – so impact is often confused with milestones, outputs and outcomes.   Most of us are engaged in complex initiatives, which cannot be boiled down into a few simple indicators. The nature of change is more complex and multilayered than the things that can be measured in numbers and statistics can capture. So focussing on targets that are relatively easy to measure may distract us from what really needs to be done. Measures need to be tailored to the specific intervention, not taken off a shelf, if they are to be meaningful. And relying on data may replace the trust and dialogue that learning requires.

‘It is vital that organisations identify and assess impacts that are truly relevant to their work, not simply transferred or taken from elsewhere…

There are growing concerns that funders and commissioners requirements are shaping and dominating approaches to impact measurement in the third sector over the needs of service users, beneficiaries and TSOs themselves….

Key questions are what implicit values underpin and characterise this environment, and whether and how public sector performance regimes reflect and capture the specific impacts of TSOs.’ Birmingham.ac.uk

This raises the question of resources: the more rigorous the measurement of impact is, the more resource intensive it is.

‘It’s complicated, expensive and often impractical for early-stage enterprises….It requires a level of research expertise, commitment to longitudinal study, and allocation of resources that are typically beyond the capabilities of implementing organizations. It is crucial to identify when it makes sense to measure impacts and when it might be best to stick with outputs — especially when an organization’s control over results is limited, and causality remains poorly understood.’ Alnoor Ebrahim, HBR March 13, 2013.

Even when millions are invested in evaluation, complete with control groups, the outcomes can look fairly modest – the evaluation of the New Deal for Communities being a case in point. The control group approaches that some funders still want are often highly resource intensive, methodologically difficult and ethically dubious. A rigorous design focusing on outcomes and impacts determined at the start might squeeze out miss unanticipated developments, unanticipated outcomes. And what happens if you want to change your approach midstream? Change is not linear – A does not necessarily lead directly to B.

Then there is the time factor – when can we expect changes to take effect? In early year’s work we might have to wait for more than a decade. Too often funders want evidence of impact before a particular programme or intervention has even finished. Sometimes a sensible question is simply asked at the wrong time[5]. Rarely is the process of impact measurement introduced right at the start, so there is no baseline to measure progress against.

After all this, there is also the question of ‘attribution’. If we do manage to demonstrate some sort of impact, is really down to us? Can we really say it is what we are doing that is making the difference? And if we don’t, does that mean we have failed?   There are many factors that go to make changes. Most of us are part of a wider pattern of provision. And the rest of the world doesn’t stand still. Can we isolate what we have done from all the other factors and interventions that have affected people’s lives? If nothing changes, it may be because things way beyond our control have got in the way. Floods, populations changes, austerity, the loss of a local employer may have far more of an impact than what we have done. Sometimes standing still in the face of adversity is an achievement in itself.

‘…development and change are the results of wider processes that are the product of many social, economic, political, historical and environmental factors – including power struggles between different interest groups. Understanding these processes is important, if the changes brought about by a given project or programme are to be properly situated in their broader context.’ Save the Children Fund (2003) Toolkits: a practical guide to planning, monitoring, evaluation and impact assessment, p. 127

So should we forget about impact?

Of course not. Despite all this, thinking about impact is important. It forces us to be clear about what we are trying to achieve and how. It focuses us on ends as well as means; it directs our attention to long-term as well as short-term aims; it can help us to learn and adapt, to test our assumptions about what needs to be done to effect change.

But it needs to be done in a way that is realistic, proportionate and fit for purpose.

‘Proof is a big concept and in social science – which is what impact research is – it is almost never found. Scientific investigations rarely prove things – disproving things is much easier – but they nonetheless help us understand how things work and hence make educated decisions. They inform our judgement by reducing uncertainty – but not to nil.’ Thirdsector

Thinking about impact should be about learning and communication. It can be done in a way that is rooted in your values and the way your organisation works and enables you to have a genuine dialogue with your funder and other stakeholders from which you can all learn.   This is where theory of change approaches come in – getting everyone round the table at the beginning to think about what change they want, how they want to make it happen and how they will know whether they are on the right path. Be imaginative about how you can tell your story.

‘If is often useful to have an introductory workshop to discuss what the different interest groups mean by impact. This can help create a shared understanding of the process and help decide which areas of change to focus on.’

Save the Children Fund (2003) Toolkits: a practical guide to planning, monitoring, evaluation and impact assessment

This involves both you and your funder thinking about change as a journey:

1. Are we clear about where we are going and why? Make sure that everyone is on the same page but first allow time for honest discussion so that you are clear about where people are coming from.
2. Be realistic about what you can find out and what you can do within the resources and time available. We don’t all have to do random control trials – it is possible to agree indicators that are meaningful to all parties and can provide a plausible account of how the intervention is likely to lead to the intended impact.
3. Be clear about why you are doing it and make sure that it is relevant and worthwhile. Use the process as an opportunity for reflection, dialogue and learning throughout – an opportunity to take stock together along the way.
4. Be honest – we learn from what doesn’t work as well as what does. This means funders need to respect that honesty too).
5. Combine resources. If you are a small organisation with limited resources, it may work to team up with others like you to learn together; if you are a funder, bringing projects together to work on assessing impact with their peers can be a powerful way of learning.
6. Agree what you are going to do with it.

[1]www.theguardian.com/voluntary-sector-network/2014/feb/24/charity-impact-measurement-results-outcomes.

[2] hbr.org/2013/03/lets-be-realistic-about-measuremnet.html

[3] blogs.ncvo.org.uk/2-15/05/18/three-reasons-why-i-hate-impact-measurement/

[4] http://www.thirdsector.co.uk/talk-charities-proving-impact-dangerous-misleading/article/1376815

[5] blogs.ncvo.org.uk/2-15/05/18/three-reasons-why-i-hate-impact-measurement/