In today’s interview, Anna highlights the challenge of measuring influence: we often measure the things we can, even when we know they don’t represent the things that matter.

Influence is our ability to shape how others think or act. And yet in much influencing work we use inward-looking KPIs, like the number of events held or reports published. They measure our activity, not the effect of the activity.

We don’t have to drastically change these KPIs to make them more useful. Rather than the number of events you have hosted, what about the number of people you are trying to influence who attended one? Rather than the number of reports you’ve published, what about how many of those same people invite you to present the ideas?

The reality is that no matter how good our KPIs, we are unlikely to know the full extent of who decided to do something—or not—because of us. Several guests in this series have spoken about how they found out about some important influence they had had not through any formal evaluation, but a random conversation.

It happened to Cast from Clay this week. At the launch of Disability Support Europe on Monday, the CEO of one of its member organisations told Natallia that a workshop she ran earlier in the year led them to rethink their communications. That revised approach helped them secure increased funding for social workers. Had Natallia not been in that room, and had the CEO not told her, we would not have known.

One way to improve the odds of finding out is to create repeated opportunities to talk to and engage the people you are trying to influence (which is what those revised KPIs are looking to do). Doing so requires organisations to evolve from seeing communications as a dissemination tool, towards seeing it as a two-way conversation.

The problem is that most don’t.

Tom

Policy Unstuck with Anna Garrod. Interviewed, edited and titled by Tom Hashemi.

The most frustrating think tank proposals I reviewed when I was a funder told me a potted version of the issues that I was expert in. That would look like a proposal with three pages on the history of the policy area, telling me why the work I was doing was really interesting and relevant, and why the government of the day was interested in it. 

I’m funding this work because I know that. By the time most funders are ready to think about policy influencing, you are already convinced this issue is important enough to spend money on.

What I was really interested in as a funder was: what methodologies are you using? What’s different in what you are offering? What’s going to make this research or this piece of policy development really stand out, and what’s going to make it media-worthy and stand the test of time? 

We heard repeatedly from Department for Education officials that the conversation stalled when ministers would ask how much will it cost, and how much will it save us? There was growing willingness in the system, yet conversations about implementation faltered because no one could answer these questions.  

So we [Impact on Urban Health] commissioned economic analysis from PwC, which showed that for every £1 spent by Treasury on free school meals for children from families in receipt of Universal Credit, there was a return of £1.38. 

The free school meal influencing sector ran with this evidence, Jamie Oliver featured it in his guest take over of the BBC Today Programme and the Mayor of London used the economic case in his roll out of free school meals to primary aged children in London. It allowed us to add an economic argument to the educational, health and social evidence that we already had. 

I agree that many funders still think short-term and go to think tanks when they want that short-term hit. It can feel quite glamorous to get a high-profile think tank to hold an event that’s going to have a brilliant all-star panel. All of us working in policy and influencing know that a report launch is just one step of the journey. The real value and impact is in the engagement that can be done with those findings and policy thinking afterwards.

The shiny outputs are easier to buy and sell because they are visible, and they are still valuable to funders, many of whom don’t have the profile or networks for that sort of external engagement. But we’re kidding ourselves that those flashy external moments are where the real impact happens. The profile and exposure needs to be accompanied by quieter, more sustained and targeted engagement.

→ Read Todd Moss’ Policy Unstuck for more on this.

The think tank model where you work on 40 different issues, and have short-term funding for a two or three month sprint on a piece of research is increasingly outdated. As a funder, you will get the most from a think tank if you properly understand their strategy and overall programme of work. You don’t want to be the project that sticks out on a limb, that feels extraneous to everything else that is being worked on. 

For a funder who has to be politically neutral, ensuring balance was always important. We’d intentionally work across a spectrum of think tanks—politically aligned, cross-party, politically neutral. That wasn’t just to give us the balance we needed as a charitable foundation, it was also the most effective way of shaping a debate.

Funders often think they can commission a set of policy recommendations from a think tank, and they can’t. What you can commission is good minds, an interesting methodology, and something that will contribute to a debate on an issue. You cannot commission somebody to say what you want.

It is always going to be most powerful if you have varied contributions to a debate. That meant having different moral and social arguments from across the political spectrum; we purposefully wanted different perspectives.

Everybody working in policy or influencing knows that progress takes years, sometimes decades, to deliver on. Our work is inherently very difficult to measure: you can deliver a programme of extensive public affairs engagement with parliamentary stakeholders, and then the moment of real impact happens in a fleeting conversation on the fringe of an event. 

I remember a MP we were meeting in a previous role. She had a photograph stuck up by her desk of a site visit we’d helped organise, and she said that the photo reminded her of that experience every single day. That turned out to be far more influential than the numerous meetings with Number 10 on that issue, in terms of her commitment to campaigning on it. 

But we can measure meetings with Number 10… 

So we all slip into the language of proxies: think tanks, funders, policy influencing campaigners, we all do it. We say our organisation has impacted x number of policies. We try to put numbers against what we do and claim credit in a way that we all know doesn’t make sense. 

If we could find a way of talking more honestly about the work we do, the impact it has, but also the time it takes to get there, that would be helpful.

→ Read Tom Madders’ Policy Unstuck on vibes-based KPIs.

Thank you to the 146 of you who have referred colleagues to this newsletter. The views expressed in Policy Unstuck interviews are those of the interviewee, and do not necessarily represent Cast from Clay’s view. You can read our editorial guidelines for more information on the goals, process, and formatting of this series.