✨ In support of vibes-based KPIs

Tom (M) replied to a previous Policy Unstuck saying “I'd be interested in a future edition that sets out what you (or one of your interviewees) think makes for a good public affairs KPI. We had something at YoungMinds where we kinda did it on vibes…”

So, we talked vibes-based KPIs, and:

1. Remembering that you’re not at the centre of the universe. Campaign planning fails when it assumes everything flows from your actions. Instead, recognise you are but one player in a complicated context.
   

2. The three tests for a theory of change: Is it easy to explain? Is it built around barriers to change? Is it easily rewritable?
   

3. Being careful what you agonise over. Tom’s team agonised over an output, only to find out it was completely ignored by target decision-makers.

See what you think, and if you’ve got an idea for a future edition of Policy Unstuck (or the name of an interviewee you think would be great), hit reply and let me know.

Tom (H)

P.S. the next cohort of our 5 week online training, Generative AI for Policy Communicators, starts later today. If you want to sneak in before it starts, you can register here.

The vibes-based approach to public affairs KPIs

KPIs tend to fall into two categories, and both are problematic. 

The first is definitive: have we won or have we not? Have we secured this change or have we not? Which is often what you're judging yourself on when you're reporting to your board or a funder. 

The second is super output-y: how many consultation responses have we submitted? How many MPs have we met? But these don’t correlate with progress in any meaningful way. 

The sorts of questions KPIs should be formulated around are vibes-based.

If you’re a team that can be honest with itself and the people you're reporting to, then you can answer subjective questions like: have we got the relationship with the minister that we need? And you can grade it from ‘they're not replying to us, zero out of ten’ to ‘we're chatting weekly, ten out of ten.’

At YoungMinds there were a host of similar questions we used: are we being invited to the big meetings? Are they checking in with us before they make a big announcement? Are we the go-to organisation for media comment? Have we got enough MPs batting for us?

You can measure those things and you can discuss them in some depth, but really you’re going on that overall sense of ‘are we making progress we need to in these areas?’

Sometimes pressure for output-based KPIs comes from people who are conditioned to want things you can count as evidence that you’re succeeding. And I suppose that’s where trust comes in.

The fallacy at the heart of many theory of change processes

The key thing that theory of change processes often get wrong is they assume that you’re at the centre of the universe, that everything flows from the actions that you take. If you write a theory of change that looks like that, you’re going to get it wrong.

But if you write a theory of change which says ‘all this stuff is happening around us, we are but a player in it. How do we navigate that external environment?’ Then it can be really useful.

So you're asking yourselves: what are the motivations of the other players? Why are they behaving as they are? Why are they making the decisions they are? And what power have you got to lead them to behave differently, or make different decisions?

To do that, you need to be really clear about what it is you are trying to change. You have to be really focused on who is the person ultimately who’s going to make this call and what exactly you want them to do.

Why isn't your target already doing what you want?

Always ask why aren’t you already winning, or why your target is not already doing what you want them to do.

If you frame everything around that question then you are getting inside their heads: ‘okay, they’re making this decision because they’re motivated by these things, or they’re under pressure from these groups, or they’re in a context which is challenging for these reasons.’ So, the reason why this person isn't already making these decisions is because of this, this, and this. 

If you can whittle that down to a top two or three, then you’re onto something because you can really focus on: what are the biggest reasons why this isn't already happening and what can we do about those things?

The three tests for your theory of change

There are a few ways to sense check your theory of change. The first is whether you can describe it in 60 seconds. If not, you’re probably focusing on too many things. The second is whether it is built around the barriers to change, and the idea that a lot of what you’re trying to do is outside your control. And lastly, is it written to be rewritten?

If you’ve written one which is really intricate and you’ve got all these lines and you're like ‘Oh god, if I move that…’ then you’re not going to rewrite it, and it won’t get updated and used as a living document–it won’t be useful.

In one recent campaign where we were being fobbed off by the minister, that lack of quality engagement  was one of our barriers to change. But once we got in the door, that wasn’t a barrier anymore, so we can take it out of the theory of change, and replace it with ‘the minister is unconvinced by our evidence base’. You don't have to spend ages rewriting the whole thing, just update it so you are focused on what needs to happen here and now.

What are you doing that you think matters, but your target does not?

I had a campaign once which was targeting Ofsted [a non-ministerial department in the UK that monitors education standards–the Office for Standards in Education, Children's Services and Skills]. At the end, I had a meeting with a key person as he was leaving Ofsted and we had a really candid conversation. 

I asked ‘what did you think when we did this or that?’ For example, we made a film and there was so much debate internally about whether it was too punchy–we agonised over it. 

He hadn’t watched the film. No one at Ofsted had watched the film. 

But one of the things that was really powerful was young people emailing Ofsted as part of the consultation. Because there was such a volume of it—and they were a little bit annoyed because there were some safeguarding things that came through—he accepted that the fact that so many people were writing in meant that they had to give that more attention in the room.

It’s a good reminder that often the things that cut through the best are when we put people affected by the policy in front of decision-makers in some way.

Thank you to the 75 Policy Unstuck readers who have referred a friend or colleague.