Here in HMRC's Policy Lab, the feedback we get suggests that we're having a positive influence on policymaking. But how can we demonstrate and quantify what is successful about our work and the value we add?
This is a common issue – and a challenge worth tackling. Understanding and proving your value is essential in a world of evidence-based policymaking, as well as the measures of cost-effectiveness that accompany it and its emphasis on “what works”.
Back in February 2019, my colleague Faye blogged about HMRC's Policy Lab and how we are aiming to develop policy in a more open, data-driven, digital and user-centred way. But what can we measure to prove it is working? Hunting for success metrics may feel like searching for the ‘Holy Grail’, especially as delivering new policy is very much done in collaboration with other teams.
Indeed, policymaking is often messy and involves many different people interacting under conditions and processes that are constantly changing. This means that single, binary measures of success or failure are often impractical, if not downright misleading. Rather, we need multiple indicators sitting within broad dimensions of success and failure.
So, with these challenges in mind, the HMRC Policy Lab team took a slot in a recent cross-government policy lab meetup and ran a workshop exploring what we mean by ‘success’ and ideas for measuring and proving it.
A cross-government workshop on success
As always, there was a big turnout for the cross-government session. This included colleagues from the Department for Work and Pensions (DWP), the Department of Health and Social Care (DHSC), the Ministry of Justice (MoJ), the Home Office (HO) and Government Digital Service (GDS) among others. Within a few, rapid rounds of discussion and the use of sticky notes, we had shared our initial ideas of what success means to our projects.
In contrast to hard, measurable facts, this quick burst of work produced many ideas that focused on 'softer' indications of success and that emphasised the quality of collaboration, partnerships and our influence in policymaking. We've summarised them in three, high-level themes.
Theme 1. The relationship with our policymaking colleagues
A strong theme running through the discussion concerned the character of our relationship with our colleagues. Principally, it related to how much of a trusted partner are we? Are our colleagues confident in involving policy labs in their programmes and how far do they accept the validity of agile activities, ideas and processes? Finally, how collaborative and reflexive are our policy partners? Building on this, the next theme went one step further to consider whether our work actually gets used.
Theme 2. Diffusing evidence, ideas and practice
How much of what we do, think and make percolates through the policymaking process and minds of decision-makers? To get an idea, we need to chart not only the spread of the evidence we generate and the ideas we co-produce, but also the diffusion of user-centred, agile practice. For example, did our policymaking colleagues attend our agile events, champion prototyping, and perhaps even develop user-centred Key Performance Indicators (KPIs)?
While evaluating the diffusion of practice is vital for determining how much long-term value we’re creating, it also helps us examine if we’re supporting our policymaking colleagues to become leaders in, and advocates for, new processes and thought.
While these first two themes emphasised the quality of our relationships, the third theme zoomed down to a project-level to consider how well we help our colleagues during their work.
Theme 3. The journey from a shared problem definition to a shared solution
This theme asked how successful we are in supporting our colleagues to identify and coalesce around a shared definition of the problem, and then in encouraging convergence around a solution that genuinely addresses it. Another point to consider is how well we facilitate and inform the journey between these two points.
So, in summary, the workshop emphasised how well we foster good, productive relationships, how well we diffuse a user-centred approach and support our colleagues in applying it, and how well we help our policy colleagues design good policy.
So, what’s next?
Turning the themes and ideas into a framework, dimensions and indicators
Borrowing some further ideas, notably Allan McConnell and David Marsh’s frameworks and dimensions, and Thomas Prehn's use of ‘indicators’ rather than 'measures' of success has led us to transform the three themes we came up with on the day into a framework consisting of a range of qualitative dimensions for explaining project success and failure.
In developing a framework, we want to be able to hold our projects to account against each of its criteria so we can produce a detailed story about whether an engagement was successful, who it was successful for, and explain why some things succeeded while others may have failed. Doing that means we can learn about what we're doing well in our partnerships, collaboration and outputs, and what we can do to improve.
Our framework is in its early stages, and looks like this.
So, like Thomas Prehn, we’re not looking for the ‘Holy Grail’. Instead, we’re building a tool to help us create explanations.
Our next steps involve incrementally testing and evolving our new framework. We hope to continue working on this with our policy and policy lab colleagues from across government.
What about you? Have you done something similar, or are you interested in taking part in developing our framework? Let us know!
Check out our current vacancies. They're updated regularly so worth keeping an eye on.
Follow us on Twitter @HMRCdigital
To make sure you don't miss any of our blog posts, sign up for email alerts