Last month we finally lifted the Beta banner from our HMRC Developer Hub which helps software developers to integrate their software with HMRC’s APIs, for example to submit tax returns or handle customs declarations. Launched back in early 2016, it currently hosts 48 APIs and manages around a million API calls per day with 99.99% availability.
Once we’d finished whooping, cheering and eating celebratory cake, we decided to sit down and have a think about what we’ve learned along the way.
Here’s what we came up with.
Lesson 1: Things change
Over the course of three years we’ve had four delivery leads, five architects, six business analysts, eighteen developers, four interaction designers and six user researchers. The project has moved offices from London to Shipley. We’ve migrated wholesale from one cloud provider to another. We’ve handled the ever-changing demands and desires of twenty or so API producer teams within HMRC and a hundred or more external API consumers. And we’ve lost count of the number of library upgrades and security patches we’ve made over that time!
This is not really news - as agile practitioners, we know we should embrace change. But it’s one thing to know it and another thing to deal with it.
One thing we found that really helped is pair programming, both in terms of software quality and continuity of knowledge share. Some of our developers love pairing, others are less keen - we have to make a conscious effort to keep pairing, and to rotate the pairs. To help us remember, we’ve put a pairing matrix on the wall as an information radiator so we can see who’s paired with whom recently.
Like many agile teams, we use a wiki to capture and share ways of working and useful knowledge. We know this is no substitute for face-to-face communication, but we also know it helps new team members get up to speed. But it’s all too easy not to keep stuff up to date. And sometimes we forget to read what we’ve previously written. So we have to remind one another to be disciplined.
Another thing that worked well for us was ongoing engagement with our service assessors. Assessment events went much more smoothly because the assessors already understood the service before they arrived.
Lesson 2: Feedback is powerful
We know that the way to deliver good software is to iterate on feedback from end users. But it’s all too easy to think you already know what the users want. This is why following the GDS Digital Service Standard is so important. We have a user researcher on the team who constantly reminds us to test changes to the service with end users. Sometimes the users confirm what we already “knew”. Other times they prove us wrong.
We’ve done over 30 cycles of user testing. User research takes time and costs money, and psychologically it can sometimes feel frustrating that we can’t get on and build our great new ideas right away. We aim to test each and every change with users, and we know that overall it’s worth the time and effort.
One thing we’ve learned is that pop-up polls are really useful. We started our beta phase with a standard “give us your feedback” link in our Beta banner and an optional survey on sign-out. We got very little feedback from these so we introduced pop-up polls on some of the key pages on our service. We get a hundred or so responses every week from these - enough to spot trends and common pain points. We think the difference is that we’re catching the user at the exact point when they are experiencing pain.
Lesson 3: Eating our own dog food has helped*
Early on we decided to build some APIs ourselves. This helped us understand what life was like as an API producer team, and using this experience we adapted the Developer Hub to make life easier for them. We were able to feel the pain points of our users.
It also allowed us to present other API producer teams with some exemplar APIs to show them what a good API looked like.
Lesson 4: Whole team collaboration works well for us
We’re quite large for an agile team - there are 13 of us in total working on the Developer Hub. We used to have a split between the design sub-team and the delivery sub-team with separate backlogs and not much collaboration. We thought that working in two sub-teams was more efficient - fewer people in workshops and so on. But in our retrospectives we kept hearing the same pain points - developers not knowing what was coming, not being involved early enough, and designers not appreciating technical constraints.
To combat this we decided to try working as one large team. We combined the backlogs to give a single story board with stories flowing through from design into delivery. As well as having the whole team at our daily stand-ups, we have twice-weekly story review sessions that the whole team attends to discuss stories in design. It’s a bit like three amigos, except that we have thirteen amigos.
It’s not perfect - not everyone is interested in every story, but we prefer it to the previous silo approach and we’ve stuck with it. We also do story estimation in these sessions, so our sprint planning is much quicker.
The whole team approach also allows us to be more T-shaped (a term used in job recruitment to describe the abilities of people in the team. The vertical bar on the T represents the depth of related skills and expertise in a single field, whereas the horizontal bar is the ability to collaborate across disciplines with experts in other areas and to apply knowledge in areas of expertise other than one's own). For example, our design team can jump in to help with testing, our developers can pair with our interaction designer on building prototypes and other team members can step into the user researcher role when we’ve had resourcing issues.
Lesson 5: Scrum works well for us
At one point we experimented with Kanban, on the premise that Scrum is for beginners and we’d outgrown it. We stopped planning sprints and we stopped estimating stories. But we lost focus and forgot about agile first principles - everything became one long continuous blur of stories. Things that got blocked didn’t get finished. Things didn’t get prioritised well because we didn’t think about how big each thing was, we just did it.
It wasn’t long before we reverted back to Scrum. Scrum has its downsides but overall we prefer it. We like the cadence - the planning, and the sense of achievement at the end of a sprint.
Lesson 6: Prioritisation is hard work but value points can help
Being an API platform, we have more than our fair share of users and stakeholders, each with their own specific needs. Balancing those needs can be hard. One stakeholder wants an API callback mechanism, another wants a sandbox environment and another wants spreadsheet integration. Which should take priority?
Also, for each feature on our backlog, it’s not always straightforward deciding what the minimum viable product (MVP) should look like. Different team members have different views on what is “good enough” which sometimes makes it tricky for the Product Owner to make a priority call. Monday morning design sessions seemed to be especially hard work!
We started using value points to help with this. Value points are like story points but they are a relative measure of value, not cost. For a given story, we estimate the value first and the cost second. Value is, of course, very subjective, and some stories (such as “mandatory” security patches) are harder to value relative to others. Nevertheless we have found that having a conversation about value helps get us all aligned, which makes our team discussions more productive and less abrasive.
We’re still learning how best to use value points, but we’d definitely recommend giving them a try if you have lots of competing user needs and/or lots of priority/scope disagreements on your team.
There’s more to do
Although we have officially come out of beta, we still have a healthy backlog of improvements to deliver. And we will continue listening to our end users on an ongoing basis because the world changes over time so we’ll need to change too.
Tony Heap, Business Analyst
*If you're not familiar with the term 'eating your own dog food' it is a colloquialism that describes a company using its own products or services for its internal operations. It's believed to have originated in Microsoft in the 1980s.
Check out our current vacancies. They're updated regularly so worth keeping an eye on.
Now you can follow us on Twitter @HMRCdigital
To make sure you don't miss any of our blog posts, sign up for email alerts
2 comments
Comment by Tony Heap posted on
Hi Iain, I think you're right. Obviously I only put the Microsoft explanation in to see if people were paying attention, not because I failed to do my research properly 🙂
Comment by Iain posted on
Great article Tony! Many congrats to the team for going live, I've been using the dev portal and apis for a few years and it's been great to see it growing.
Dogfooding.
Back in the 1980s when actor Lorne Greene served as the pitchman for Alpo dog food, the TV commercials were careful to point out that he indeed fed Alpo to his dogs. Consequently, the idea that someone would use the products they were making became known as "eating your own dog food." An alternative explanation for the term I've heard is that each year the president of Kal Kan Pet Food would eat a can of the company's dog food at the annual share-holders' meeting.
I much prefer that origin of 'Dogfooding'