Hi, I’m Tony Heap, a Business Analyst working in the Application Programming Interface (API) Platform Team.
Recently, my colleague Lee blogged that the HMRC API Platform had reached 100,000,000 API calls. At the time, most of the traffic was for “private” APIs, but Lee explained that public APIs were on the way, as per our Third Party Software and API Strategy.
Well, that time has come. We’ve released the Self Assessment (SA) Pre-Population APIs available for public use. In this article I’m going to talk about what we’ve learned about building APIs along the way.
What are the SA Pre-Population APIs?
APIs are methods to allow different pieces of software to communicate with each other. In this case, the APIs allow taxpayers and their agents to access HMRC data via third party software, primarily for pre-populating Self Assessment returns (hence the name).
There are six APIs in total:
- Individual Employments
- Individual Income
- Individual Tax
- Individual Benefits
- National Insurance
- Marriage Allowance
The APIs are “user restricted” - the end user must sign in to their Government Gateway account and grant authority for their software to access their data. If the end user is an agent, they must also have an agent-client relationship in place (via form 64-8).
Lesson 1: What’s in a name?
When we build an API, we usually have a specific use in mind. But the whole point of APIs is to be as generic as possible, thus catering for unexpected use cases.
So you will notice that the API names reflect the data they provide - Individual Tax, National Insurance - as opposed to the primary usage (pre-populating Self Assessment returns). These names are taken from our domain model as explained in Richard Baines’s recent blog.
That said, we do still refer to the APIs collectively as the “SA Pre-Population APIs”, and that terminology has stuck. In so doing, we have potentially limited use of the APIs by creating a “tunnel vision” effect. With hindsight we should have avoided this.
Lesson 2: Security is hard work
Security is a big deal at HMRC, especially where personal data is involved.
Before the APIs could go live, we needed 2-step verification (2SV), where users receive a 6-digit code to their mobile phone or landline. It’s been in place at HMRC for individual taxpayers for some time, but we were the first service to require it for agents.
We also needed identity checks, where users answer questions from their passport, P60 or other personal documents. Again, nothing new for taxpayers, but completely new for agents.
Making security work for agents is sometimes challenging. We’re expecting them to have access to either a mobile phone or a single fixed landline in their place of work, and also to have personal documents to hand.
Lesson 3: User research is important
Doing user research on APIs is different from online services because you are one step removed from the end users. But it’s just as important. We collaborated with software developers during private beta, and with their help we also collected feedback directly from end users too (via an online satisfaction survey).
They told us what was right but also what was wrong with the API design. They needed pensions data individually by provider, not as a single aggregated figure. They wanted state pension data, and student loans data, and more. Some developers wanted data available for all PAYE taxpayers, not just those in Self Assessment.
The feedback was invaluable. We now know what version 2 of the APIs should look like.
But...most of the really useful feedback only came once the service was live, even though we shared the API documentation early on. Earlier feedback might have allowed us to improve the version 1 APIs before they went live. We’re still learning.
As part of this learning, we’ve been working with the Government Digital Service (GDS) to figure out how APIs fit into the Digital Service Standard. What do the discovery, alpha and beta phases look like for APIs? How do you do user research when you are one step removed from the end users? Building these APIs has given us answers to some of these questions, but there is more to do.
Lesson 4: Getting APIs live early is probably a good idea
The GDS Digital Service Standard encourages us to iterate and improve frequently.
This is good advice, but does it make sense for APIs? Changing APIs is harder than changing online services because of the knock-on effect of “breaking” changes on API consumers.
Clearly, there’s a balance to be had but, based on our experience with these APIs, I would say that getting something “out there” early, and having to suffer the pain of change, is more valuable that waiting until the design is “just right”.
Lesson 5: API documentation is important
Initially our API documentation was minimal, and focused very much on the technical design - field names and formats. During private beta, we discovered that what developers really needed to know was the business rules. When does the data become available for a given tax year? How exactly is the total pension amount calculated? And so on.
As queries came in from developers, we made a point of updating the documentation to include the answers. We shared a log of known issues. We now have a much better idea of what a good API specification looks like.
Lesson 6: Supporting API services is tricky
When end users are having problems with an API service, getting an answer might not always be straightforward. Some problems can only be resolved by their software provider, others require HMRC’s help. And there are different HMRC support teams depending on whether your problem is with authorisation or tax data. We’re doing our best to find the right solution.
That said, end user support calls have been a valuable source of user feedback. By watching for patterns in support calls we have been able to improve the service and/or the documentation.
Conclusion: We’re not there yet but we’re getting there
We have learned a lot, but there is more to do. This is reflected in overall user satisfaction, which on average was 3.1 out of 5. We’d like to improve that.
However, the APIs are proving useful. Agents involved in the trial reported a 16% reduction in the need for agent-client contact and a 19% reduction in the need for agent-HMRC contact. And they are increasing accuracy - agents have also reported finding employments that their clients had forgotten.
We’d like to say thanks to everyone who has helped us get this far, especially the software developers and end users who provided invaluable feedback along the way.
Tony Heap
Check out our current vacancies. They're updated regularly so worth keeping an eye on.
Now you can follow us on Twitter @HMRCdigital
To make sure you don't miss any of our blog posts, sign up for email alerts
1 comment
Comment by Sanjay Waghela posted on
This is indeed an excellent article with brilliant results
Lots of listening to feedback and thought processes to move forward
Well done to all involved