Sometimes our job is to publish the dots

Every week, an analyst somewhere updates a dashboard.

Before they do, they need to notice a new release, find the data and check what has changed. They may also need to investigate problems or reshape the data before they can use it. This can take hours. Even automated processes can fail when column names, categories or file structures change between releases. Multiply that across thousands of datasets and organisations, and analyst time disappears into routine preparation.

As statistics producers, we often think about value in terms of the analysis we produce. We explain trends, write commentary and build tools. That work matters. But we also create value by making it easier for other people to analyse our statistics.

People use our statistics to solve problems we know little about. User engagement helps, but we can’t anticipate every use. We therefore have a duty to publish data that is clear, consistent and easy to reuse. Sometimes our job is to publish the dots, so other people can join them.

Solving the publishing problem once

Explore Education Statistics has been the home of the Department for Education’s official statistics for more than six years. We designed the service around consistent data from the start. It now holds thousands of consistent open data files across hundreds of statistical releases.

That foundation allowed us to add a public application programming interface, or API, in March 2025. An API allows software, dashboards and analytical tools to retrieve data automatically.

The benefit is simple. Users set up a query once and receive new data as we publish it. But the key design decision sits behind the scenes.

We built the API into our existing publishing process. Publishers already prepare their data in the format the service requires. Once a dataset meets those standards, they can make it available through the API at the click of a button.

We did not create a separate publication route. We did not ask analysts to produce one version of their data for the website and another for the API.  That is why the service has scaled.

We launched with 3 datasets. There are now more than 140 available through the API, each bringing together files across successive releases. In its first 17 months, the API has served almost 3 million queries. Roughly one every 15 seconds.

Because we built the API into the publishing process, it can support thousands more datasets with little extra publishing effort.

We solved the problem once, at the right level, rather than asking every statistics team to solve it separately.

Consistent data creates room for innovation

None of this was possible until the data was consistent. Instead of creating another data standard, we followed the widely accepted principles of tidy data. Each row represents one observation, and each column represents one variable.

For example, instead of putting age, sex and subject together in one column heading, we record them as separate variables. This makes the data easier for people and software to understand.

Agreeing common standards took time. Many teams had good reasons for thinking their data was different. But solving those problems once created benefits for everyone. New datasets can use the same approach rather than inventing their own. Users see more consistency across releases. We can also introduce new features, like the API, without asking every team to redesign their data.

We build trust by making change predictable

For automation to work, users need to know when something changes. Rather than treating every publication as a separate collection of files, we now link files across releases and make changes visible.

Publishing a dataset through the API is a deliberate choice based on user need. It also creates a commitment: users must be able to trust that the data will remain available and that changes will be clear and predictable.

We use semantic versioning to show the scale of a change. A major change may affect existing queries. A minor change adds new information without affecting existing queries. A patch corrects an issue in the data.

One dashboard used at the centre of government previously required several days of work every fortnight to stay up to date. It now connects directly to the API and retrieves new data automatically.

The team only needs to act when we make a major change. This gives them more time to analyse the data and expand the dashboard. That predictability means users can automate with confidence.

Value we can’t always see

Millions of queries tell us that people use the API. They do not tell us what people do with the data.

Much of the benefit is invisible. It comes from removing manual work that had become normal without anyone stopping to question it.

Each automated update means someone no longer has to repeat the same process. Local authority teams tell us that data they once downloaded by hand now arrives automatically. They spend that time on analysis instead.

The API has also changed what we can attempt ourselves. We are building a public summary of education statistics that we had previously ruled out. Keeping it current by hand would have cost more than it was worth.

We will never build every dashboard our users need. We will never answer every question that can be asked of our statistics. But we can make sure the data is ready for the people who want to try. Sometimes those people will be us.

New tools will change what people can do with our statistics. That makes the often unseen work of publishing clear, consistent and reusable data even more important.

Good analysis joins the dots.

Good statistical publishing makes sure the dots are there to be joined.