Now in its fourth year, the Civil Service AI & Data Challenge has just seen a 160% increase in applications. Bill Wilson delves into the data to see what we can learn about the challenges and opportunities preoccupying civil servants
2025 ended on a high: when civil servants were asked to send in their ideas for how to make better use of data and AI in government, we were deluged. There were more than twice as many ideas as last year – with entries to the Civil Service AI & Data Challenge rising from under 100 to more than 250!
That’s a tribute to the work of our ‘Sponsors’ – the senior data leaders recruited across government to represent their departments within the Challenge – as well as the promotional work undertaken by our ‘Champions’ and Judges. It also helps that previous winners’ and finalists’ ideas are now up and running – with Natural England blogging about a peatlands project built around the winner in our inaugural Challenge. Moreover, it also shows how civil servants are appreciating the possibilities of data and AI like never before. Delving into the entries themselves gives one an insight into the direction of travel.
As NTT DATA’s AI & Data SME attached to the Civil Service AI & Data Challenge, I’ve been privileged for the last four years - perhaps uniquely so - to gain an insight into the latest in AI & Data innovation in UK government. The Challenge is a cross-government competition that invites civil servants of all grades to submit ideas for how to use data more effectively in service delivery and policy-making; we added AI this year, tapping both into government’s new policy drivers, and to the technology’s rapidly-evolving capabilities.
The competition therefore provides a snapshot of the latest ideas that civil servants have for improving policy and front line services with AI and Data. Applications closed in November and we are in the sifting process in consultation with civil service leaders. Part of my job is to classify the ideas – and funnily enough it’s not a job that AI does well, because ChatGPT lacks the detailed understanding of how the UK government operates that is essential to the task. The agent complained that there wasn’t enough context for most ideas – which is probably what an intelligent human who hadn’t worked in government would say. Here are the trends this human mind discerned!
A record breaking year
Within the 252 ideas, a wide variety of concepts and approaches can be found. I categorised them in two dimensions – first into 18 types of data or AI solution, then by the policy field – covering 62 topics, ranging from accessibility to warehousing. We will dive into these a little more below, but suffice to say they represent the effective intersection of AI’s value-add and the business of governments.
It is immediately obvious that the internal training (e.g. that provided under this year’s One Big Thing initiative, which is focused on AI) has paid off: civil servants have a good idea of how AI can be leveraged and how to address risks from the outset. As every year, there are a few entries pushing a particular agenda to which ‘enabled by AI’ has been tacked on. These aside, civil servants are going after strong AI use cases such as:
- Internal-facing agents for job roles such as work coaches or Civil Service communications officers;
- AI agents to help citizens with everything from tax and benefits to energy and healthcare;
- Augmented analytics –AI assistants that can answer what-if questions from curated datasets;
- Intelligent case management;
- Intelligent document redaction.
Thinking big
As in previous years, there were several ideas about transforming both AI and data across government. I applaud these people who want to see systemic change – although the Challenge is not always the ideal vehicle for this. Examples include:
- Re-usable technology platforms;
- New frameworks e.g. for sharing AI-related metadata;
- Synthetic data generation;
- Process re-design tools;
- Joined-up data views across departments covering themes such as planning, deprivation and early years provision.
Building on government knowledge assets
A few ideas highlighted the huge treasure trove of knowledge and data that the government has accumulated over decades and how to use AI to open this up. In fact, a couple of ideas centred on the historic interest alone of archive data – and some recent policy decisions have allowed this data to be exploited more extensively.
One domain governments are great at is detailed and carefully thought-out reports. I’m writing this blog on a plane between meetings, and hopefully it’s none the less interesting for that; we get our insights out quickly. Governments usually move slowly and methodically and act in a considered way. The AI Playbook for Government is one example of a well-considered and comprehensive document. This means that we can direct our AI at well-curated materials, almost instantly drawing out insights from a wealth of reliable sources in a way that would otherwise require weeks of research time.
To give a specific example, fourteen ideas were about making better use of data about delivery excellence. Public scrutiny and independent bodies mean we’ve accumulated a lot of evidence about how not to deliver projects and programmes in the public sector, and AI can beat any one person’s knowledge in these kinds of tasks.
Data as a foundation for AI
There’s a reason the Challenge covers AI and data. Some ideas use information that is provided in the moment: for example, advanced fine-tuned OCR for archived data, which considers one page at time. Others would work from a relatively static knowledgebase such as an agent that can advise about energy grants. However, many AI solutions either require curated knowledge or a data platform in order to work in the first place, and 19 suggestions were of this nature. A further 70 ideas used AI to help curate a dataset that could then be used for analysis; domains included agriculture, climate change, prisons, water, and the distribution of skills in the UK. The best of these 70 will be those that focus on a very specific use case to start with that has a defined payback.
Lessons from industry
Twenty-one ideas were actually achievable using commercial, off-the-shelf tooling (hence falling outside the Challenge); so either participants were unaware of these solutions, or they felt that government needs were too specialised to be met by them. Some classic AI use cases in industry turn up in government, including:
- Contact centre agents;
- Translation and transcription;
- Media sentiment analysis and background checking;
- Internet of Things solutions.
What’s next?
The initial sift has reduced the list down to around 100 ideas, all of which have been classed as high-potential by both the project team and our network of Sponsors across government. This is phenomenally good relative to previous years (see my previous blog ‘The Civic Datapreneur’ about what makes a good idea). This doesn’t mean there are 152 hopeless ideas, but instead that the format of the Challenge and the focus of government is around the down-selected set. Although we are proud of two live solutions from past editions of the Challenge – with more in the works – we want to see an even higher hit rate. Previous winners and runners-up have lacked internal sponsorship, and we’ve made a host of changes to the programme this year to improve our implementation hit rate.
I’m excited not just by the technology possibilities of these ideas, but more importantly how we can deliver change for the UK public. If that sounds like something a politician might say, in a way you’d be right: I started my career on the bottom rung of government 32 years ago, and I’m still excited by the work.
To paraphrase Liberal leader David Steel’s 1981 speech: “Go back to your constituencies and prepare for AI”. Steel, as it turned out, was over-optimistic about his election chances. But as the Challenge has shown, AI ideas proposed by civil servants really can turn into powerful new public services. I very much look forward to helping our partners in government realise that potential.