Context
HM Treasury supports the Chancellor of the Exchequer in defining fiscal and monetary policy, setting the direction of the UK economy and working to achieve strong and sustainable economic growth. HMT is organised into 12 specialist groups, each focusing on specific areas of economic policy, such as Financial Stability, Public Spending and Enterprise Growth.
HMT aims to ensure the stability of the macroeconomic environment and financial system and increase employment and productivity. To achieve these goals, HMT must be able to make informed decisions. Most government policy is data-driven but this is especially so with economic policy. Therefore, all recommendations and Ministerial questions are underpinned by data and insights from a broad collection of statistical models, surveys, other government departments, financial markets, and external economic research organisations.
The challenge
One of HMT's unique characteristics is that it is a net importer of data from other departments. Most data is acquired from within government departments or from research organisations, which creates a substantial data acquisition, transformation and cataloguing challenge. Data volume is not a typical problem, but data variety is enormous. Alongside this, the work covers both methodical, well-planned and repeating data transformation processes as well as fast-past, typically urgent, ad-hoc questions received from Ministers and officials.
With a large set of inbound data, mixed workloads, and rapid turn-around demands, data analysts have learnt to use their skills and tools to meet demands but with limited time to automate process flows.
HMT recognised there were opportunities to increase efficiencies in its data management processes; to improve data sourcing, standardising, sharing, re-using, exploiting and visualising data.
How NTT DATA helped
NTT DATA designed and led a series of structured workshops with a mix of data analysts and policy specialists across the organisation, which identified hundreds of findings that were grouped into key headlines.
Overall, the workshops discovered a talented organisation that was struggling to consume more and more data without the capacity, focus or expertise to define better ways of working. As such, NTT DATA made a series of recommendations, mapped against each finding, which included:
- Increasing automation using a modern data platform with a data engineering capability
- Introducing business and technical metadata capabilities to catalogue source data, models and derived outputs and describe the lineage between them
- Formalising data ownership and stewardship aligned to data domains to enable clear communication of expertise, standards and responsibility
- Establishing a central data management capability to promote good practice, provide expertise, data services and education
HMT had already decided to develop a central data management team - the findings and recommendations helped refine the types of roles and services needed as part of this team.
NTT DATA created a detailed 'reference architecture for data management and governance'. This was centred around the paradigm of multiple data-centric personas, co-existing across the work-mix spectrum of ad-hoc to highly repeatable.
Bringing the capabilities to life
The reference architecture introduced many new processes, tools and technologies, roles, and capabilities that would ultimately impact all data analysts and data flows - a non-trivial undertaking for a fast-paced organisation with limited capacity to receive a large-scale change programme. This was built on the success of the HMTs Development Environment – a dedicated virtual environment for data scientists delivered and managed by NTT DATA – that provided access to a data sandboxes.
The "thin-slice"
NTT DATA recommended beginning implementation by choosing a few key use cases to enable progress towards the reference architecture end-state. These use cases would force the implementation of several key reference architecture features; a new cloud data platform, automation of data acquisition, data transformations, storing data in both human and system readable locations, and establishing a new metadata repository. Thus the concept of the "thin slice" through the architecture was born.
The Chancellor’s UK economic indicators dashboard
HMT identified an opportunity to utilise the 'thin slice' approach to enable the development of a new dashboard for the Chancellor's office. This dashboard drew together many of the key UK economic indicators into a single interactive visualisation, consolidating slowly changing historical measures with near-real-time financial market and economic indicators.
As HMT departments fully understood the data sets, the project could focus on engineering rather than identifying and sourcing new data sets.
Working closely with the HMT project team to build the pilot, NTT DATA brought subject-matter-expertise in the enabling solution components, modern data platform design, cloud infrastructure implementation, security, data engineering patterns and a metadata management solution.
A data lake architecture was defined by extending the Azure Cloud Adoption Framework for HMT's needs and building a Data Lakehouse architecture foundation with Synapse. In addition, NTT DATA initiated a Microsoft Purview implementation, combining automated data asset scans with Data Steward-led data classifications and business definitions.
Outcomes
The project laid the groundwork for initiatives that HM Treasury were able to execute including establishing a full data strategy, understanding the range of skills that the new central data team needed and defining best practice for secure data infrastructure. The thin slice project showed that it was possible to break off one part of the reference architecture and use Azure tooling to create an infrastructure baseline quickly. The workshops had indicated the initial focus areas for automation and the payback was rapid. Data champions also had detailed recommendations to cascade to their individual units.
HMT is now well-placed to mature its new data platform and central data management capability. With the introduction of data engineers and data governance professionals under a newly appointed Chief Data Officer, HMT is increasing data-delivery efficiency and rigour, and is better placed to enable future economic policy setting.