Regulatory Language Processor: insights and outcomes from tagging metadata using Large Language Models

About this research paper

This research paper sets out the process, insights and outcomes from the Department for Business and Trade’s usage of Large Language Models (LLMs) to tag metadata for regulatory content. This project was the result of a collaboration of many teams, including Digital Regulation, Analytical Data Science, and the National Archives. We refer to all the combined work throughout the report.

For more information about how generative artificial intelligence (AI) tools are being used in the Civil Service, please see the Algorithmic Transparency Recording Standard Hub.

Introduction

The Smarter Regulation Directorate Digital Team in the Department for Business and Trade (DBT) is responsible for supporting businesses with their regulatory compliance through better digitisation. We are developing a public facing digital service that brings together regulatory guidance into a single database.

The UK regulatory landscape is fragmented. There are over 100 regulators who publish documents in many different styles and formats across a variety of websites. This includes:

• GOV.UK – for example: the Environment Agency
• GOV.UK sub-domains – for example: the Health and Safety Executive
• regulators’ own websites – for example: the Civil Aviation Authority

This means it can be difficult for businesses to find essential information relevant to them and their business activities. The inconsistency in publishing formats and lack of clear contextual information can make it hard to determine which documents are current, legally binding and relevant. This presents obstacles to compliance. Classifying regulatory documents is a tedious and time-consuming process which is not currently undertaken by any centralised body due to many factors including resource constraints and costs.

To support the creation of a single database for regulatory documents, we have used a Large Language Model (LLM) to ‘tag’ (or label) regulatory documents with ‘metadata’.

Metadata

Why we care about metadata

The act of tagging documents with metadata is the first step in making publications machine readable and fully accessible. It provides an additional layer of information that describes attributes of a document, including:

• summarising content
• labelling dates
• categorising topics

This metadata can be applied to all document formats and styles in all locations. This consistency would allow regulatory guidance to be searched, sorted and filtered from a single database.

The effectiveness of metadata

Open Regulation Document Standard (ORDS) is the metadata standard we used to tag regulatory documents. It is designed for use by all UK Regulators who publish legally enforceable guidance, codes of conducts, standards and similar documents online in HTML, PDF, or any other format. It can also be used by organisations who re-publish or create indexes of regulatory documents, such as the “Find business regulations” service (FBR).

The FBR platform is designed to index the regulatory documents tagged with ORDS. It has a database hosted and held on DBT’s cloud platform called Data Workspace. This database feeds into a front-end, allowing users to search, filter and group regulatory documents on a GOV.UK site. The database can also be accessed using an Application Programming Interface (API), allowing large-scale access to the data. The FBR platform is underpinned by the quality of data published in the database with the ORDS data standard. It is currently in private beta testing, and only contains manually created data from the Construction sector. To expand FBR, we need a reliable and large-scale method of producing ORDS metadata at pace.

FBR users will be:

• businesses
• RegTech (Regulatory Technology) companies – innovators in interpretation and analysis of regulations
• other government departments
• legal advisors

Desired outputs

The desired metadata for regulatory documents can be broken down into three categories: ‘objective’, ‘subjective’, and ‘controlled list’.

Metadata list

ORDS was created by The National Archives working with DBT’s Regulation Directorate. We had several internal consultations and discussions about which ORDS fields would provide useful examples to test a generative approach to metadata. We decided to leave out fields such as ‘related legislation’, which establish a link between guidance and appropriate legislation, as we considered that an LLM may struggle to perform well with these fields. We decided to work on assigning a selected number of metadata fields, to the documents in our collection of texts, known as a ‘corpus’.

Subjective metadata

We considered the following fields to be representative of subjective metadata:

• ‘audience’ – the intended audience of the document, for example ‘construction companies’
• ‘description’ – a brief description of the policy document
• ‘regulatory topics’ – the main regulatory areas that the policy document relates to, for example ‘construction’, ‘health’, ‘import tariffs’, ‘road safety’

Objective metadata

We considered the following fields to be representative of objective metadata:

• ‘date issued’ – the date on which the document was issued
• ‘date modified’ – the last date the document was modified
• ‘date valid’ – the date the policy document becomes effective
• ‘status’ – whether the provisions in the policy are currently active or inactive
• ‘title’ – the main title of the policy document

Controlled list metadata

We considered the following fields to be based on controlled vocabularies or predefined lists:

• ‘document type’ – the type of policy document, for example ‘regulation’, ‘guidance’, ‘promotional’, ‘notice’, or ‘international treaty’
• ‘geographical coverage’ – the country or countries within the UK where the policy provisions apply, for example ‘UK’, ‘Wales’, or ‘Northern Ireland’ and ‘Scotland’
• ‘language’ – the language the document is published in, for example ‘English’ or ‘Welsh’
• ‘sector’ – selected from a predefined list:
  • ‘Agriculture, forestry and fishing’
  • ‘Mining and Quarrying’
  • ‘Manufacturing’
  • ‘Construction’
  • ‘Utilities’
  • ‘Real estate’
  • ‘Wholesale Retail and Consumer Services’
  • ‘Food and Drink’
  • ‘Accommodation, Leisure and Tourism’
  • ‘Media and Creative Industries’
  • ‘Transportation and Storage’
  • ‘Financial and Professional Services’
  • ‘Education and Training’
  • ‘Scientific Research, Innovation and Technology’
  • ‘Health’
  • ‘General business regulations’

Incentives

Advantages of automation

The task of manually tagging metadata can be slow, error-prone and tedious for human taggers completing the task. Several aspects of the process present risks.

Consistency

The tagging of the entire regulatory document landscape will require numerous different human taggers, from both the Regulation Directorate and regulators. Many of the fields are objective, such as ‘date valid’ or ‘format’. These datatypes are susceptible to human error during data entry, particularly with high volumes of documents.

Other fields are subjective, such as ‘regulatory topics’ or ‘description’. These fields require the tagger to read and develop a comprehensive knowledge of the document content. Across numerous human taggers, this leads to variability in answers, interpretations and approaches to the fields.

With LLMs we aim for a similar or acceptable performance at pace.

Speed

The average time taken for an officer to tag a document with ORDS was 5.32 minutes. Taking this average, the estimated 30,000 UK regulatory documents would take an estimated 355 full time employment (FTE) days of consistent tagging. This is a resource and time intensive task. This could not be easily done by one person or even several people.

Flexibility

Parameters for metadata tagging need to be well-defined at the start of the process. Changes to the requirements or definitions of metadata would require retraining taggers and potentially repeating existing tags over time.

Large Language Models (LLMs) for metadata extraction

LLMs offer advantages for automatic tagging of metadata. They are fast and flexible, as they can deal with new formats and provide summaries as well as self-label topics. This flexibility reduces the time and financial burden on human colleagues to manage varied and changing requirements. This does not, however, mean they are perfect and developers should be aware of the limitations and caveats that surround the use of LLMs.

Our success criteria for the project are not for it to be as good as a human or a well-rested archivist. We consider LLM generated metadata fields a success if they are better than not having them at all. Fundamentally, we want to know if the benefits of them outweigh any quality risks in LLM generated metadata fields.

This project offers a test case to assess the effectiveness of LLMs for processing and understanding regulatory documents. This work paves the way for future projects to help understand documents, with the potential to expand out of the regulatory landscape.

Success criteria for this project

Accurate and abundant metadata are vitally important for the UK Government and the public. Use cases include but are not limited to:

1. Government policymakers who need to gather any existing regulatory documents to inform effective future policy or review current regulations.
2. Third party businesses who need to adapt their business based on provisions outlined in government regulatory policy, or to simply understand what regulations are relevant to them or to the wider sector.

As stated previously, there are many challenges with manual tagging. As metadata is so important, we wanted to develop a system that support these varied government and business processes.

Our main focus was to develop a tool that could:

• handle a variety of documents and types of formatting
• tag documents with speed (compared to human tagging) and accuracy

Due to known issues around hallucination, LLM outputs are not taken at face value. They go through a quality control process by human taggers. The methodology maintains a ‘human-in-the-loop’ mentality which informs the application of generative AI in important areas such as government. This is in line with the AI Playbook for the UK Government. This ensures the design of the project is quality assured so AI-generated content is not released without supervision.

Speed and Accuracy

We predicted that the LLM would produce tags at a rate of less than a minute per document. This is substantially faster than a human. This prediction followed from several observations about the practical function of an LLM.

If a document was five times longer than the first, it would take a human five times as long to read, summarise and tag. This does not consider the variety in complexity of documents, which can potentially increase the time taken too.

Even where a corpus is composed of short texts, if the number of metadata tags required doubles during course of the project, this directly corresponds to an increase in the time a human would need to populate all the extra tags.

An LLM provided a strong alternative to this, alleviating both issues. Firstly, it only takes an LLM a fraction of the time it takes a human to read through a given amount of text. So, while an increase in length may increase the processing time, compared to a human, it does not substantially increase the amount of time taken to process it.

Secondly, to an LLM, asking for five or fifty metadata tags is near equivalent. When designed and tested effectively, they are robust to complexity (Jia He, 2024). This is important because it is highly likely that the scope of a project will change during the process, and an LLM tagger can handle these changes.

We tested these predictions in this project, conducting analysis of time taken for both human and LLM to tag metadata. We considered success to be achieved if an LLM takes a shorter amount of time to tag a certain number of documents and metadata fields than a human and the outputs are comparable. This would support the claim that LLMs save time and resource.

Impact

The production of mass metadata, at pace, will have wide benefits. The database created by this project will support the FBR and likely extend to other DBT metadata tagging business needs.

The following users of FBR will benefit from enhanced searchability and organisation of regulatory documents:

• businesses will benefit by reducing the research time and associated costs of regulatory compliance – these can be reallocated to core business activities, encouraging growth and innovation
• RegTech (Regulatory Technology) companies will have access to a unified source of information – this will allow them to develop solutions that enhance regulatory compliance and simplify the regulatory landscape
• government departments will be able to access and analyse regulatory documents more effectively – this will help them improve policy development, work more effectively with other departments , and enhance decision-making
• legal advisors will have streamlined access to an indexed repository of regulatory documents and legislation – this will reduce the time needed for research and enable them to provide more accurate and timely advice to clients

Techniques and methodology

‘Traditional’ tagging techniques

There have been many approaches to overcome manual tagging, aiming to move towards a process that is more reliable, with less repetitive human effort.

Before the emergence of LLMs, other methods existed to carry out programmatic tagging. These approaches have the advantage of being relatively fast compared to manual data review but have drawbacks.

Model selection

Locally hosted models

When investigating the efficacy of an LLM, model choice affects the entire course of the process, from design to results.

For security reasons, this project focused on locally hosted LLMs. This prevents any data being sent externally and processed beyond our direct control. For example, the leading OpenAI models available during the initial phase of the project did not offer the facility to be downloaded and run on a local machine. These models were therefore discounted as an option for the project.

We were therefore restricted to models that could be stored and run locally on a cloud platform. Our ultimate choice of models was in the ‘Llama’ family from Meta, which met these requirements and performed comparably in terms of performance metrics to GPT-4 [Guangyu Hou, 2024]. Although, it should be noted that GPT-4 is usually assessed to perform better than Llama [Dasari, 2024].

There are also several versions of Llama which fit a variety of use cases. For example, ‘micro’ models can be downloaded that require minimal computing power to run, for use cases where computing resource is scarce. There are also quantised models that are also less memory-intensive than their ‘full’ counterparts, but are fine-tuned to perform effectively [Wei Huang, 2024] on types of language tasks, such as generating code or responding to medical requests.

Data Sources

For this pilot assessment, we extracted the text from 829 documents, comprising of PDFs and webpages from a variety of regulators. Documents were chosen to maximise breadth of regulators, sectors and document types, while also including high-priority regulators that had previously been assessed through human tagging. This sample became the full quantity of documents that had been manually tagged and therefore the largest baseline that could be used to assess LLM performance.

Quality Assuring LLM outputs

As mentioned before, LLM outputs are prone to hallucinations. For LLM outputs to be viable alternatives to human generated metadata tags, we need to make sure the results are comparable. It is worth noting that there are several sources of inconsistency for human taggers too. This could come from subjective interpretation of certain fields, as well as human oversight. We would likely never get a 100% agreement on metadata fields were it done by professional archivists. If the disagreement between humans and LLMs are comparable to that of between two humans, we deem the LLM as a suitable tool for metadata generation. The Analytical Quality Assurance process to evaluate the suitability of LLMs consisted of the following steps:

1. The LLM data tags were created for full corpus as previously described in this publication.
2. We asked 2 independent human contractors to create metadata fields. These humans did not see the metadata fields generated by the machine or the ones generated by each other. They were representative of the civil servants and contractors who would typically do such a job.
3. We asked an archivist to evaluate all three sets of metadata tags (referred to as Human-1, Human-2, LLM-1). This was a series of ‘incorrect’, ‘correct’ values.
4. Then we checked if the human-human evaluation disagreed more than the human machine evaluation pairs.

LLM as a judge for improving triage for humans

So far, we have generated the metadata using LLMs, but we have used human review to assess quality across every document and all fields. With this we have adhered to the Government AI playbook principle 4 applying “meaningful control at the right stages”. However, when operationalising our tool, we must find a scalable option to ensure a balance between automation and the level of human oversight.

Option 1

We could use a human evaluator for all metadata tags. While we would have saved some time by machine generating the tags, this level of scrutiny is not an option because a professional archivist would still need to conduct a lengthy review of the tags. This would radically reduce the savings in resources.

Option 2

We could periodically send a random sample of records across all tags to an archivist for evaluation and see if there is any metadata field or record type that needs more human oversight. This is often combined with options for users and the public to correct the data by providing contact details. This is a commonly accepted way of ensuring continuous quality control with saving resources.

Option 3

To improve on Option 2, we could build on the work of Linamin Zheng (2023). The aim of this is to maximise the use of the human reviewers’ time by improving on the random selection method of human reviewed records, using the method widely known as the “LLM as a judge”. This would involve using the method described in Lianmin Zheng (2003) to provide the LLM with the previously outlined metadata fields and then create tags and build in extra quality assurance. We would need a secondary LLM to perform its own assessment of those tags, based on the content of policy documents.

We chose Option 3. The method of using LLM as a judge helps with triaging for human reviewers. We predict that this can help select more complex documents for a human to manually review. Part of the advantage of this approach is that the code for LLM as a judge was easily created by adapting the already engineered tagging methodology.

While humans are still better at identifying difficult cases, we found evidence in the literature for this methodology with “strong LLM judges like GPT-4 matching both controlled and crowdsourced human preferences well, achieving over 80% agreement. This was similar to the level of agreement between two humans” (Lianmin Zheng, 2023). It is worth noting that, due to the subjective nature of metadata fields and human error risks, we do not expect 100% agreement between metadata created by humans. So, we did not expect the LLM to agree with humans 100% of the time either.

It should be noted that based on our literature review and informal testing, the ‘best practice’ in this emergent field is to use a different LLM for the judging than the one used for inference. However, due to the procedural restrictions of getting permission to use different LLMs on DBT’s Data Workspace, we were limited to using Llama for inference and judging. For readers looking at recreating this project, we recommend using a different model to judge.

Development design

Security

As previously stated, LLMs used in DBT must be hosted ‘locally’ in a virtual private cloud (VPC) to ensure data protection and meet cyber security requirements. In context, this means an LLM that is run on a private network with no internet access.

We host our models using Amazon Web Services (AWS) SageMaker. SageMaker refers to a set of tools available on AWS that cater to machine learning tasks. It is a cloud-based platform designed for developers to create, train, and deploy machine-learning (ML) models directly in the cloud.

We used Llama 3.2 3b-instruct through SageMaker.

System architecture

The project was coded in Python, as the industry standard, with all libraries and tools readily available, and easily runnable on the AWS notebooks.

We followed coding best practice to facilitate high quality and repeatable result generation. We used modular design, error handling, configuration management and data validation. This included detailed logging of results and performance.

We used features from several external libraries throughout the custom code we designed. The most notable library was LangChain, which is a freely available suite of tools that allow a more object-oriented approach to handling LLM inputs and outputs. Object-oriented design allows modular running of code and easy setup and import when used in different contexts.

Prompt engineering

‘Prompt’ refers to the input passed to an LLM. DBT data scientists ensured that the input prompt for the model was tested to achieve the highest quality metadata. The team used the following structure for this work:

1. Persona: what role should the LLM be playing?
2. Task: what do you need it to do?
3. Structure: what structure should the output take?
4. Format: what format should it be provided?

Providing clear, structured and concise instructions can support the chances of the LLM generating a useful response. We used the following system prompt:

“You are an expert assistant tasked with assigning metadata to a government policy, provided as a text string. All output must be in valid JSON. Don’t add explanation beyond the JSON. ONLY OUTPUT JSON WITH NO SURROUNDING TEXT.

You will be given a JSON with the fields desired and a description of what they are. Replace the descriptions with the actual field data.

If you don’t know the answer, please don’t share false information, simply return NULL as the value for that field.”

[RLP System Prompt]

This prompt is generalised, lacking any specific reference to the fields themselves, or the policy text. This is because the prompt is the System Prompt, which is required for an LLM to work properly, as this is what it is trained to expect.

JavaScript Object Notation (JSON) is an open standard file format, used for transmitting data between systems. This is the format we needed the LLM to output the metadata to, and why we mention it in the prompt.

We used direct language in the System Prompt to alleviate the issue of hallucination.

As well as the System Prompt, there is the User Prompt. For our tagging, the User Prompt introduces the specific task we want Llama to perform:

“Read the following government policy, then fill the metadata fields provided below, using the descriptions of each field provided for context. Output must be in valid JSON like the following example {{“title”: “Policy Title”, “explanation”: [in_less_than_ten_words]}}. Output must include only JSON.

Policy text:

{input_policy}

Desired metadata:

{blank_data_prompt}”

[RLP User Prompt]

The specifics of the task are introduced in this user prompt. The prompt string is formatted using the policy text itself, and a skeleton (template) JSON that contains all the desired fields mentioned previously, plus one extra one, which provides an explanation.

Code structure

Code was designed and tested to handle various types of documents and automatically format metadata into a structured database. This helped to ensure this project achieved the desired benefits, including speed and flexibility.

Document loading

This is the first step in the workflow and handles the project’s input data. It loads the regulatory content from the internal DBT data storage system and preprocesses text, redacting any potentially sensitive personal information from documents using regular expressions. It also loads in the required metadata fields.

LLM chain

The second step in the workflow, this involves loading the LLM, ready for metadata generation and prompt for the LLM.

Metadata production

Next, a final prompt for the LLM is constructed by pasting together the text of the regulation to be tagged into the template prompt outlined previously.

This final prompt contains the question for each metadata field to tag. This is passed as a single input to the LLM for each regulatory document. Each document was sent independently. The LLM generates all metadata fields, holding the response in programme memory.

This step also handles any delays in response time and other fail safes to ensure consistent metadata production.

Validation

The validation step holds the methods for the judging of metadata tags, as described in section LLM as a judge. It also contains basic metrics of how many tags were successfully produced.

This includes one function that constructs a prompt using the generated tags, desired tags and original policy text, so the LLM can ‘read’ both the generated tags and the original regulatory text, to assess how accurate they are.

Output was also requested as JSON.

Extract and store metadata

This final step stores the metadata within DBT, creating the database that supports the backend of the FBR.

Challenges and insights

Memory

For each inference (or regulatory document), memory is needed to record the response from the LLM. This is the production of metadata. Looping through many documents sequentially (one after another), risks high memory usage and potential overflow (which can lead to crashing).

To mitigate this risk, we ensured that the system was designed to save the output from the LLM after every document. Designing a robust system that captures and informs the user of any crashes manages the risk of losing metadata generation and provides an extra layer of confidence in the tool.

Hallucination

Hallucination is when a LLM provides incorrect information when it doesn’t ‘know’ the correct response to an input.

Occasionally, our LLM output would struggle to output ‘null’ when a piece of metadata is not present in a text. For example, when “publisher was one of the metadata fields we generated, there was an extreme bias to populating this with ‘Department for Trade'” – the example provided in our prompt (this was an instance where our initial metadata examples made the issue much easier to spot, as the hallucinations were things like “Kanto Science Department” and “type: ghost/poison“). For this reason, we reduced the usage of example data in our prompts, only including the structure and explicit instruction not to include anything false. We therefore reduced the effects of hallucination through clear and robust prompt engineering that specifically requested the LLM to not output metadata fields where there was no evidence or data to support it.

Results

The design of this project and codebase means the tool can extract and create metadata for all the ORDS metadata fields outlined at the beginning of this report.  However, the Government Digital Service API (which is how we currently ingest the content of around 27,000 GOV.UK regulatory documents into DBT for tagging) already has some ORDS metadata tags. These fields are ‘title’, ‘date published’, ‘date modified’ (last updated) and ‘publisher’ (organisations). For these 27,000 documents we can therefore take these four fields already provided by the API and use our tool to create all other metadata fields.

This section focuses on these latter fields to evidence the strength of LLMs in summarising and categorising. The ability to extract the former fields is still vitally important for regulatory metadata production of content not on GOV.UK API, including content on regulators own websites, which may not have any metadata.

Metadata creation was performed by three separate entities: two different humans (referred to as Human 1 and Human 2) and the LLM. Due to resource constraints, only 604 of the 829 documents had human created metadata. To assess the effectiveness of the tool, we have produced a variety of statistical and frequency focused tests. To assess correctness of created metadata fields, we worked together with a professional archivist and data scientist from the National Archives, who served as a human judge when assessing the quality of all created metadata. While it is possible to have some variation in evaluation between archivists, we have accepted the archivist’s evaluation as fact in this case to create metrics that can be easily used in statistical techniques. Fields were labelled as either ‘correct’ or ‘incorrect’ by the archivist.

1. A human expert’s evaluation of the correctness of metadata fields labelled by the three entities. This means there was a value of ‘correct’ or ‘incorrect’ for each metadata field. We applied statistical tests on every field to identify statistically significant differences in quality of outputs between the three entities, for all fields.
2. LLM as a judge method: The LLM’s evaluation of the correctness of the metadata fields by the three entities. This was performed identically to the human judgement.

Human evaluation

Chi-squared Tests

Chi-squared tests are statistical hypothesis tests used to determine if there is a significant relationship between categorical variables. The chi-squared test for homogeneity was testing a hypothesis based on how many times the human judge classified a metadata field as correct. The test compares the results of what two humans create against what the LLM creates.

Null hypothesis is where the 3 metadata tags are from the same distribution. This means there is not a significant difference between how often the two humans are correct, and how often an LLM is correct when it comes to metadata generation from regulation content.

Alternative hypothesis is where the 3 metadata tags are not from the same distribution. There is a significant difference between how often two humans are correct, and how often an LLM is correct when it comes to regulation metadata generation.

Table 1: Chi-squared results for every metadata field, comparing the difference between distribution of “correct” scores between the LLM, Human 1 and Human 2

Table 1 shows the results of Chi-squared tests of homogeneity on the 604 human evaluated metadata fields. It identifies if the distribution of the variable (“correct”) differs from that of the others. This means we are testing to see if there is a difference in the distribution of “correct” scores between the LLM, Human 1 and Human 2. Here we assumed that the samples are large enough that the differences cannot be due to sampling-error. We used Bonferroni correction because each metadata field has its own Chi-squared test. The p-value of 0.05 is corrected (or reduced) to ensure we minimise family-wise error rate.

Table 1 shows there was significant differences in the Chi-squared test results for 7 out of the 8 metadata fields. This means that for 7 fields out of 8, the number of correct metadata fields are not the same between human 1, human 2 and the LLM.

The Chi-squared test shows that there are differences between all three entities correct scores, but it does not tell us which pairs are different. The Marascuilo test enables us to compare all pairs of groups (Human 1 compared with LLM, Human 1 compared with Human 2, Human 2 compared with LLM) and identify if those pair-wise comparisons are statistically significant.

Marascuilo test

All pair-wise comparisons in every field, did not have a statistically significant difference.

Full test statistics

Table 2: Full Chi-squared test statistics

Table 3: Full Marascuilo test statistics

For all metadata fields, the results show that there is a difference in correct scores across all groups, but the pairwise differences are not statistically significant. The absolute differences of all pairs of groups are not larger than the critical range. This may seem counter intuitive as we had significant chi-squared results, but this can happen because:

• Chi-squared tests evaluate whether there are any differences amongst groups
• Marascuilo performs a pairwise comparison between groups (entities)

So, while there is an overall detected effect, no single pair differs enough to be statistically significant. The reasons for this include the distribution of differences being spread across the groups and small effect sizes.

In practice, this means that there are consistent differences between what humans create and what LLMs create, but this is not statistically significant. It is no different to the disagreement between what two humans would create.

So, while there is consistent, but small, differences, the benefits of speed, cost and feasibility of LLM generated metadata, provides low risk for using this method.

Overall correctness

For a further look into the metadata fields performance, Table 4 summarises the results of the human judge evaluation, providing the correctness score for each metadata field, split by two human taggers and LLM.

Table 4:  Human judged correctness scores by tagger and field as a % of total metadata fields

Figure 2: Human judged average correctness scores by tagger and field as a % of total metadata fields

This chart shows the percentage of correct metadata tags produced by each tagger (‘human_1’, ‘human_2’, and the LLM) across multiple fields such as ‘Date_valid’, ‘Document_type’, and ‘Sector’. Each field is presented as a separate panel, with bars indicating the percentage of correct tags for each tagger. The bars are generally similar in height within each field with only a few notably shorter bars for ‘human_2’, indicating close agreement between taggers. Variation between taggers is small and consistent across most fields, with no extreme outliers or isolated spikes in performance.

Table 4 shows the LLM does well generating metadata for most fields. While overall the humans were able to maintain a higher standard of correctness than the LLM, the LLM also performed well. The results show that:

• all fields apart from audience and document type achieved over 50% correctness
• 5 fields achieved over 80% correctness
• the LLM’s strongest fields were description, sector and status – this is likely due to the LLM’s strength in summarising and extracting keywords
• while the LLM performance was less than 50% on two fields, it is important to note that human tagger 2 was also less than 50% correct for two fields

Meanwhile, the lowest performing human tagging was the Business Analyst contractor (Human 2). Their skills were not in regulation. This provides reasoning to the low performance on the domain knowledge needed on date valid and regulatory topics. As mentioned in the chi-squared section of this publication, even humans can disagree, or in this case, create metadata differently to another human. However, the skills of this contractor are representative of those who would be contracted to do the manual tagging of metadata.

In reference to the lowest performing LLM metadata fields, a limitation we identified when assessing the results was due to the ‘generalist’ nature of an LLM. The LLM lacks regulatory experiential context, so it will interpret the type of tag output to generate based on its expansive training data.

The human judge marked in a very strict fashion, particularly for audience. An example of this is a document relating to “Oil storage regulations for businesses”. The LLM created an audience tag of “businesses” but Human 1 created a tag of “Companies storing oil”. This was marked incorrect and correct respectively. The LLM did not perform strongly on the lower-level audience that the human archivist wished to see. Further prompt engineering, that includes correct and incorrect examples of the level of detail required by the LLM would likely improve this. Curating a full controlled list is not feasible for audience as regulatory content is relevant to such a wide and varied mix of persons. A defined list will likely never be completely representative of all groups.

Human taggers also expressed that document type is sometimes difficult for a human to definitively classify, due to the complexity and variety between guidance and regulation. The significant reason in low performance is that our description did not explain the document type as a definitive list of “guidance, standards or codes of conduct”. This means the LLM inferred more than we required it to. We have updated this description for operationalisation of the tool from the description seen in at the beginning of this report. After updating ‘document type’ to a definitive list, a sample quality assurance (QA) showed an increase in performance, to 70% correct (up from 35%). This evidences how proper evaluation improves the results of the LLM. We will monitor and review the output, aiming to continue to increase the quality of metadata produced.

LLM as a judge

Table 5 shows how the same (Llama) LLM judged the metadata fields. This table replicates the human judge in Table 4. Metadata fields were labelled as ‘correct’ or ‘incorrect’ by the LLM, just like the human judge did.

Table 5: LLM judged correctness scores by tagger and field as a % of total metadata fields

Figure 3: LLM judged average correctness scores by tagger and field as a % of total metadata fields

The chart follows the same structure as the human-evaluated version, with panels for each metadata field and bars for each tagger. Across fields, the bars are generally higher and more closely grouped than in the human-evaluated chart, showing a more uniform pattern with fewer visible differences between taggers.

In general, the LLM labelled tags as ‘correct’ more often than the human judge did. As previously mentioned, the human judge marked in a much stricter fashion. Looking at the “audience” field, one example was the tag “construction”, where the LLM created a tag of “construction operations”. The human archivist marked the LLM field as incorrect. The LLM marked “construction operations” as correct, providing some insight into the LLM’s higher correctness score of audience in Table 5, compared to Table 3. While the professional archivist is seen as a gold standard for metadata evaluation for the subjective fields, the looser LLM tags can be just as useable from a searchability point of view. Alternatively, it is possible to reduce the number of tags after completing the work of tagging. A restricted list can be created using natural language processes.

The LLM also lacks specific domain knowledge. This means it may label a tag as correct, when the nuance means it may be slightly incorrect.

We have therefore learnt to use more domain specific and restrictive language in our metadata field descriptions to ensure fields can provide higher quality results. These results are also an example of how using the same LLM to tag and judge can be subject to bias. While in its current form we cannot recommend using LLM as a judge as an alternative to random sampling for quality control, the process may be improved by using a different LLM in future to alleviate this risk of ‘marking your own homework’.

While LLM as a judge cannot be used on its own without random sampling, with more work we hope it can be used to pick out complex cases for humans to review. We found LLM as a judge ‘lenient’ because it did not mark as many metadata labels as incorrect as a human did. In the future, we would want the LLM to be a stricter judge that captures more of the incorrect flags and accepts more false positives. While this would increase the workload of human reviewers it would improve overall metadata quality. With further metadata description refinement and the usage of a different model, it is likely our tool will improve in identifying low performing tags by Llama. By improving the extraction methodology and using a different model as a judge we expect further improvement in identifying low performing tags, when operationalising the tool.

For every batch of machine tagged metadata produced, we use random sampling of machine tagged metadata and human taggers too, in addition to LLM as a judge. This approach adheres to quality assurance standards. For example, 30,000 documents tagged with metadata would result in around 400 documents (around 1.5%) chosen for sampling. If we can improve on the performance of LLM as a judge, we can reduce the random sampling frequency and improve on overall metadata quality.

Timing comparison

We also assessed how long the LLM took to tag documents to quantify the reduced time and cognitive burden on taggers because of the LLM’s at pace tagging. To understand exactly how the LLM compares to humans, when performing inference, we recorded the time taken to process each document. Figure 4 presents a histogram of the range of time taken to tag each of the 829 documents.

Figure 4: Histogram of time taken to perform tag inference

This histogram shows the distribution of time taken by the LLM to generate metadata tags for documents, measured in seconds. The distribution is right-skewed, with a high concentration of observations at shorter durations and a gradual decline as time increases. A small number of documents form a long tail at higher durations, indicating occasional slower processing times compared to the majority.

For 73% of the documents, it took the LLM 20 seconds or less to perform inference. It took longer for some very long documents (consisting of 1 million characters and over), but all documents were under 120 seconds.

This is significantly faster than the time taken for a human to tag documents, with an average time of 318 seconds per document. One of our taggers commented that, “in many cases it took longer than 5 minutes (300 seconds), however for certain documents from repeated sources, I was able to copy and paste some of the fields between documents, which saved time”.

In addition, our taggers struggled to maintain tagging quality past an hour of working due to repetitiveness. This meant they had to spread the task into sessions over several days. An LLM can loop through a corpus of any length without any breaks and continue to take on more afterwards. Humans cannot do this, unless they split the tasks between many colleagues, but then you increase the variability of the results.

We would like to emphasise that this reduction in time taken when using an LLM is not seen as a way of entirely replacing the work of a human. Our intention is to provide a tool that frees up time and energy for humans to work on more engaging and useful tasks.

Conclusion

While there is room for improvement for metadata tags produced by LLMs, the experimental first results provide adequate results for more structured regulations queries and a promising path forward. While we found some differences between human and LLM tagging, this project has demonstrated that LLM can perform a first pass of tags and auto-populate fields, so that humans could eventually be partially guided by ‘LLM as a judge’ to highlight difficult cases for human to review more closely. We endeavour to continuously improve the LLM as a judge method with feedback and peer review to achieve this. The initial work was undertaken in 2024. Continuous improvement in models is expected to improve both metadata tags and LLM as a judge methodology.

Our results show that an LLM can generate metadata fields for government documents to a similar level of accuracy to a human, and at a pace unmatchable by a human tagger. We envisage a production platform that takes the learnings and design patterns from this project to help civil servants, archivists and members of the public in better classifying, organising and finding documents relevant to any purpose. Our recommendation for an operational platform would be to:

• use a different model to act as a judge
• maximise controlled list use
• ensure there is a comprehensive quality assurance framework with humans in the loop

Correctly classifying and storing documents may be considered a repetitive and time-consuming task. Nevertheless, it is essential for informing good policy decisions, making accurate historical records, and showing the population for whom we work how their interests are served.

LLMs are a powerful addition to the library of digital tools available to help with this task, and we envisage them becoming a major part of the archiving process in the future.

AI governance

This is one of the first DBT projects to use LLMs. As such, we have done a considerable amount of work to assess the security and ethics potential models. Although not one of the intended outputs for this work, a major output has been the exploration of different models that could be used in public sector settings. We were involved in explorations of:

• how locally hosted models offer increased security when compared with API calls
• how to redact personally identifiable information in documents
• LLMs as a judge for analytical quality assurance

Future work

Building an LLM-powered tool

We strive to continuously evaluate and improve this project.  Through this project we have learnt that to achieve the highest possible quality metadata tags, descriptions should be a controlled list, with a particular focus on terminology and notation that a human would write. This will further increase the accuracy of the tool for further metadata field tagging.

After engineering and building the analytical pipeline needed for this project, much of the underlying codebase can be repurposed and used to power an internal tool, which will generate metadata for other teams within DBT. These projects will go through proportional evaluation to ensure the output from the LLM is suitable and accurate for the new policy area. While similar LLM-powered tools already exist and are well-suited for general tasks, one area in which they struggle greatly is the generation of structured data. This is one which our project did very well, and learnings on how to get an LLM to produce properly structured data could be used to improve the effectiveness of other tools, or create a whole new dedicated one.