Case study: Innovative Horizon Scanning Identifies the Science and Technology of the Future, Dstl

Case study details

Team name and Department

Discovery Analytics Team, Defence Science and Technology Laboratory (Dstl).

Situation and Action

Horizon scanning for emerging science and technology (S&T) is a critical but complex challenge for defence.  With hundreds of thousands of academic papers and scientific articles published monthly, manually identifying disruptive or novel developments was inefficient and unsustainable.  Analysts previously reviewed up to 800 articles per month with only a 1% “hit rate” for actionable insights. 

 The Discovery team at Dstl’s Futures programme built an automated pipeline to transform this process.  They: 

• Used publicly available online data sources to collect academic and non-traditional content
• Applied topic modelling to cluster millions of papers into 1,000 topics and 50,000 sub-topics
• Identified weakly linked papers in graph databases to surface potentially novel or disruptive content
• Filtered articles using over 200 key phrases and recency criteria
• Designed custom prompts for Large Language Models (LLMs) to assess documents against a range of criteria  

They also engaged users throughout development, incorporated peer feedback, and collaborated with international partners who have similar objectives and expertise. 

Outcome / Impact / Results

The impact has been transformative: 

• The pipeline now automatically triages over 300,000 articles per month
• Analyst “hit rates” have increased from 1% to up to 40%
• The system enables faster identification of strategic opportunities and threats in S&T
• The team’s work has drawn interest from across government and internationally
• Plans are underway to share the code more widely, amplifying its impact

Without this innovation, defence would continue to rely on slow, manual processes, risking missed opportunities and delayed responses to emerging technologies.  

Feedback and endorsement

• Feedback from peer reviewers and international partners has been positive, with constructive suggestions already incorporated. 
• The team’s work is recognised as a benchmark for horizon scanning in defence and is being adopted more broadly. 
• Dstl’s work won the 2025 Analysis in Government (AiG) Award in the Innovative Methods category, with judges commenting that, “This felt like a real gamechanger/paradigm shift in terms of sheer scale and strike rate, with multiple approaches used, risks recognised, and international peer review undertaken.” 

Further reading

• AiG Awards 2025 – Innovative Methods Award Winners  
• Learning outcomes from Analysis in Government (AiG) Month event 
• AiG Month blog, Horizon scanning and futures thinking: tools for government analysts 

How this work supports the Analysis Function strategy

Dstl’s work demonstrated behaviours which support the Analysis Function strategy by transforming horizon scanning for defence priorities, used LLMs and topic modelling, and collaborated internationally. 

Find out more: a strategy for analysis in government 2025 to 2028

 

Return to Case Studies