Using qualitative research to align survey questions with respondents’ mental models
Insights from the Labour Force Survey transformation: accidents at work and work‑related illness
Understanding how people interpret survey questions is critical for high quality statistics. The Health and Safety Executive (HSE) commissions questions in the Labour Force Survey (LFS) to gather insights into self-reported work-related illnesses and workplace injuries and associated working days lost.
As part of recent work to transform the LFS , the Research and Design (R&D) team conducted qualitative research to explore respondents’ mental models when answering survey questions on accidents at work and work-related illness questions.
A mental model helps explain how a person understands a concept and the expectations they bring to an interaction.
This blog post shares details of our approach to the research and our findings.
Background and history
Since 1990, HSE has regularly collected data through the LFS, focusing on three main areas:
- self-reported work-related illness
- self-reported workplace injuries
- working days lost due to self-reported work-related illness or workplace injuries
Questions about workplace injuries have been asked annually since the 1993 to 1994 survey period. Questions about ill-health have been asked annually since the 2003 to 2004 survey period (with the exception of the 2012 to 2013 survey period). Ill-health questions were also asked periodically before the 2003 to 2004 survey period.
The Office for National Statistics (ONS) has been transitioning the LFS to a transformed, online-first version, called the Transformed Labour Force Survey (TLFS). The aim is to create a more adaptive and responsive survey. This transformation is designed to better meet data user needs, enhance the respondent experience, and improve the quality of labour market statistics.
Work began on the HSE funded questions at the end of 2018. The R&D team in the Social Survey Transformation (SST) division launched a comprehensive transformation project for the HSE funded questions and HSE worked closely with ONS on the requirements at an in-person workshop at Newport. However, this initiative was paused in early 2020 due to the COVID-19 pandemic. The project resumed in 2023 as the ‘essential tweaks’ initiative. This initiative was designed to address critical issues in the HSE question blocks before the LFS is due to be fully decommissioned. This work evolved in 2024 into a full transformation of the ‘accidents at work’ and ‘work-related illness’ blocks.
Our approach
To transform the HSE survey questions in the Transformed Labour Force Survey (TLFS), SST undertook a research and design programme to inform the design of the questions for online-first use. This research project was guided by best practice in questionnaire design and as well as the principles specified in the Government Digital Service (GDS) for the development of user-friendly digital services. Alongside this, this research used the Respondent Centred Design (RCD) approach which means that the respondent informs the design of the survey, rather than the data use.
The Respondent Centred Design framework (RCDF) was applied across the whole design process, giving particular emphasis to mental model research, which strongly influences question design. Learning about mental models early in development can guide question wording, routing, and context.
Methods
We used a range of research methods throughout this project and used the RCDF within agile delivery.
The Discovery phase
In this phase we undertook the following activities:
- discovery research – this involved gathering stakeholder requirements and previous desk research
- focus groups with field force interviewers to uncover early issues
The Alpha phase
In this phase we undertook the following activities:
- cogability testing – this involved pairing cognitive interviewing (comprehension, memory, language) with usability testing (navigation, interaction, friction points) in an online environment
- retrospective probing to uncover decision processes, expectations, and interpretations
The Beta phase
In this phase we analysed, redesigned and retested our findings in 3 iterative cycles.
Cogability testing was used to gain insights on mental models. The R&D team conducted three rounds of approximately 30 cogability interviews to refine the questions on these topics. The preparation phase for this testing began in October 2023, with the project concluding in January 2025.
We recruited participants who had experienced an accident at work, a work‑related illness, or both. Interviews lasted approximately 60 minutes, and sessions were recorded, transcribed, and analysed thematically. Findings were used to redesign the survey questions and then revalidated through retesting.
Insights from testing
These findings were two of the most significant findings from this project.
Finding 1: The term “accident” did not meet respondents’ mental models and stakeholder remit
Respondents in testing included mental health incidents in their definition of an “accident”. This interpretation conflicted with the requirement of the Health and Safety Executive, who needed to capture physical injuries only for their annual statistics.
“I was thinking of a mental accident I had when I started my current job. I was kind of taken on as a new trainee, but they weren’t really equipped to handle the trainees.”
The implications of this finding :
- the risk of misclassification – this could mean a potential overcount and incorrect outputs
- a participant feeling that the follow up questions were irrelevant – for example, physical‑injury questions being presented to those reporting solely mental health incidents
- burden and frustration – this could mean that respondents feel poorly represented
Conventionally, additional guidance might be added to advise respondents not to include mental health accidents. However, many respondents do not read or rely on guidance during completion, so this was not considered an option. This is covered by RCDF component 6 “design without relying on help“. Therefore, we looked to redesign the screening question in the accident block, and we split the question into two parts:
- “Have you had any accident at work?”
- “Did this accident result in a physical injury?”
Result on retest
The first question allowed the respondent to answer based on their own framing of the question. The second question categorised and routed respondents to the correct follow up based on the requirements of the stakeholder.
Respondents who had mental‑health‑only incidents naturally answered “no” to the physical injury follow‑up. This aligned the data with the stakeholder remit, while preserving a respondent‑friendly experience. As one participant reflected:
“I think at the time, maybe I was thinking mental accident…I answered ‘no’ because it wasn’t a physical injury.”
Finding 2: The transition from accidents to illness created confusion and duplication
HSE collect insights through two separate blocks in the LFS; accidents at work and work-related illnesses. Respondents with both an accident and an illness continued to describe accident‑related outcomes in the illness block, despite the requirement to capture illnesses unrelated to accidents. This led to repetition and incorrect reporting, and the respondent felt confused about repeating information they gave in the accident block.
To address this, we added two questions at the end of the accident block:
- a check for other injuries or illnesses related to the accident
- an open text prompt for respondents to provide any further detail
This allowed respondents to complete the accident narrative, including mental health impacts if relevant, before moving into the illness block that explicitly covers conditions not related to accidents.
Result on retest
Respondents used the open text to finish the accident story and did not continue accident content in the illness block. This reduced burden, duplication, and confusion, and improved the accuracy of illness statistics, while meeting the respondents’ mental model. We also redesigned the screening question in the illness block for those that answered “yes” to accident, reminding them not to include illnesses related to the accident they reported. The question now reads:
“Have you ever suffered from any other physical or mental health conditions or illnesses caused or made worse by your work, that are not related to the accident you have just told us about?”
What this means for survey design
This project highlighted the need for designing with the respondent in mind. Post-implementation monitoring should be carried out to ensure reduced burden and improved data quality.
Our findings present evidence that it would be beneficial to extend mental‑model research to other TLFS blocks and related surveys where concepts are complex or contested.
We plan to share the example of best practice across the Government Statistical Service (GSS) and the Government Social Research (GSR) Profession to build consistent approaches for online instruments where interviewer support is absent.
Acknowledgements
We thank:
- participants for sharing their experiences
- the Health and Safety Executive for requirements and feedback
- ONS colleagues in Social Survey Transformation for working with us and giving their support
Contact us
For further information on methods or design changes, please contact the Research and Design team at Research.And.Design@ons.gov.uk.
Dr Cat Shannon