Survey development toolkit: an overview of the research objectives and methods for each Agile phase of survey development

Metadata item	Details
Publication date:	10 July 2023
Owner:	Government Statistical Service (GSS) Harmonisation and Data Quality Hub
Who this is for:	Anyone involved in survey design
Type:	Toolkit
Contact:	Harmonisation@statistics.gov.uk or DQHub@ons.gov.uk

The purpose of this toolkit

This toolkit can be used to support the development of survey products for social and business surveys. For example, you can use this toolkit to help inform your approach to developing the survey questionnaire and respondent materials (such as an invitation letter or information leaflets).

This toolkit has been mainly developed by the Data Quality Hub and the Government Statistical Service (GSS) Harmonisation teams (with broader input from other areas) in the Office for National Statistics (ONS). It can be used by social researchers and user researchers across government and other sectors working on survey development.

This toolkit has two aims:

To explain the different Agile phases of survey development, the objectives associated with each phase, and suitable research methods and activities to use to achieve the objectives.
To explain the different research methods and activities associated with each Agile phase, what insights you can gather from each research method, and when it is appropriate or not to use them.

This toolkit does not provide:

an exhaustive list of research methods (you may wish to use other methods, depending on specific user needs — these are what we recommend as standard)
an explanation of each research method in detail
instructions on how to conduct each research method

This toolkit presumes some basic prior knowledge about research methods. Further reading of existing wider literature and resources is recommended to learn more. We have included links to wider literature throughout this toolkit.

This toolkit will include links to third parties. The Office for National Statistics (ONS) have not developed content for or vetted any content associated with these links. ONS do not endorse or have a contract with the third parties for these products. The links are for recommended further reading only and other sources are available.

Throughout this toolkit, we may use terms that are new or unfamiliar to you, so we have developed a glossary to define these terms. We recommend you read this before using the toolkit. We have highlighted some terms below that are important to understand before using this toolkit.

“user” refers to the person using a product or service — for example, this could be the respondent to a survey
“respondent” refers to a person completing a questionnaire — may be used interchangeably with “user” in this context
“data user” refers to the customers of your end-product’s data (in this context, the data gathered from your survey) — these will be the people using your data, for example, to produce outputs or inform policy

How to use this toolkit

We recommend that you read the Agile phases section before reading the research methods section.

Each research project is different, and the specifics of your project will influence what research methods you should use to meet the objectives of the Agile phase that you are in. Think about the context of your own project (such as any constraints you may be facing and the aims of your end-product), read through the details of each suggested research method, then use this information to choose which research methods you should use.

This toolkit is not intended to be prescriptive. Although suggestions are made, ultimately you should decide yourself what research you need to undertake to meet your project’s specific goals. Separate guidance produced by ONS’ Data Quality Hub and the GSS Harmonisation team called “The three levels of Respondent Centred Design (RCD)” has been produced to specifically help you make decisions about the amount of research and design needed for your survey questions and materials. It also informs you about the associated risks to the quality of your end-product if you were to reduce or stop certain activities outlined in this document.

This toolkit has been designed to function as a roadmap for the process of survey development – that is, activities that should be considered when developing a new survey. If you’re developing an existing survey (for example, modifying existing questions or adding new ones) it is possible that not all activities listed in this toolkit will be relevant. As stated, the activities you should undertake will depend on the context of your specific project.

As such, there may be additional considerations or specifics you should consider within your own project. Ensure you consult and engage with relevant experts when planning and conducting various development phases.

When planning your work, and what activities you should undertake, you must consider:

your research aims
what your end-product is (in this context, your questionnaire, or respondent materials)
the purpose of your end-product
your stakeholders’ needs
your available resource
the topic and context of your research.

Through using this toolkit you will be able to create a robust plan for your research project, consider a wide range of possible research methods, and think about how each method may provide specific insights to help refine your questionnaire and materials.

Agile phases of survey development

Agile is a method used for project management and product delivery. Agile delivery focuses on starting with a vision and developing in an adaptive and incremental way. As described by the Association for Project Management, Agile is:

“A family of development methodologies where requirements and solutions are developed iteratively and incrementally throughout the life cycle.”

The fluidity and flexibility of Agile development — and the emphasis on gathering insights regarding user needs early — helps ensure that the end-product is fit for purpose, and meets both data user, and respondent needs.

Agile development is an alternative method to the more traditional Waterfall approach to project management. Waterfall has a more rigid and linear approach, where the end-produce is defined during the early stages of project development. Subsequent development stages would be established immediately after initiation, with little opportunity to deviate from these plans.

Agile emphasises the importance of working in an adaptive and constantly iterative way, allowing you to ensure that your product is developed in conjunction with your users’ needs, ultimately resulting in an end-product that is more likely to be effective and fit for purpose.

There are four main phases of Agile delivery:

Discovery – explore the problem space
Alpha – testing with hypotheses
Beta – building and refining
Live – launch and continuous improvement

Each of these phases has a specific goal within project development. It is important to have a good understanding of the purpose of each phase – in addition to how the work conducting aligns with this purpose – to ensure you’re developing in an efficient and optimum manner. A detailed description of each phase is provided in later sections of this toolkit. More information can be found in the GOV.UK Service Manual.

The benefits of Agile development

There are a number of benefits of developing surveys in an Agile way. This includes:

being able to better react to changes in policy, technology, funding, resources, or priorities
reducing the risk of creating a product or service that is out of date when it is completed
helping to facilitate the creation of user-centred products and services
allowing you to fail fast so you can prevent an issue becoming a long-term problem — the flexible nature of agile development means you can then iterate your project to address the issue

Respondent Centred Design Framework

As well as adopting an Agile approach to survey design, you should also adhere to the components outlined in the Respondent Centred Design Framework to ensure your product is fit for purpose and meets user needs.

Establish the data user need
Research how respondents conceptualise topics — this is known as “mental model research”
Understand user experience and needs
Use data to design
Create using appropriate tone, readability, and language
Design without relying on help
Design optimally for each mode — this is known as taking an “optimode” approach to design
Use adaptive design — this means the interface will adapt to the screen size and display accordingly
Combine usability and cognitive testing — this is known as “cogability testing”
Design inclusively

By conducting research to understand the needs of respondents, you can ensure the products or services we develop meet the following goals:

to reduce respondent burden so that you are likely to maximise response rates and data quality, and decrease risks of attrition (people dropping out from your survey — either during the survey, or between waves of your survey)
are highly accessible and usable so that your products are intuitive and easy for respondents to engage with — designing with accessibility in mind will also help you ensure your end-product is more accessible and inclusive, which also benefits all users
use language that respondents understand as intended so that you gather data of high quality — if your questionnaire is designed in a way that is not clear or easy to complete for respondents, they are more likely to answer questions incorrectly and reduce the quality of your data
results in a positive experience for your respondent so that they remain engaged during the questionnaire and provide accurate answers — this reduces risks of poor data quality, respondent attrition, or respondents deciding not to participate in future surveys due to a poor experience with your organisation
gathers data of high quality so that any findings or recommendations produced from your end-product are accurate, reliable, and valid

Relevant standards and frameworks

There are a number of other relevant standards and frameworks that should be considered in relation to survey development. We recommend considering:

the Code of Practice for Statistics, which sets the standards that producers of official statistics should commit to
the Generic Statistical Business Process Model (GSBPM), which describes and defines the set of processes needed to produce official statistics
the UK Service Standard, which provides information on how inclusivity and accessibility should be considered throughout the design process and at every phase in your research
GSR Ethical Assurance for Social and Behavioural Research and to find out more about ethics in research

There are different objectives you should aim to achieve over the course of survey development. We will give details of these objectives as we explain each Agile phase. While this toolkit provides an overview of each of these objectives, in addition to research methods and activities you should consider undertaking to meet these objectives, it is not a comprehensive tool (it would be impossible to cover the specifics of all different surveys), and it should not be followed completely prescriptively.

Different phases of development require different skills and expertise. You should ensure that you’re engaging with relevant experts to develop your survey at each development phase to ensure your end-product is fit for purpose. For example, this toolkit does not explain the details of how to develop a weighting and imputation strategy. You should engage with relevant experts as part of the survey development process to help achieve technical objectives like this.

Discovery phase

The Discovery phase is the first phase of Agile development. This phase is essential for laying the groundwork of your project and ensuring that product development is informed by insights and data. The activities featured in the Discovery phase are often overlooked in traditional development approaches. It is important that you allocate appropriate thought and resource into the work in this section to ensure your end-product is of high quality.

The Discovery phase is an exploratory phase where you will gather insights to inform later development phases. The aim of this phase is to establish and understand:

what the product is that you are being asked to work on
the needs of both respondents and data users

You will use these insights about your respondents and data users later in your project to determine how to design your product in a way that meets their needs.

When developing a survey, it is important to understand that you should not build or test anything during this phase. This includes designing survey questions and respondent materials. These activities should only be undertaken once the Discovery phase has been completed to an extent where you have gathered a comprehensive understanding of needs. This will allow you to undertake an approach to design that is informed by data and insights.

During this phase, you should engage with data users to thoroughly understand their data requirements. You should also engage with users to ensure you understand their needs at each step in their end-to-end user journey. You should only progress to later development phases when you have achieved these objectives.

You should refer to insights gained in the Discovery phase when you are iterating in later phases. Needs outlined in this section will serve as a basis for all subsequent design and iteration.

Notes

During later phases of development, you may uncover new insights regarding respondent needs. This may mean you should return to the Discovery phase to undertake further research to identify if these new insights have the potential to affect the design

The objectives listed below do not all need to be researched independently. For example, you may use an in-depth interview to establish respondent needs and get a preliminary understanding of respondent mental models at the same time.

Objectives of the Discovery phase

Why this objective is important

The very first step you should take is to understand what you are being asked to develop. During this phase you will also learn about your stakeholders’ needs. For example, you should think about:

what are they looking to measure or understand using the survey?
what will this data be used for?
who will this data be gathered from?

It is essential to ensure you have a comprehensive understanding of the purpose of the products, and the needs of stakeholders and data users. This will mean that your end-product, and all research you undertake to develop it, will align with the main purpose of the work.

To gain this understanding, you should ask stakeholders what they want to measure, rather than asking them for specific questions they want to include in a questionnaire. This will ensure you, or relevant questionnaire design experts, can then design questions that aim to measure what is intended.

You should gather this information early in the development process, as it will be used to validate that the design you are producing meets the needs of stakeholders.

Suggested research methods or activities

Create a document that lists all stakeholder and data user needs for your end-product. You can obtain these needs by running stakeholder engagement activities, such as:

speaking with stakeholders and data users
sending stakeholders and data users a questionnaire that includes questions about their data requirements
reading tender documentation

This process may involve many negotiations. Spend enough time to ensure you have a comprehensive understanding of these needs. Your job is to ensure that the end-product you produce successfully meets these aims.

Why this objective is important

By identifying your target research participants, you will be able to understand who you should be undertaking research with at each phase. You should clarify this information with stakeholders. If your end-product aims to gather data from a specific sub-set of the population, you should conduct research with people from that demographic and gain insights from them.

Suggested research methods or activities

You should identify your target research through stakeholder engagement activities. This information will also be provided by the survey commissioner at the start of a project. If it is not provided, make sure you gather this information when engaging with stakeholders and data users.

The scope of the intended respondents may change during the project. If this happens you should adapt your planned research design to gather insights and test your products with the updated, relevant populations.

Why this objective is important

You should find out if anyone:

in Government or Academia is working on a similar project
has previously worked on a similar project

If anyone has been working on something similar there could be opportunities to work together or learn from previous research. Before you use any learnings, you should check to make sure the research was also respondent centred.

You should also find out whether there are any existing questions across-government which have been tested and may be useful to your research. As well as saving time and resource, this can make your data more comparable, consistent, and coherent.

You can use harmonised standards in your survey without the need for cognitive testing. Harmonised standards are survey questions that have been developed by the Harmonisation team following consultation with the Government Statistical Service (GSS). By using harmonised standards you can ensure the data your survey gathers about harmonised topics is consistent and comparable to other data collected across the UK. Learn more about harmonised standards and design patterns.

Suggested research methods or activities

This information should be gathered “at desk”. This means that you will not gather data yourself. Instead, you will review previous research to get a better understanding of what already exists in a particular area of work.

Why this objective is important

Respondent needs refer to the needs that a user has of a service or product. The service should be designed in a way that satisfies respondent needs and results in the correct outcome for them. For example, being able to complete a survey with ease.

Understanding what your respondents’ needs are will allow you to ensure that everything you design, from your initial prototype to your end-product, is developed in a way that allows and encourages respondents to participate. This will allow you to design prototypes based upon data and insights, rather than your own assumptions.

If you do not research and understand the needs of your respondents, you risk designing the wrong thing. You should avoid designing something based upon your own assumptions of what would encourage someone to participate. It is very rarely the same as what will actually encourage the general public to participate.

Learn more about gathering respondent needs.

Suggested research methods or activities

We recommend considering the following research methods and activities to establish respondent needs:

in-depth interviews
focus groups
pop-up research
participatory methods
card sorting

If working on a pre-existing survey, it may also be appropriate to speak to existing survey interviewers. They may be able to give insights on things like common reasons for attrition. But you should be careful about what information you use. You should not make changes based on assumptions or things that are hypothetical, for example.

Why this objective is important

Mental models refer to:

what a user believes or understands about something
the thought process a user goes through to reach a certain response

This informs how a user will interpret something, how they will think and feel about it, and how they will respond to it.

But it is important to remember that mental models refer to beliefs and personal interpretation rather than objective facts. Each person may have a unique mental model. For example, as a survey developer, you will have a more in-depth understanding of your product than a user who is seeing it for the first time. This means your own perception of your product is very unlikely to be the same as your users’ perception of it.

Therefore, you should undertake research with users to understand their mental models. You will later use these insights to ensure that:

your product aligns with users’ mental models
your product is intuitive
questions and respondent materials are understood as intended

Learn more about mental models.

Suggested research methods or activities

We recommend considering the following research methods and activities to understand respondent mental models:

in-depth interviews
focus groups
participatory methods
card sorting

You should be aware that respondent stories and respondent journeys are the same as user stories and user journeys. The use of the word “respondent” instead of “user” just makes it specific to the context of a questionnaire.

Why this objective is important

Respondent stories describe the respondent and why they are completing your survey. These stories are essential to ensure you develop a questionnaire and respondent materials that meets respondent needs.

You can learn more about how to develop respondent stories on GOV.UK.

Respondent journeys refers to the process that the respondent will go through in order to complete a goal. In this context, the goal is to complete the questionnaire. By documenting each step in the journey, will be able to identify the barriers and difficulties respondents face. You will use these journeys to inform the design of your questionnaire.

Learn more about respondent journeys.

Suggested research methods or activities

We recommend considering the following research methods and activities to understand respondent stories and journeys:

in-depth interviews
focus groups
pop-up research
participatory methods
card sorting

Why this objective is important

You need to identify how you will measure success so that you can evaluate what went well in your project, and what did not go so well. You should decide on the method you will use to measure success at the beginning of your research.

It is common practice to measure success at the end of a project. This allows you to continue doing the things that worked well, and work to improve on the things that did not go so well.

By ensuring you have developed clearly established research aims and success measures during Discovery you can ensure the work you undertake throughout your project is:

relevant
resource-efficient
designed to meet both stakeholder and respondent needs

Suggested research methods or activities

There is no direct research method to determine how you should measure success. Instead, it is about choosing the correct methods and approach for the research. A research grid may be helpful to complete as this will help you make research decisions.

A research grid is an effective tool to define and direct the research aims and associated tasks. It can help you track your progress and can be adapted as you progress through the Agile phases. You should review your research grid throughout development.

Learn more about research grids.

How you define your success criteria will determine the steps you need to take.

Consider the metrics you will use to measure success, such as:

scope — has the project met the objectives?
schedule — was it completed on time?
stakeholder feedback — were the customers’ needs met? Are stakeholders happy with how the project was run?
budget — was the project successfully delivered within budget?

These are some examples. You should decide what is important for your research and create success measures that are specific to the context of your own project.

Begin to think about the evaluation methods you may use to measure success. Learn more about evaluation methods.

Why this objective is important

You must always gain assurance before moving on to the next Agile phase. In Discovery, this means being confident that you have gathered enough insights to progress to Alpha.

In Discovery, you will want to ensure you have all the necessary insights before starting to design and test your questionnaire and materials. This will help ensure you:

have a clear plan for your project
have a comprehensive understanding of your respondent needs and your stakeholder needs
design in a manner that meets the needs of your respondents and stakeholders

Remember, if you discover more insights about respondent needs during later phases, such as your Alpha, you can return to Discovery to gather more insights about these needs and to inform your development further.

Suggested research methods or activities

There is no direct research method to check you are developing the right product. Instead, you should make a judgement about this based on the insights you have gathered and the aims of your research.

You should be confident you have the necessary information to be able to start developing respondent-centred products. If you find yourself saying “I think a respondent would need X to do Y”, this indicates that you do not have sufficient insights to inform design. You will need to do more research before prototyping. It is also important to check that what you are planning to develop will meet your stakeholder needs.

A research grid can help with this. At the end of Discovery, you can check whether you are ready to progress to Alpha by referring to your research grid and making sure if you have completed all necessary activities and collected the insights you need. Learn more about research grids.

Alpha phase

The Alpha phase is where you start building and testing working versions, or “prototypes” of your product or service, such as your questionnaire or respondent materials. You will use the insights gathered from the Discovery phase to inform the design. The prototypes will be iteratively tested in rounds of user research. You should test, refine, retest.

Once approved, the prototypes will be passed on to the next phase of delivery, “Beta”. If you need more insights to progress, you can return to the Discovery phase.

You should test prototypes for every product involved in your project. This is important because each product will form part of the respondent’s journey and overall experience. You must check for usability and comprehension when you are testing your products.

The Alpha phase is an opportunity to be innovative and radical with your design. You may choose to design your materials in a new or different way. But it is important to remember that all design choices should be:

based upon insights gathered during Discovery
based upon a solid understanding of respondent needs
intended to address a specific respondent need

You may consider organising an ideation session to generate ideas about solutions to address respondent needs. The aim of an ideation process is to generate a large quantity of ideas that can be refined to help you achieve the best design. Learn more about organising and conducting ideation sessions.

If you have already conducted your Discovery, you do not need to wait until a design is perfect before testing it. Instead, prototyping and testing early in Alpha allows you to gather feedback and iterate and redesign based upon this feedback. This is called “failing fast”. You then retest your changes to validate their effectiveness. This approach is more efficient as it also helps you to learn quickly and create a high-quality product, which is validated by respondent feedback.

Objectives of the Alpha phase

Why this objective is important

Inclusion is an important part of survey design. Designing inclusively allows you to gather richer data and make respondents feel valued. It ensures that your questionnaire works for everyone.

You should also design surveys with accessibility in mind. This involves making it as usable for as many people of possible.

If you design without inclusivity and accessibility in mind, you risk excluding certain people or groups from your survey. Ensure you understand inclusivity and accessibility when designing products and materials. You should then validate that the products and materials you have designed are inclusive and accessible through research.

Learn more about inclusivity and accessibility.

Suggested research methods or activities

We recommend considering the following research methods and activities to measure inclusivity and accessibility:

cognitive interviewing
usability testing
cogability testing
unmoderated testing
focus groups

There are tools you can use to test the accessibility of your products. Learn more about accessibility testing.

Why this objective is important

In Alpha, you will validate your understanding of:

your respondents and their needs
respondent’s mental models
respondent stories and journeys

This to ensure that needs you have identified in Discovery are still present in the survey environment.

By testing prototypes in Alpha you will be able to validate your initial insights. This may even result in you uncovering further insights. Do not be afraid to return to Discovery to uncover further information about needs, if necessary.

Suggested research methods or activities

We recommend considering the following research methods and activities to validate your understanding of insights gained in Discovery:

usability testing
cognitive interviewing
cogability testing
unmoderated testing
focus groups
feasibility testing

Why this objective is important

It is important you identify whether respondents can understand your questions and materials as you intend them to. For example, is the instruction letter clear and easy to understand? Do respondents understand the terminology used in the survey questions?

Without checking this, you could be designing a questionnaire or invitation letter that is hard for respondents to use or asking questions that do not get the intended response. This poses a significant risk to data quality.

These insights are vital to ensure you design respondent-centred products before testing at scale in Beta.

Suggested research methods or activities

We recommend considering the following research methods and activities to learn whether respondents understand your products as intended:

cognitive interviewing
cogability testing
unmoderated testing
focus groups

Why this objective is important

These insights are important to ensure you are not designing a questionnaire or invitation letter that is burdensome or difficult for respondents to follow. If your survey is burdensome survey users may drop out, or choose not to respond to some questions

By validating that your survey is easy and intuitive for respondents to use, you can gain assurance about the burden of your survey. Similarly, if a respondent can successfully follow the instructions in the letter to log into the online survey, then you know the instructions are fit for purpose.

Understanding how easy or difficult a product is to use in Alpha, you will be able to identify and fix problems with a smaller sample, before you later test at scale and with higher cost in the Beta phase.

Suggested research methods or activities

We recommend considering the following research methods and activities to learn whether respondents can use your products as intended:

usability testing
cogability testing
unmoderated testing
focus groups

Beta phase

The Beta phase is where the new and or approved version of your survey products, including your questionnaire and respondent materials, will be tested at scale to a closed sample or an open sample. A closed sample is called a “private beta” and an open sample is called a “public beta”. If satisfied with your findings, you can progress to Live. If you find you need further insights, you can return to the Alpha or Discovery phase.

When testing your survey in Beta it is important you try, as far as possible, to use a sample that the intended sample design for the end-product in the Live phase. This is to ensure the data and insights you gather during Beta are representative of your target population. You should also ensure the test sample is large enough to ensure you can detect statistically significant differences when analysing the quantitative data gathered during Beta.

Objectives of the Beta phase

Why this objective is important

It is essential to ensure that the workflow of your survey is operating correctly and that there are no blockers or manual interventions required.

Any problems with the operation of a survey can result in a poor user experience for respondents. This may result in:

missing data or reductions in data quality
attrition and reduced response rates if respondents decide to drop out of a survey midway through, or drop out of a longitudinal survey between waves because of a poor experience
survey satisfaction among respondents, and a reluctance to participate in future surveys

By ensuring the operational process is working as planned you will be able to identify and fix any problems. It is essential to fix these issues before the survey enters the Live phase. If you do not, you expose your survey to the risks mentioned above.

If you identify major operational problems, you should delay progressing the project to the Live phase. You should extend the Beta phase to address these issues.

Suggested research methods or activities

We recommend considering the following research methods and activities to learn whether operational process is functioning as planned:

dress rehearsal
pilot survey

Why this objective is important

It is important to conduct your testing at scale to gather statistical data from your prototypes.

This data should be analysed through statistical significance testing to make comparisons between groups, and descriptive statistics should be produced, to provide insights about the effectiveness of the survey prototype.

Analysis of the data you obtain will provide findings about:

the effectiveness of your prototype survey
whether it will succeed in capturing the desired data of stakeholders when it is made live

This is an important requirement to validate that the survey is fit for purpose and will successfully capture intended data. This work should be undertaken before the survey progresses into the Live phase. If you discover that any part of your survey is not fit for purpose, you can return to Discovery and Alpha to conduct research to determine why this is the case, and to re-design any such questions.

Suggested research methods or activities

We recommend considering the following research methods and activities to gather statistical survey data insights:

dress rehearsal
pilot survey

Why this objective is important

The Beta phase can be used to statistically measure the effectiveness of elements of your survey design. This could include the effectiveness of your chosen respondent incentive, or how and when you send reminders to respondents. In a field survey, it might be how many visits an interviewer makes to the household before interview, or the effectiveness of any responsive or adaptive designs.

Beta tests can also be used to conduct split sample experiments to establish the most effective survey designs.

Suggested research methods or activities

We recommend considering the following research methods and activities to test the effectiveness of your statistical survey design:

dress rehearsal
pilot survey
A/B testing, or split sample experiment

Why this objective is important

Paradata refers to data collected about the survey process, rather than data obtained from item response. In other words, it’s data about your data. Examples of paradata include:

the overall response rate to the survey, and response rates for sub-groups of the population, such as different regions
how long a single question takes a respondent to complete
how long a household takes to complete the whole survey
any error messages that respondents encounter during the survey, if the survey is online
whether respondents have “gone back” to any previous question and changed their answers, if the survey is online
what browser or device type a respondent has used to complete the survey, if the survey is online
various field metrics for interviewer led surveys, such as time and day of interview, number of calls before a contact, number of calls before an interview, time with the household, any interviewer observations collected from open or closed questions.

Gathering and analysing paradata will provide you with useful insights about the performance of your survey. This is important to gather during Beta and before your survey is live. This will give you confidence in the performance of your survey, or the opportunity to fix problems before the survey goes live.

Paradata is important to:

assess survey performance — for example, to assess how much longer it would take people to complete your survey if you added a new question
identify and diagnose problems — for example, it could help you understand whether respondents are having difficulties accessing an online survey on a particular device or browser
evaluate survey “pain points” or difficulties — for example, it could help you see whether there are patterns of several respondents encountering the same error message for an online survey, or whether many people are changing their answer to a particular question

Based on these insights, you might wish to return to Alpha or Discovery to conduct further research or test alternative prototypes. This will give you the opportunity to address these issues before re-testing an iterated prototype at Beta again.

Suggested research methods or activities

We recommend considering the following research methods and activities to gather statistical paradata insights about your data:

dress rehearsal
pilot survey

Why this objective is important

Once the data collection period has ended and you have gathered your survey data, you will need to process the data.

Examples of data processing methods include:

data cleaning
data aggregation
coding of open-ended responses from open questions, or “other” categories at closed questions
disclosure control methods
data weighting
data imputation
the process for producing derived variables

The Beta phase allows you to test these methods to ensure they’re suitable for the Live phase. You can use data obtained from your final survey prototypes to test these methods and make changes where necessary before your survey goes live.

Suggested research methods or activities

We recommend considering the following research methods and activities to test the effectiveness of data processing methods:

dress rehearsal
pilot survey

Why this objective is important

For population studies, a weighting and imputation strategy should be developed as part of the Beta phase. Weighting and imputation are two key components of the survey process.

For sample surveys, weighting is applied to convert the survey responses into an estimate of the desired population characteristic. The weights your survey use will likely need to be modified in order to ensure accuracy. Modifications should be based on how representative the random sample of your survey is of the true population.

Weighting can also be used to help account for non-response. But, when only a subset of variables are affected by non-response, imputation is often used to help account for this.

Imputation is a method used to estimate individual values for sampled units when they are missing or unusable. Imputation methods generally rely on either:

relationships between the non-responder and similar responding units
information available about the non-responder, which may be available from other or past sources

There are multiple different methods of imputation available. The best method to use will vary based on the context of your situation.

As there are multiple different methods available for weighting and imputation, it is important that you select the method that is most appropriate. Choosing the wrong method can have severe consequences on the quality of the output you produce. For this reason, it is highly recommended to seek expert advice about the best methods to use for your specific situation. It is also highly recommended to allow sufficient time to develop strategies for weighting and imputation.

Suggested research methods or activities

We recommend considering commissioning expert research to develop appropriate weighting and imputation methods.

Why this objective is important

You should have a plan of how you are going to evaluate the Beta phase. Ideally this should include a set of objectives or quality targets which need to be met before you proceed with the Live phase.

Your targets should be clear and measurable. When they are successfully achieved, they should provide confidence and assurance that your survey will meet requirements. It is beneficial to decide in advance which targets need to be met during the Beta phase to approve the survey so it can progress to the Live phase.

If you have not created an evaluation plan you will find it difficult to determine if the survey is fit for purpose. This would mean:

your work in the Beta phase would not provide assurance that you are ready to go Live
you risk going to Live with a flawed survey

If you fail to meet your pre-established objectives in the Beta phase, you should return to Discovery or Alpha to conduct further research regarding issues. You should then redesign a refined prototype based on further insights. The whole survey should then be retested at Beta before progressing to Live.

Suggested research methods or activities

There is no single research method that can be used to evaluate your project. Instead, you should evaluate the findings from all Beta activities you undertake to determine whether your survey is ready to progress to Live.

As part of your evaluation, you should compare your Beta results to pre-established quality targets. If applicable, your evaluation should include:

a debrief with interviewers
a debrief with enquiry telephone line operatives

This will help you fully explore their insights into the Beta.

Live phase

When your product or service is Live, you must ensure it is supported in a sustainable way. The Live phase is about continuing to iterate the product or service and making improvements. Continuous improvements will ensure the product or service continues to meet the users’ needs and is fit for purpose.

In this phase, you will be evaluating the survey and ensuring it is continuing to meet user needs. You should be aware of any factors, including things like societal changes, which may mean that products will need to be updated. To identify whether your product is fit for purpose or needs improving you will use:

the insights gathered from quantitative analysis
feedback from respondents
feedback from interviewers, if applicable

You will not need to make changes if the analysis shows that the products are producing successful results and meeting user needs. But you should continue to evaluate the success of your product.

If you find your survey no longer meets your users’ needs you may need to return to previous phases, such as Alpha or Discovery to identify why this is and what improvements need to be made, if any. For example, the needs of your users may have changed since the survey was launched, so you would return to Discovery to identify the current needs.

It may be that your survey is no longer needed altogether, or that it is not cost-effective to keep running. In this case, it would make sense to retire the survey, which means shutting it down. You will still need to consider your user needs here, and how these will be met after its retirement. Learn more about retiring your survey.

Objectives of the Live phase

Why this objective is important

It is important to continue to monitor your questionnaire to ensure that respondents experience as little burden as possible, and that your end-product continues to meet user needs over its life cycle.

For example, you may look at the comments in “other, please specify” response options to see if a lot of respondents are providing the same write-in response. If they are, this may indicate that you need to add a new response option. But you should also consider the implications of adding an additional response option before doing so. For example, is there a data user need for adding a particular response option?

You may also be able to get feedback directly from respondents if you have allowed them the opportunity to provide feedback at the end of the survey.

Suggested research methods or activities

We recommend considering the following research methods and activities to identify where you may need to amend questions or response options:

analysis of survey feedback
analysis of write-in responses
stakeholder engagement

Why this objective is important

It is important to conduct analysis of response rates to ensure you are obtaining a representative responding sample. This may include:

analysis of the different characteristics of your respondents
a comparison to the characteristics of your target population

It is also important to analyse response patterns to determine:

when respondents complete the survey
when respondents drop out of the survey
if reminder letters or emails are needed

With the correct paradata setup you may be able to monitor daily returns, which allows you to calculate:

return rates
refusals
non-contacts
ineligibles
survey attrition

Paradata can also be used to identify peaks in response, and when response plateaus. This will allow you to determine the levels of success for:

reminder letters, emails, or texts
intervention activity in the field

Suggested research methods or activities

We recommend analysing survey paradata to understand more about your response rates.

Learn more about paradata.

Why this objective is important

You should aim to ensure that your achieved sample is a close approximation to the composition of your target population. For social surveys, analysis of socio-demographics such as age, sex, ethnicity may provide the insight required to determine this.

If your achieved sample does not approximate to your target sample, then biases are likely to occur.

You will not have the equivalent data for non-responders, but you may be able to gather some insight from non-survey data sources which match some characteristics from other data sources.

Suggested research methods or activities

We recommend considering the following research methods and activities to monitor the characteristics of responding and non-responding households and individuals:

analysis of survey data
analysis of non-survey data sources

Why this objective is important

“Pain points” refer to a specific part in a respondent’s journey where they encounter an issue when trying to complete the survey. If your questionnaire is administered in an interviewer-led mode, you can gather feedback from interviewers to provide useful insights about potential issues within your survey.

For example, you could review interviewer notes regarding:

particular questions
visits or calls to households

Identifying patterns in these notes can pinpoint problems encountered.

This will only be possible for surveys that are interviewer-led.

Conducting debriefings with interviewers can provide more qualitative insight into how the operation of your survey is performing. Try to ensure the sample of interviewers you debrief is representative of your target population as there may be different issues encountered in different areas. For example, this may involve conducting debriefs with interviewers from high or low performing areas. This may help you identify:

where issues occur in your survey
why issues are occurring with your survey

Suggested research methods or activities

We recommend considering the following research methods and activities to gather interviewer feedback on any pain points in your survey:

review interviewer notes
conduct interviewer debriefings
conduct regular review meetings

Why this objective is important

By analysing survey data you will be able to identify frequencies and proportions of “don’t know”, “prefer not to say” or skipped answers. This will help to identify gaps in your data which may:

influence your outputs
indicate any potential problems with questionnaire routing
indicate problems with questions or answer options

You should review questions with high rates of missing data to identify why there are gaps in the data. Qualitative research methods may be used to provide more context about this.

Suggested research methods or activities

We recommend analysing your survey data to identify where data may be missing.

Why this objective is important

Paradata can allow you to monitor the performance of your survey operation and make adjustments where needed. For example, this could include analysis of: ·

how many interviewer visits are needed to make contact with a household
how much response increases when reminder letters arrive
the proportion of the sample that call in for help with the survey and what times these calls occur

These measures can all be used to improve your operational processes, but you must consider that you will need enough data gathered over a reasonable time frame for a robust analysis.

Remember there will be seasonal differences because of public holidays and other events, such as school term times. This should also be factored into any analysis or possible adjustments.

As always, we would recommend testing any adjustments before implementing them at scale.

Suggested research methods or activities

We recommend analysing your paradata to see how you can improve your operational processes.

Learn more about paradata.

Why this objective is important

For population studies, a weighting and imputation strategy should ideally have been developed in the Beta phase, and then subsequently implemented at Live.

Weighting will involve calibrating the survey data to known population totals. For social surveys, this may be based on characteristics such as, but not limited to, the age and sex of respondents.

At the Live phase, you will have sufficient volumes of data to produce estimates that have been weighted to represent your target population. But it is possible the weighting process could change, and more or fewer variables are used in the methodology to improve the accuracy of the resulting outputs. This would also change the final estimates of your data. Any changes should be considered carefully, as they can produce substantial changes to population estimates. Changes should therefore not occur frequently.

If you do make changes to the weighting process, you should consider revising previous estimates using the new methodology.

Similar considerations should be taken for imputation. Ideally, imputation methods will stay consistent, and the variables used to impute will not change. But there are scenarios where changes may be required. For example:

If changes are made to questions that are used for imputation
questions are added or removed, from which imputation would improve accuracy

When considering imputation methods, you should consider:

the frequency of making changes
the potential need to revise back series data
how any changes will be communicated to data users

Suggested research methods or activities

We recommend considering the following research methods and activities to review weighting and imputation implementation:

analysis of survey data
analysis of non-survey data sources

Why this objective is important

You will be producing data tables and commentary based on the survey specification developed at the start of the survey. You may also be making datasets from the survey accessible to data users.

You should consider the audience for the outputs and how the data will be published. For example, you may:

have multiple audiences who need different publication styles — some people may need a more technical publication and others may need a more explanatory publication
be presenting the data at conferences
publishing on a website

All these methods of sharing data will require different formats

The datasets to reduce the risk of the identification of households or individuals to an appropriate level.

Multiple datasets with different levels of disclosure control may be required. For example, approved researchers could be allowed to access more sensitive or disaggregated data with approval, whereas publicly available data would need to be anonymised to have negligible disclosure risk.

Learn more about disclosure control.

Suggested research methods or activities

We recommend considering the following research methods and activities when sharing data outputs and producing plans for your publications and data dissemination:

produce draft publications
follow disclosure control principles

Why this objective is important

User guides determine how the design of the survey, the data structure, and information on the data collection operation are communicated to users. These should provide all the information a user will need to understand:

the survey
how to use the survey data
any limitations that survey data has

You should provide details about things like the design, methodology, variable guides and sample composition to provide a comprehensive and understandable set of guidance.

User guides will need to be updated over time for continuing surveys as aspects of the survey change. This could include:

adding new questions
differences in derivation of variables
changes to definitions
changes to collection processes

You should update user guides to reflect when the changes to the survey occurred.

Previous versions of guidance that relate to previous instances of the survey should be kept and labelled to demonstrate which periods they cover. This will allow users to see a complete history of the survey and to see when, where, and why changes occurred.

Suggested research methods or activities

We recommend considering the following research methods and activities when producing user guidance:

develop draft user guidance
review draft user guidance

Why this objective is important

It is important to test survey performance to determine whether your survey is meeting quality targets. If it isn’t, you will need to identify potential improvements to improve the quality of your survey.

By analysing your survey data and paradata, you will be able to see if you are meeting your agreed quality targets. For example:

is your response rate sufficient to provide robust estimates?
is your achieved sample representative of your target sample?
is there sufficient data in your dataset for important outputs or are there gaps in the data?

By analysing these areas, you will be able to test how your survey is performing against your targets. If it does not meet requirements, you will be able to identify improvements which will likely need to be tested before they are implemented

You should conduct separate testing on potential improvements before implementing them within your live survey.

Suggested research methods or activities

We recommend considering the following research methods and activities to ensure your survey is meeting agreed quality targets:

analysis of survey data
analysis of paradata

Research methods and activities associated with each Agile phase

We have mentioned a variety of research methods in this guidance. These are described further in this section.

We have provided information on:

what each research method is
the insights you gather from each research method
when the research method may be appropriate to use, with reference to Agile phases
when the research method may not be appropriate to use, with reference to Agile phases

You can use this information to decide what research methods would be appropriate for you to use in your project to meet the objectives of each Agile phase.

This guidance does not provide details of how to conduct each research method. It is important to ensure you have an awareness of how to conduct research with a particular method before using it in your project. We have provided links to resources throughout this section where you can learn more about each research method.

It is important for anyone conducting research to be appropriately trained in the specific method they are using. Training will equip you with the tools needed to handle different situations, be aware of your own biases, and facilitate discussions that yield insightful information. Having trained researchers will increase the likelihood of data of good quality being collected, in addition to creating a better experience for the participant. For example, trained researchers are more likely to know techniques to build rapport with participants and establish a comfortable environment, as well as appropriately asking follow-up questions. Trained researchers are more likely to be able to get participants to engage with you, so you’re more likely to get the information you need from them.

In addition to completing training, experience of conducting research methods in a working environment is an essential part of learning. To build capability and expertise, you will need to gain additional experience by observing and conducting research yourself.

In-depth interviews are typically conducted on a one-to-one basis with participants. This allows the researcher to establish rapport with the respondent and explore topics in detail.

During the session, the interviewer will ask the participant questions from a pre-prepared topic guide. The topic guide provides an outline of key areas of questioning. The interviewer may follow-up with probing questions to gather further insights from the participant.

This method provides the interviewer the opportunity to cover specific topics that are of significance to the participant in detail. This means this method is useful for learning in-depth information about participants and their perspectives.

The participant may also bring up topics that were not initially considered to be of interest by the interviewer. The flexible nature of interviews (if using a structured or semi-structured approach) allows the interviewer to dig deeper into these insights which could enrich the research.

The participant may also talk about topics that were not initially considered to be of interest by the interviewer. There is a lot of flexibility if you are using a semi-structured approach for your interviews, which allows interviewers to explore topics in more depth. This can enrich the research.

Learn more about in-depth interviews.

Insights gathered from the method or activity

You can use insights gathered from in-depth interviews to gather detailed information from people about their:

opinions
thoughts
expectations
motives
interpretations
experiences

The insights can also be used to gain a deeper understanding of mental models, respondent needs, or insights to inform user stories or the next phase of your research. For example, you may use an in-depth interview to understand the language participants use to talk about a certain topic. The insights gathered during the interview could then be used to inform the wording of a questionnaire or invitation letter.

In-depth interviews can also provide context to quantitative data. For example, quantitative data might show that a survey’s response rates are dropping or a high amount of missing data for a particular question. This data would show what was happening, but it would not explain why. You can use in-depth interviews to help explain the reasons for the declining response rates and missing data.

When the method or activity may be appropriate to use

To explore something in more depth in a one-to-one setting

The one-to-one setting of an in-depth interview allows the interviewer to establish greater rapport with respondents, which may allow the participant to feel more comfortable sharing information. The one-to-one setting of an in-depth interview also allows the researcher to ask follow-up questions to gather richer insights, and ensure responses are not affected by other research participants. An example of where this may be useful is when researching topics that are sensitive or of a personal nature.

To be able to adapt your questioning approach to yield relevant insights

If you are at the start of the development process, you may not know much about the research topic yet. By taking a flexible approach, you can get more detailed answers from the respondent and may gather relevant information that you had not considered when planning the research. Taking a flexible approach will allow you to gather the data you need and allow you to explore unexpected or emerging topics.

To make complex research topics clearer for participants

An interviewer will be able to explain the topic to a participant in more depth. The participant will also be able to ask any relevant questions to help them understand the questions and the topic. The one-to-one setting of in-depth interviews helps to facilitate this.

When you have sufficient time, money, and researcher capability to conduct this research

Although in-depth interviews gather rich insights, they cost more money and take longer to conduct than other research methods because they are longer and only gather insights from one participant at a time.

When you are in the Discovery phase of survey development

In-depth interviews can be used to gather insights to inform question development or the design of respondent materials. This should take place in the Discovery phase.

You may wish to conduct follow-up in-depth interviews following a research activity in a different Agile phase. For example, your cogability testing in your Alpha phase might show that respondents are having difficulty understanding your invitation letter prototype. This may mean you decide to undertake in-depth interviews to explore respondents’ mental models of this topic in greater detail. Although this would be undertaken following an Alpha activity, it would technically be considered a return to Discovery, as you are gathering insights to inform development again.

When the method or activity may not be appropriate to use

When you want to generalise findings from your research

You cannot generalise results if you have small sample sizes. Equally, you cannot generalise results because samples are not random and are subject to bias.

When the research would benefit from group conversations

If your research would benefit from group conversations, you should consider using an alternate qualitative research method, such as a focus group.

When you have limited time, money, or researcher capability

In-depth interviews are by nature expensive, time-consuming, and require significant researcher capability.

When you are in the Alpha, Beta, or Live phases of survey development

In-depth interviews are typically used in the Discovery phase to gather respondent insights that inform the subsequent design of respondent material or questions. This should be done before developing materials in Alpha. You should return to the Discovery phase if issues arise during Alpha, Beta, or Live phases. You can then conduct further research about respondents to learn more about these issues and how to fix them.

Focus groups are a form of group interview led by a moderator. Participants are put into groups of similar characteristics that are of interest to the topic being researched. This does not mean that they should have similar attitudes, demographics, or experiences in general. Only the characteristics of interest to the research should be similar.

Focus groups are typically longer in length than in-depth interviews as you need to consider the group interactions and discussions.

There is a less directive style of interviewing from the moderator than with in-depth interviews. This is because the moderator will encourage a variety of views about a topic and will let participants discuss them as a group. It may also be possible to include interactive exercises within the focus group, where participants are given a task and asked to share what they have done with the rest of the group. This would then lead to a group discussion.

Learn more about focus groups.

Learn more about conducting activities in focus groups.

Insights gathered from the method or activity

Like in-depth interviews, focus groups can be used to gather detailed information from individuals about their:

opinions
thoughts
expectations
motives
interpretations
experiences

They can also be used to gain a deeper understanding of mental models, respondent needs, or insights to inform user stories or the next phase of your research. Focus groups can provide additional context to quantitative data.

You can gather insights about respondent needs using focus groups. For example, in the context of developing respondent materials, you could conduct a focus group to determine what information a respondent needs to be able to participate in a survey.

You can also use focus groups to assess how well respondents understand your product. For example, you could use a focus group to evaluate whether respondents can understand prototyped respondent materials, like invitation letters, as you had intended.

When the method or activity may be appropriate to use

When the research would benefit from group conversations

Examples of situations that may be of benefit to your research and would require a group dynamic include:

group discussions
challenging opinions
sharing ideas

When you need to gather information from groups defined by a certain characteristic

Focus groups are useful when you want to gather lots of insights from a particular group defined by a certain characteristic. For example, you might need to gather insights from people with a similar health condition, or a similar job role.

Focus groups will allow you to gather various ideas, opinions, or views of different members of a particular group in one research sitting.

When you want to generate ideas for producing design solutions

Because of the group nature of focus groups, you will likely gather insights from people with different perspectives in each session. This may be useful for identifying issues with a design prototype, or for generating ideas about a design solution.

You may consider conducting an ideation session during a focus group setting. Learn more about ideation sessions.

When you do not know much about the research topic and want to gather insights and further information about it

If you are at the start of the development process, you may not know much about the research topic yet. Conducting focus groups with multiple participants present will allow you to gather lots of information from a variety of different perspectives. This may highlight information you hadn’t considered when planning your research and can help inform subsequent design and development.

When you have limited time or money

Focus groups can be effective when a lack of resource prevents you from doing extensive in-depth interviews. Focus groups are cheaper and less time-consuming to run than in-depth interviews, as they allow you to gather insights from multiple participants in one research activity.

When you are in the Discovery phase of survey development and want to gather insights about respondent mental models and needs

In Discovery, the purpose of a focus group would typically be to gather insights about respondent mental models and respondent needs. These insights would be used to inform design in the Alpha phase. This applies to both question design and respondent materials development.

When you are in the Alpha phase of survey development and want to gather insights about respondent materials prototypes

In Alpha, the purpose of a focus group would typically be to gather insights about respondent material prototypes and evaluate them. For example, you could use a focus group to:

gather respondent feedback about a survey invitation letter
conduct activities to validate respondent needs and enhance your prototype

When the method or activity may not be appropriate to use

Due to the group nature of focus groups, you should not use them when:

the research is sensitive or of a personal nature
there are concerns regarding anonymity
there is a fixed ordering of questions

When you have constraints that make focus groups unfeasible

There are more practical implications to consider when organising focus groups, such as the location of the research. You will need to consider the feasibility when planning your research and recruiting participants.

When you are in the Beta or Live phases of survey development

Focus groups are typically used to gather respondent insights that inform the design of respondent materials or questions. This should be completed before you reach Beta or Live.

Pop-up research consists of short, informal interviews or usability testing. It happens in the places used by the people you are interested in speaking to.

Usability testing would involve taking the product or service with you and observing how participants use it. Short interviews would involve asking participants a set of questions whilst being in their own environment.

Pop-up research could take place in a range of different places, such as:

in a café
on a high street
on a university campus
in shops

It can take place anywhere that is used by the people you’re aiming to speak to. Unlike other methods, this means you will not recruit participants before the event. Instead, you will recruit them during the event.

You can mitigate potential bias by creating a sampling matrix. You can use it to:

help you consider the range of characteristics included in your sample
inform who you conduct research with whilst conducting pop-up research

Pop-up research is useful for gathering early insights, but you should always use it with another research method if you want to gather more representative data.

Pop-up research is a quick and low-cost method of conducting research and works best when you have clear and simple objectives. You should always do pop-up research in pairs, with another interviewer present. It makes the activity both safer and easier.

Learn more about pop-up research.

Insights gathered from the method or activity

Pop-up research can be a useful method to gather insights when you are exploring different problems and opportunities in the early stages of your research. It can increase your understanding for when you design more in-depth research.

You can use pop-up research to gather respondent needs. For example, you could use it to find out what information respondents need to have before they decide to take part in a particular survey. This can help inform further research into design choices and information provided to respondents.

Pop-up research can be conducted to test the look and feel of prototypes. For example:

do they align with respondents’ mental models?
are they attractive and usable?

You can gather insights about an existing design and the:

user experience – for example, you could gather information about how easy or difficult the product is to use or navigate, or how easy it is to login to the survey
current problems or “pain points” encountered by the user

Pop-up research is short and informal. This means it is often best to avoid testing whole questionnaires or invitation letters using this research method. You should test specific aspects of a question pattern or a small feature, like a progress button, instead.

When the method or activity may be appropriate to use

When you want to gather insights in a short timeframe

Pop-up research can be conducted in a short timeframe as you gather lots of insights in rapid succession.

You can conduct pop-up testing quickly at the start of your survey design, to gather quick insights to help inform areas for future research and provide an early understanding of respondent needs and mental models.

When you want to conduct mass testing with little expense

Pop-up research allows you to gather insights from large numbers of people without spending a lot of money.

When you want to gather insights on “hard to reach” populations

Pop-up research can be used to gather insights from “hard to reach” populations. This refers to people who you might typically struggle to engage with for research purposes, such as care home residents. If you conduct your pop-up research in a location where your “hard to reach” population are likely to be, you are likely to be able to gather insights from them.

When you want to research regional variations

As pop-up research takes place in a physical location, and uses opportunity sampling to recruit participants, your findings will only represent the area where your research takes place.

If you repeat your pop-up research in multiple locations, you can gather insights about regional variations.

When you want to include short participatory activities in your research

Pop-up research is useful if you want to include short, practical tasks within your research. For example, this could include finding what information a participant finds useful in an invitation letter, or capturing their initial reactions to an envelope design.

Due to the voluntary nature of pop-up testing, you should ensure any participatory activities are short and simple.

When you have a limited budget

Financial incentives are not typically needed to gather data because this method uses opportunity sampling

When you are in the Discovery phase of survey development

Pop-up research can be used to gather early insights to help inform future research topics. This should take place in the Discovery phase.

When you are in the Alpha phase of survey development and want to gather early insights about a developed prototype

In Alpha, you may choose to conduct pop-up testing to gather early insights about a developed prototype. But you should always follow this research up with a more in-depth method to test the prototype.

When the method or activity may not be appropriate to use

When you want to conduct research about a sensitive topic

Due to the public nature of pop-up research and the use of opportunity sampling, it may not be appropriate to conduct research regarding a sensitive topic.

When you want to conduct in-depth research

Due to the voluntary nature of pop-up research, it is best suited for quick research sessions that are not in-depth.

You will need to consider the limitations of the tasks you can do during pop-up research. For example, it may not be an appropriate method for taking people through a questionnaire and gaining insights on how easy or difficult they found it to complete.

When you want to research a complex topic

Pop-up research is best suited for short and simple research exercises. If you are researching complex topics that need a lot of time or context to gather insights, you should consider a different research method.

When you are not planning to conduct further in-depth research to accompany it

Pop-up research should not be the only method used in your research. There are many limitations to pop-up research, such as short sessions and unexpected challenges. This means it should always be used in combination with an alternative research method.

When you have sufficient time, money, and research capability to conduct more in-depth research

If you have a large budget, time, and researcher capability, you may choose not to conduct pop-up research. This is because you would have resource to undertake more in-depth research with a wider sample of respondents, such as in-depth interviews or focus groups.

But pop-up research still has value even if you have the resources to conduct more expensive types of research. It could be used to gather some quick early insights to inform later research.

When you are in the Beta or Live phase of survey development

Pop-up research is typically used to gather early respondent insights that inform the design of more in-depth research. This should be completed before you reach Beta or Live.

Pop-up research should not be used as a standalone method to test materials or questions in Alpha. It can be useful to gather early insights, but more in-depth research should be conducted to test prototypes.

Card sorting is a user research technique designed to gather insights to improve the usability of your product.

This research method involves presenting a series of topics to participants.

Participants are then asked to organise these topics, or “sort these cards”, into groups that make sense to them.

These insights will show how participants interpret different topics and how they relate to each other. You can use these insights to design your product in a user-centred manner.

For example, a web developer may use these insights to determine how a website’s navigation panel should be formatted. If participants do not consider two of the most important topics to be similar, the developer may choose to present them separately in a navigation panel.

A survey developer may use card sorting to ensure that questions that participants consider to be similar are presented in the same section of the survey or that answer options are listed appropriately. This may help improve the flow of the survey and the respondent’s experience, reducing burden.

There are many variations of card-sorting, such as:

open card sorting, and closed card sorting
moderated card sorting, and unmoderated card sorting
paper card sorting, and digital card sorting

Learn more about card sorting.

Insights gathered from the method or activity

Card sorting gives you insights on how your users think about the content, and their expectations and understanding of topics. It allows you to identify how people understand and categorise information.

When the method or activity may be appropriate to use

When you want to plan the ordering of your survey’s questionnaire

Card sorting can be useful to help you plan the order of survey items when you are developing a new questionnaire. Card sorting helps you work out what topics participants consider to be similar.

When you want to plan the structure of your respondent materials

Card sorting can be useful to help you plan the ordering and structure of your respondent materials.

When you want to conduct mass testing at low cost

If you run a card sorting activity remotely, you can gather lots of data quickly at a low cost. This is useful when you have limited time or budgets. You can also get a good geographical spread of participants.

When you are evaluating an existing survey

When evaluating an existing survey, card sorting may be useful to help you understand whether the existing question topic ordering is appropriate, or whether questions should be reorganised.

When you are in the Discovery phase of survey development

Card sorting can be used to inform the ordering of questions in your questionnaire, or the structure of your respondent materials. This should take place in the Discovery phase.

When the method or activity may not be appropriate to use

When you are using it as your only method of assessing usability

Card sorting cannot be used as a replacement for usability testing.

This is especially important as card sorting findings do not necessarily translate from one context to another. Just because participants consider topics to be related in a card sort, does not necessarily indicate that the same is true within the context of a survey questionnaire. This is why you still need to do usability testing.

When you have too many topics

Card sorting is less effective when you have large numbers of topics. This is because it can become burdensome to the participant if you ask them to sort too many topics. This can also make analysis complex.

When you are in the Alpha, Beta, or Live phase of survey development

Card sorting can be used to inform the ordering of questions in your questionnaire, or the structure of your respondent materials. This should be completed before the Alpha, Beta, or Live phase.

Co-design is rooted in participatory and user-centred design.

The aim of co-design is to involve stakeholders, such as data users or respondents, in early phases of the design process. It concentrates on designing with people.

During co-design, you will work with a collection of different stakeholders to discuss stakeholder needs and conduct exercises to produce design prototypes.

But the aim of co-design is not to produce final designs at the session. Instead, the purpose and benefit of the session is to:

generate discussion among stakeholders
identify stakeholder needs
gather insights to inform subsequent design

Learn more about co-design workshops.

Insights gathered from the method or activity

Co-design will help demonstrate stakeholder needs. It may also help you understand the prioritisation of stakeholder needs, if there are any conflicts between them.

You should manage your expectation from any co-design workshops. It is unlikely that any prototypes developed during the session will be suitable to be included in your final design. But it will provide insights regarding stakeholder needs, and serve as an ideation session for designs you work on after the session.

When the method or activity may be appropriate to use

When you want to generate insights about stakeholder needs and how to prioritise them

Engaging with a variety of stakeholders during a co-design workshop will allow you to:

generate insights about stakeholder needs
understand which stakeholders have which needs — this may help you to prioritise certain needs when designing your products later on

To generate ideas about how you could design a question

It is unlikely that designs developed in your co-design workshop will meet respondent needs and be fit for purpose. But they will generate ideas of potential designs you can consider in the future, after you have gathered respondent needs.

When you are in the Discovery phase of survey development

Co-design workshops can be used to generate insights about stakeholder needs. This should take place in the Discovery phase.

When the method or activity may not be appropriate to use

Co-design workshops should not be used instead of gathering respondent insights and needs

Co-design should not be used to produce finalised question prototypes in the survey development process. Your prototypes should be designed based upon respondent insights that you gather.

Feasibility testing is a smaller version of a full-scale pilot study. The main purpose of feasibility testing is to do some early work to understand respondent comprehension of important survey concepts.

This research method can be used to evaluate whether your research is both:

feasible, meaning that it can practically be carried out
acceptable, meaning that participants find the research appropriate

Insights gathered from the method or activity

Feasibility testing can provide insights into various aspects of the survey process. For example, this could include aspects such as:

sampling
collection modes
operational testing
developing evaluation methods

Feasibility tests tend to be small scale. If you are looking to test multiple aspects of a survey, then a pilot study may be more appropriate.

When the method or activity may be appropriate to use

When you are developing a survey on new concepts that have not been measured before

Feasibility testing is important to conduct if you are attempting to measure new concepts that have not been measured before. This is to understand whether it is possible and acceptable to conduct research about these topics.

When you have sufficient time and resource – but feasibility testing does not typically cost a lot of money

A feasibility test does not typically need a large sample size, so it does not usually need a large budget. But it is important to be aware that a reasonable amount of time and resource is needed to develop the test.

When you are in the Discovery phase of survey development

Feasibility testing would typically be used at the Discovery phase if the survey concept has been developed far enough. Otherwise, it would form part of the Alpha phase. If you are doing a feasibility test, you should do this early in the development process.

When the method or activity may not be appropriate to use

When there is enough evidence in existing research literature

Feasibility testing may not be necessary if there is enough evidence in existing research literature. This pre-existing evidence will provide the insights you need to determine whether researching a particular topic is feasible and acceptable.

When you are looking to test your survey at scale

Feasibility testing is typically conducted on a small scale. This is because it is early research to determine whether your intended research is feasible and acceptable.

When you are in the Beta or Live phase of survey development

You should not use feasibility testing in the Beta or Live phase. This would be too late in the process to test the feasibility of a project.

It would risk money, time and resource being used for large scale piloting on something which may prove to not be feasible. Feasibility testing should be conducted before the Beta or Live phase.

Cognitive interviews are one-to-one sessions with a participant and a researcher. You would usually run 30 interviews in one round to enable thematic analysis.

The purpose of the interview is to evaluate how participants understand the survey questions. This will help you to learn about the quality of data.

Cognitive interviewing often uses a “think aloud” approach. This is where participants are invited to talk through their thoughts and feelings as they perform a task or use a product. This helps the researcher to understand how the respondent interprets and comprehends the task, and the thought-processes behind their decisions.

There are two different types of probing techniques that you can use to test your questions and materials and gather user feedback.

“Retrospective probing” involves asking the participant their experience in hindsight, after they have fully completed a task.

“Concurrent probing” requires the participant to complete a task whilst speaking their thoughts aloud. If they say anything of interest, the researcher can ask follow-up questions during the task. This is more useful when probing would not disrupt the natural flow of participants completing a task.

Learn more about retrospective probing and concurrent probing.

Cognitive interviewing is commonly used to pre-test questions within a survey questionnaire, to identify whether they are fit for purpose or need to be changed. In the case of a new survey, the testing would occur before the survey has gone “live”. But cognitive interviewing can also be used for a survey questionnaire that already exists, if you are considering changing a question for the next wave, or next administration of the survey.

Insights gathered from the method or activity

Cognitive interviewing allows you to study the ways in which individuals mentally process and respond to survey questions.

The research method can help you understand:

question validity, or the extent to which your questionnaire measures what it aims to measure
response error, or the return of false or subjectively modified information from survey respondents

You can use cognitive interviewing to gather insights on participants’ mental models, which includes learning about their thought process, and understanding how they arrive at their answer.

Cognitive interviewing can help you learn about the language and words participants use to describe their circumstances. You can then use that wording in your designs.

This type of interviewing can also help you gather insights on survey materials, such as ensuring the wording of invitation letters is understood.

You can use cognitive interviewing to understand changes in the data for existing surveys when undergoing transformation. For example, if interviewer-led surveys are being changed to an online self-completion mode. Cognitive interviewing could help to understand if the questions are being interpreted the same, or differently, because of this change.

When the method or activity may be appropriate to use

When you are developing new questions and materials for a survey

Cognitive interviewing can be used to validate whether participants understand survey questions and respondent materials as you intend them to. Cognitive interviewing helps you validate whether designs you have produced are fit for purpose.

When you are developing a survey for self-completion mode

Cognitive interviewing can be used when developing new questions and materials for all survey modes. But it is even more important for self-completion surveys, where there is no interviewer present to support respondents and help them understand the questions.

When you have sufficient time, money, and researcher capability to conduct this research

Cognitive interviewing costs more money and takes longer to conduct than other research methods because they are longer and only gather insights from one participant at a time. You will likely need to spend money on financial incentives for participants to encourage them to participate.

But cognitive interviewing is a necessity to validate that newly designed questions are understood as intended and will gather high quality data. You should ensure to plan appropriately to incorporate cognitive testing within your survey development timeline.

When you are in the Alpha phase of survey development

Cognitive interviewing is used to evaluate how participants understand the survey questions and materials you have developed. This should take place in the Alpha phase.

When the method or activity may not be appropriate to use

When you are using GSS Harmonised standard

Harmonised standards are survey questions that have been developed following question testing and consultation with the Government Statistical Service (GSS). Using harmonised standards in your survey helps ensure the data your survey gathers about harmonised topics is consistent and comparable to other data collected across the UK.

As they have already been tested to ensure quality, you can use harmonised standards in your survey without the need to cognitively testing them.

But if you are using a combination of newly developed questions and Harmonised standards, you should consider testing the newly developed questions. Remember to consider the effect of the order and flow if you are only selecting certain questions for testing.

Learn more about Learn more about harmonised standards.

Cognitive interviewing should not be a replacement for usability testing

Cognitive interviewing should not be used as a replacement for usability testing. The two methods aim to gather different insights.

When you have limited time, money, or researcher capability

Cognitive interviewing is by nature expensive, time-consuming, and requires significant researcher capability. You should consider timelines and budgets when deciding whether cognitive interviewing is appropriate.

When you are in the Discovery or Beta phase of survey development

Cognitive interviewing can be used to inform the design of questions in your questionnaire or your respondent materials.

This should be completed after you have gathered respondent insights in Discovery to inform your design.

You should then conduct cognitive interviewing in Alpha, before then testing your survey at scale in Alpha

Usability testing is a method of seeing how easy or difficult something is to use by testing it with real users. This is done through one-to-one sessions where users are asked to complete tasks whilst being observed by a researcher. The sessions focus on the physical interaction with the product or service, rather than the participant’s understanding of questions, for example.

Usability testing can be done when designing a new product or service, or when you’re improving an existing product or service.

You can use retrospective and concurrent probing to test your questions or materials. Learn more about retrospective probing and concurrent probing.

Insights gathered from the method or activity

Usability testing allows you to:

identify whether your questionnaire is easy or difficult to navigate or complete
identify any problems or “pain points” your users are encountering so you can then iterate and fix any problems before launching the survey at full scale – you should make sure you re-test any changes you make to ensure they are effective
test the invitation letters and login pages, to see how easy or difficult they are to follow and navigate through.

When the method or activity may be appropriate to use

When you are developing new questions and materials for a survey

Usability testing can be used to see how easy or difficult your respondent materials and questionnaire are for respondents to use. Usability testing helps you validate whether designs you have produced are fit for purpose.

When you are developing a survey in self-completion mode

Usability testing can be used when developing new questions and materials for all survey modes. But it is even more important for self-completion surveys, where there is no interviewer present to support respondents and help them navigate the questionnaire.

When you are redesigning an existing survey

Usability testing is useful when redesigning an existing survey. You should conduct usability testing before making any decisions about design changes to ensure that you are working with a user-centred design approach.

When you are in the Alpha phase of survey development

Usability testing is used to evaluate how easy or difficult your respondent materials and questionnaire are for respondents to use. This should take place in the Alpha phase, after you have designed your questionnaire and materials based on insights gathered in Discovery.

When the method or activity may not be appropriate to use

Usability testing should not be a replacement for cognitive interviewing

Usability testing should not be used as a replacement for cognitive interviewing. The two methods aim to gather different insights.

When you have limited time, money, or researcher capability

Usability testing is by nature expensive, time-consuming, and requires significant researcher capability. You should consider timelines and budgets when deciding whether cognitive interviewing is appropriate.

When you are in the Discovery or Beta phase of survey development

Usability testing requires you to have designed prototypes of your questionnaire and materials. This means you should complete your Discovery phase before you are able to conduct usability testing.

You should conduct usability testing to validate your questionnaire and respondent materials are usable and fit for purpose before you test them at scale in the Beta phase

Cogability testing is where cognitive interviewing and usability testing are conducted at the same time. It can be beneficial to research these two aspects together because of the interactions between question wording, instrument design, and the response process. You would usually run 30 interviews in one round.

Survey questions and any survey materials should be tested to ensure they:

are understood as intended
are usable
meet respondent needs
are not burdensome for respondents

Insights gathered from the method or activity

Cogability testing will give you insights on:

the comprehension and understanding of your questions
how easy or difficult the questionnaire is to navigate and complete

This is important as one may affect the other. For example, the look and feel of a questionnaire may affect how the respondent understands it. A good example of this is square response options boxes. They usually indicate that more than one response option can be selected.

We recommend reading the sections on “cognitive interviewing” and “usability testing” to understand the different insights cogability testing can gather.

When the method or activity may be appropriate to use

When you want to follow survey development best practice to ensure quality

Cogability testing is the recommended approach to testing survey questions and materials because of the benefits of combining cognitive interviewing and usability testing in the same session, as well as the reduced burden on time and resource.

When you want to gather insights about the usability or comprehension of your questionnaire and respondent materials

Cogability testing is useful when you want to understand the quality and comprehension of your developed survey questions and materials, alongside how easy or difficult they are to use.

When you are developing a survey in self-completion mode

Cogability testing can be used when developing new questions and materials for all survey modes. But it is even more important for self-completion surveys, where there is no interviewer present to:

support respondents
help them understand the questions
help them navigate the questionnaire

When you are in the Alpha phase of survey development

Cogability testing is used to evaluate:

how easy or difficult your respondent materials and questionnaire are for respondents to use
whether participants understand survey questions and respondent materials as you intend them to

This should take place in the Alpha phase, after you have designed your questionnaire and materials based on insights gathered in Discovery.

When this method or activity may not be appropriate to use

When you have limited time, money, or researcher capability

Cogability testing is a time-consuming and expensive method that requires significant researcher capability. You should consider timelines and budgets when deciding whether cognitive interviewing is appropriate.

But it requires less time and resource than conducting cognitive interviewing and usability testing separately. If you are planning on undertaking these research methods, you will save resource by conducting cogability testing.

When you are in the Discovery or Beta phase of survey development

Cogability testing requires you to have designed prototypes of your questionnaire and materials. This means you should complete your Discovery phase before you are able to conduct cogability testing.

You should conduct cogability testing to validate your questionnaire and respondent materials are usable and fit for purpose before you test them at scale in the Beta phase.

Unmoderated testing is a user research technique where usability testing may be completed remotely, without a moderator or interviewer present.

Unmoderated testing may consist of the administering of a task or may simply consist of a series of questions.

While typically used to assess usability, the technique may also be applied to assess comprehension and acceptability of survey items – but note that it is not as robust as cognitive interviewing.

Learn more about unmoderated testing of survey questions.

Insights gathered from this method or activity

Unmoderated testing allows you to gather insights on the user experience. For example, it will help you understand how easy or difficult your questionnaire or materials are to use or navigate.

When the method or activity may be appropriate to use

When you want to quickly validate designs with a diverse group of users at scale

Unmoderated testing allows you to engage with a diverse group of users at scale because:

participants can take part remotely
the unmoderated nature of this method means it requires less resource to administer

When you have a specific research question you are investigating

There will be no researcher present during unmoderated testing. As a result, it is most effective when it involves a short task and investigates a specific research question.

When you have limited resource and want to gather mass insights about usability, comprehension, or acceptability

Unmoderated testing is useful when you want to conduct research regarding usability, comprehension, or acceptability at scale. This is because it can be done remotely and requires fewer resources than cogability testing.

Because it is unmoderated, it can be useful for mass testing when budgets, resources or time are limited.

When you are in the Alpha phase of survey development

Unmoderated testing is used to evaluate how easy or difficult your respondent materials and questionnaire are for respondents to use. This should take place in the Alpha phase, after you have designed your questionnaire and materials based on insights gathered in Discovery.

When the method or activity may not be appropriate to use

Unmoderated testing should not be used instead of cognitive interviewing or cogability testing

It is not appropriate to use unmoderated testing instead of cognitive interviewing or cogability testing. If time and budget allows, you should always conduct these methods instead, as they are more comprehensive forms of testing.

When you want to ask follow-up probes

You should not use unmoderated testing when you want to ask follow-up questions or allow your respondents to ask questions.

But it may be possible to conduct follow-up in-depth research, such as qualitative interviews with respondents about their feedback in unmoderated testing, to gain further insights.

When you are in the Discovery or Beta phase of survey development

Unmoderated testing requires you to have designed prototypes of your questionnaire and materials. This means you should complete your Discovery phase before you are able to conduct unmoderated testing.

You should conduct unmoderated testing to validate your questionnaire and respondent materials are usable and fit for purpose before you test them at scale in the Beta phase.

A/B testing is a quantitative test of two or more potential design options which allows comparisons of their impacts on the results.

All other aspects of the survey design, apart from the options you want to compare, should be balanced equally across those approaches. This should include demographic variables that may be associated with survey outcomes. These variables should be balanced across the options to make the comparison of the impact of different conditions more precise.

These options you test can cover multiple aspects of a survey, such as:

differences between sampling methods
questionnaire wording or order
user interface designs
incentives
design of communications

Where you need to distinguish the separate impacts of these, the experiment should be carefully designed so that these are crossed in the sample.

Insights gathered from the method or activity

A/B Testing allows you to identify differences between variations on a specific item. This may be needed where you have multiple ways of implementing an element of the survey but are unsure which is the best option.

An A/B test would allow you to see the results of these differences, with all other parts of the survey design staying the same. Any differences detected would then be solely due to the different options tested in the experiment.

The number of test conditions should be considered alongside time frames and sample sizes. More conditions will need a larger sample size to be powerful enough to detect differences in impacts across those approaches.

Learn more about A/B Testing.

When the method or activity may be appropriate to use

When you have multiple options for implementing a design aspect and are unsure which to use

A/B testing can be used to identify which design option would be best to use when you have multiple choices.

When it is possible to limit your test to only testing a single, specific aspect of design

A/B testing should only be used when it is possible to limit different tests to only one change in design. If you have multiple design changes in the tests, you will not be able to establish what outcomes are established with what design change.

When you have sufficient budget, and your end-product is important enough to justify the expense

A/B testing can be expensive because of the cost of getting large enough sample sizes to detect statistically significant differences. The cost is often high because complex testing can be difficult to implement operationally. This means that A/B testing can only be conducted when you have sufficient budget and when your end product is important enough to be worth the expense.

Despite the cost, the benefit of the insights gathered may outweigh the expense. It can also be an effective way of mitigating the risk of a survey failing, or requiring further budget for additional refinement when live.

When you are in the Beta phase of survey development

A/B testing is typically used in the Beta phase to assess the effectiveness of different survey processes, designs, or conditions. It may also be used during Live as part of your continuous improvement process.

When the method or activity may not be appropriate to use

When you want to test multiple aspects of your survey

You should not use A/B testing when you want to test multiple aspects of your survey. For example, you would be unable to test questionnaire wording and user interface designs on the same test. This is because it can be difficult to establish which aspect is influencing the outcomes and to what degree. You would need to conduct separate tests for each element.

When you are in the Discovery or Alpha phase of survey development

A/B testing is not suitable for the Discovery or Alpha phase. This is because the survey concepts and survey designs are unlikely to be developed enough for this research method.

A pilot survey is where the researcher sends out a newly developed survey to a smaller sample size compared to the target audience. The sample will not always be entirely representative of your target population, but the sample should be random probability based. This means the selection of a population sample is based on the principle of random selection. For a pilot survey, you should use the main survey sample design, but only send the survey out to a random subset of the selection for the pilot.

Pilot surveys are commonly used as they allow you to assess your entire questionnaire and materials under survey conditions. This allows you to identify any problems before implementing the whole survey.

Typically, following a pilot, small changes will need to be made to the:

questionnaire
respondent materials

You may also need to make changes to the sample design.

Insights gathered from the method or activity

Pilot surveys can provide insights on:

likely response and return rates
response profiles
likely fieldwork costs
paradata, such as question or questionnaire length, completion rates, and partial completion rates
the quality of results
initial methodological considerations
some early operational considerations

When the method or activity may be appropriate to use

When you want an early indication of survey response rates and data quality

Pilot surveys can be used to gather insights regarding survey response rates and data quality.

When you want to test the respondent journey in a quantitative setting

Pilot surveys allow you to test the respondent journey in a quantitative setting. This will allow you to identify any barriers to completion.

When you want to test your interviewer briefing

Pilot surveys will allow you to test your interviewer briefing with interviewers as part of a live trail. This will help you work out whether your interviewer briefing is fit for purpose or needs any changes.

When you want to gather interviewer feedback about your survey

Pilot surveys will allow you to gather feedback from interviewers about how the survey works from end to end, and how it works anecdotally with respondents. It is essential to hold debrief sessions with interviewers to get these insights.

When you have sufficient resource and money

Pilot surveys can be expensive and resource intensive. But the benefits of the insights gathered outweigh the costs, and pilot surveys can be an effective way of mitigating the risk of a survey failing when live

Pilot surveys can be smaller scale, which makes them less expensive if you are planning for a pilot and dress rehearsal stage.

When you are in the Beta phase of survey development

Pilot surveys involve testing your survey at scale. They should be used in the Beta phase.

When the method or activity may not be appropriate to use

When you want to gather information about respondent comprehension of survey questions

Pilot surveys cannot help you understand if respondents understand your survey questions.

This understanding should have come from earlier stages of survey development.

When you want to gather information about the acceptability of survey questions

Pilot surveys cannot help you understand if respondents are willing to answer your survey questions.

This understanding should have come from earlier stages of survey development.

When you are in the Discovery or Alpha phase of survey development

Pilot surveys are not suitable for stages before Beta, as the survey concepts and survey designs are unlikely to be developed enough for this research method.

Description of method or activity

A survey dress rehearsal is a test of the operation of the end-to-end survey design. They are typically done for large or very complex surveys, especially if they are new.

The emphasis of a dress rehearsal is about operational learning. The expectation is that no changes should be needed to the questionnaire, materials and sample design. Most changes between the Dress Rehearsal and the Live phase will be from operational learning.

Insights gathered from the method or activity

A dress rehearsal gathers insights on:

the efficiency of the operation, which will help to ensure the survey runs smoothly when it is launched
survey costs
estimate population variances
the quality of the data being collected

A dress rehearsal should cover all aspects of the survey design which flows continuously from end-to-end. This is an opportunity to test operational processes and statistical methods at scale and should be a precursor to a full survey launch.

As in the theatre, the dress rehearsal should mimic the Live stage as much as possible. No design changes should be planned to happen between the rehearsal and the Live stage. But any unforeseen problems during the dress rehearsal should be corrected for the Live stage.

When the method or activity may be appropriate to use

When you have completed your feasibility testing and your pilot survey

A dress rehearsal is a final test of the operation of your end-to-end survey design. You should have completed your feasibility testing and pilot survey before conducting it. This is so you can identify and fix any issues before undertaking a dress rehearsal.

When you have sufficient resource, budget, and operational ability to conduct a full end-to-end test cycle

You should ensure you have the resources and operational ability to conduct a full end-to-end test cycle.

Dress rehearsals can be expensive to operate, but they can help you significantly mitigate the risks of survey failure at the live stage.

When you are in the Beta phase of survey development

Dress rehearsals are typically the final piece of research you undertake in the Beta phase. This is where you are testing your survey at scale before advancing to the Live phase.

When the method or activity may not be appropriate to use

When aspects of survey design are unclear

You should ensure aspects of survey design are clear before undertaking a dress rehearsal. There should not be any aspects of design that require more individual detailed testing. If there are, you should research these individually before undertaking a dress rehearsal.

When you have not fixed issues identified in your feasibility testing or your pilot survey

Your feasibility testing and pilot surveys are an opportunity to identify any issues with your survey design.

You should ensure you have addressed any issues identified in these before undertaking an expensive end-to-end operation of the entire survey design.

When you are in the Discovery or Alpha phase of survey development

Dress rehearsals are typically the final piece of research you undertake in the Beta phase. They should be undertaken once you have validated the quality of your survey design.

Because of this, dress rehearsals should not be used in the Discovery or Alpha phase, as survey concepts and designs are unlikely to have been developed far enough.

Participatory methods do not refer to a single specific research method.

Instead, the term refers to incorporating participatory elements into your research design for participants to complete. For example, you may choose to conduct focus groups that include a participatory task, or design your research interviews in a way that prioritises participatory elements.

The aim of incorporating participatory methods will vary on the specific nature of your research aims, but they typically involve working with members of the participant population, rather than conducting research on them. This allows you to gather insights from the targeted participant population through a method that may better suit specific participant needs.

Examples may include:

gathering respondent insights in focus groups — you could create a task for participants to review a series of different prototypes of invitation letters and then design their own
using research methods that include participatory methods if your target population have specific needs — for example, if your target population includes young people with special educational needs or disabilities you could use research methods that include participatory methods to better meet their needs.
engaging with experts by experience (EbE) advisory groups to help ensure research is suitable for the targeted participant population — by interacting with EbE, and getting their feedback about research design, you can ensure future research you undertake is suitable for your targeted participant population

Insights gathered from the method or activity

Participatory methods can be used for different purposes, so the insights gathered from this approach will vary. Examples of insights this approach can gather include:

respondent insights regarding design prototypes
target population insights to ensure future research design is informed and suitable
respondent insights from groups that may find it challenging to engage with more traditional research methods.

Participatory methods are flexible and typically relaxed. Participants are encouraged to freely discuss their thoughts, opinions, and experiences, rather than being guided by a topic guide, or any other discussion framework. This allows participants to feel more comfortable sharing their views and gives them more control over what they are comfortable sharing, and not sharing, with the researcher.

When the method or activity may be appropriate to use

When you want to maximise inclusivity and appropriateness of language throughout the research

The flexible and participant-focused nature of participatory methods research maximises inclusivity and helps make sure that language and communication is appropriate throughout the research. These methods minimise the risk of harm or exclusion.

When flexibility is important to be able to meet participant needs

The flexible nature of participatory methods research helps to ensure you meet participant needs and can match specific preferred engagement styles.

When a more relaxed and informal research environment may be needed

Participatory methods can be used when a more relaxed, informal research environment may be necessary to help participants feel comfortable and build trust with researchers. Using these methods can help participants feel more able to share their experiences or perspectives

When you do not know much about the research topic or targeted population

Participatory methods may be useful when you do not know much about the research topic or targeted population. This is because participatory methods can be used to facilitate participant control in an exploratory setting that is not limited by researcher assumptions.

When you are in the Discovery or Alpha phase of survey development

Participatory methods can be used for multiple different research aims or purposes. This means you can use participatory methods at different Agile phases, depending on the context of your individual project.

If you want to gain insights to inform later research, you should use participatory methods during the Discovery phase. This will help ensure you have an informed understanding of participant needs at an early stage to inform your design.

If want to use these methods for evaluative purposes, such as gathering respondent insights regarding invitation letter prototypes, you should use participatory methods in Alpha.

When the method or activity may not be appropriate to use

When there is limited time to conduct research

Participatory methods may not be appropriate when there is limited time to conduct research. It takes time to engage with and build trust with group members.

When there is no flexibility in the research aims

For meaningful participation to occur, you should allow respondents to discuss what they want to talk about. The insights gathered from using participatory methods may not align with your intended research aims, but it is important that the researcher does not interrupt.

When you are in the Beta or Live phase of survey development

Participatory methods research is typically used in the Discovery or Alpha phase to gather respondent insights or for evaluating designed prototypes.

This should be done before the Beta or Live phase.

Glossary

Agile

Agile is a development methodology used for project management and product delivery. Agile delivery offers flexibility through the implementation of stages of development that are iterative, incremental, and sometimes cyclical. As per the GOV.UK service manual, Agile is the recommended approach for building and running government digital services.

Discovery

Discovery is the first stage of Agile development. It is an exploratory phase where you gather information regarding user needs that will inform subsequent development of your end-product.

Alpha

Alpha is the second stage of Agile development. In this phase, you use insights gathered in Discovery to begin designing and testing your end-product and associated materials. Once designs are tested, you should refine them based on user feedback and then retest.

Beta

Beta is the third stage of Agile development. At this stage, you conduct quantitative testing on prototypes you have developed in previous Agile stages. This is to validate the effectiveness of your prototypes at scale and to determine whether your survey is ready to go live.

Live

Live is the fourth stage of Agile development. At this stage, your survey will be live and collecting data. You should only enter this stage once your Beta testing has been conducted, quality targets have been achieved, and the survey is deemed fit for purpose. Live is about continuing to monitor the end-product and making improvements and iterating when appropriate.

Waterfall

Waterfall is an approach to project management where the end-product is defined during the early stages of project development.

User

The person using a product or service. For example, this could be the respondent to a survey, or a participant in a focus group.

Respondent

A user in the context of completing a questionnaire. May be used interchangeably with “user” in this context.

Stakeholder

The main customers of your project and end-product. For example, this may be the survey commissioner or policy makers, who you may be meeting to discuss data requirements of your end-product.

Data user

The customers of your end-product’s data (in this context, the data gathered from your survey). These will be the people using your data, for example, to produce outputs or inform policy.

Mental models

What a user believes or understands about something, and the thought process a user goes through to reach a certain response. This shapes how a user will interpret something, how they will think and feel about it, and how they will subsequently respond to it. Learn more about mental models.

Respondent burden

The degree to which a survey respondent perceives participation in a survey research project as difficult, time consuming, or emotionally stressful. Interview length, cognitive complexity of the task, required respondent effort, frequency of being interviewed, and the stress of psychologically invasive questions all can contribute to respondent burden in survey research. Learn more about how to monitor and reduce respondent burden.

Respondent Centred Design

Respondent Centred Design is an adaptation of User Centred Design, adapted specifically for application to survey development. Learn more about Respondent Centred Design.

Respondent insights

Information you gather directly from respondents regarding their needs, question comprehension, or mental models.

Respondent journeys, or user journeys

The process that a respondent will go through in order to complete the goal. In a survey context, that would be accessing and completing the questionnaire. Learn more about respondent journeys.

Respondent needs, or user needs

The needs that a user has of a service or product. Learn more about gathering and recording respondent needs.

Respondent, or user stories

Respondent stories describe the respondent and why they are completing your questionnaire. Learn more about respondent stories.

User-centred products

Products (in this context, a questionnaire and respondent materials) that have been designed in a user-centred manner to meet user needs.

“At desk”

The activity of gathering information by reviewing previous research, instead of conducting new research yourself. Examples of at desk research may include:

conducting a literature review
contacting colleagues or peers to investigate whether they have conducted any research on the topic previously
investigating whether harmonised standards exist for the topic you wish to gather data on

Concurrent probing

A follow-up technique used by researchers in user-research settings. It involves the researcher interjecting the participant, to immediately ask follow-up questions if the participant says something of interest. Learn more about concurrent probing.

Retrospective probing

A follow-up technique used by researchers in user-research settings. Unlike concurrent probing, retrospective probing would be used after the user has completed a usability task. During task completion, the researcher would remain silent and allow the user to complete their task uninterrupted. Learn more about retrospective probing.

Data requirements

Your stakeholders’ requirements for the data that your survey will collect.

Design patterns

Patterns are best practice design solutions for specific user-focused tasks and page types. Using these design patterns makes Government products and services consistent with GOV.UK. Learn more about design patterns.

End-to-end

The process of participating in something in its entirety. For example, the end-to-end process for a user participating in a survey would include all of the following:

being invited to participate in a survey
the process of agreeing to participate
all ensuing communications during participation
the process of actually engaging with the questionnaire (this may be navigating to an online portal, travelling to meet or hosting an interviewer at your house, or engaging with an interviewer over the telephone)
the process of proving responses to each questionnaire item
submitting or completing the survey
being re-contacted for future survey waves (if the survey is longitudinal)
being debriefed or receiving incentives upon survey completion

Harmonised questions

Harmonised standards produced by the Government Statistical Service (GSS). Harmonised standards are survey items that have been developed by the GSS’ Harmonisation team and have been tested to ensure the questions are of high-quality. Learn more about harmonised standards.

Ideation session

The process of generating ideas and solutions to problems. An ideation session is a collaborative method of undertaking this process with colleagues. Learn more about organising and conducting ideation sessions.

Interviewer-administered modes

Any mode of data collection where an interviewer is present to ask the participant questions and assist with data collection.

Item response

Data obtained from item response refers to data gathered in response to a question within your survey.

Prototype

A working version of a product. Examples include an invitation letter, a specific question with your questionnaire, or the entire questionnaire itself. Prototypes are used to test the concept and gather qualitative or quantitative insights regarding its performance and comprehension.

Research grid

A tool to define research aims and direct associated tasks. It is used to track progress and can be adapted as you progress through Agile phases. Learn more about research grids.

Respondent attrition

Respondent attrition refers to respondents dropping out of a survey midway through. This may refer to respondents dropping out of your survey midway through a questionnaire, or, in the context of longitudinal surveys, dropping out between survey waves.

Respondent materials

Any materials respondents may be asked to engage with whilst participating in your survey. This includes an invitation letter, information leaflets, the questionnaire itself, and any other resources that are presented to respondents during survey participation.

Topic guide

This is used by researchers during research sessions. Topic guides outline key areas of questioning to guide a qualitative interview or focus group. They may include specific questions or just topics of interest.

User experience (UX)

How a user interacts with and experiences a product, system, or service. This includes user’s perceptions of ease of use, efficiency, and utility: in other words, how ‘usable’ a product, system, or service is.

User interface designs

The design (appearance and functionality) of software that users interact with. In a survey context, this may refer to the online portal that will host your survey, which respondents will be using to complete the questionnaire.

User research

The methodical study of target users, to ensure that developed end-products meet user needs. User researchers aim to understand user needs, behaviours, and mental models, through conducting research with users. Learn more about user research.

Table of contents

The purpose of this toolkit

How to use this toolkit

Agile phases of survey development

The benefits of Agile development

Respondent Centred Design Framework

Relevant standards and frameworks

Discovery phase

Notes

Objectives of the Discovery phase

Understand what you are being asked to develop and the purpose of the product

Why this objective is important

Suggested research methods or activities

Understand who your target research participants are

Why this objective is important

Suggested research methods or activities

Conduct at desk research

Why this objective is important

Suggested research methods or activities

Establish and document the respondent needs

Why this objective is important

Suggested research methods or activities

Preliminary understanding of respondent mental models

Why this objective is important

Suggested research methods or activities

Develop draft respondent stories and respondent journeys

Why this objective is important

Suggested research methods or activities

Identify how you will measure success

Why this objective is important

Suggested research methods or activities

Gain assurance that you’re developing the right product

Why this objective is important

Suggested research methods or activities

Alpha phase

Objectives of the Alpha phase

Ensure your survey is inclusive and accessible

Why this objective is important

Suggested research methods or activities

Validate your understanding of the insights gained in Discovery and gather more

Why this objective is important

Suggested research methods or activities

To learn whether respondents can understand the products as intended

Why this objective is important

Suggested research methods or activities

To learn whether respondents can use the products as intended

Why this objective is important

Suggested research methods or activities

Beta phase

Objectives of the Beta phase

Ensure operational process is functioning as planned

Why this objective is important

Suggested research methods or activities

Gather statistical survey data insights

Why this objective is important

Suggested research methods or activities

Test the effectiveness of your statistical survey design

Why this objective is important

Suggested research methods or activities

Gather statistical paradata insights about your data

Why this objective is important

Suggested research methods or activities

Test the effectiveness of data processing methods

Why this objective is important

Suggested research methods or activities

Develop a weighting and imputation strategy

Why this objective is important

Suggested research methods or activities

Undertake evaluation of survey design

Why this objective is important

Suggested research methods or activities

Live phase

Objectives of the Live phase

Identify where you may need to amend questions or response options

Why this objective is important

Suggested research methods or activities

Analyse response rates, including return rates and response patterns

Why this objective is important

Suggested research methods or activities

Monitor the characteristics of responding and non-responding households and individuals