Inclusivity and accessibility in survey development

Inclusivity and Accessibility are a part of several laws and standards we must follow. This includes the Equality Act 2010, the Human Rights Act 1998, and The Public sector wquality duty, which came into force in April 2011.

The Equality Act 2010 provides a legal basis to make sure that people are not discriminated against on the grounds of nine protected characteristics, which are:

• age
• disability
• gender reassignment
• marriage and civil partnership
• pregnancy and maternity
• race
• religion and belief
• sex
• sexual orientation

Other laws and official guidance to be aware of when designing and developing your survey products, for example:.

• if you are building something that runs on a website or mobile application, you will need to meet the Public Sector Bodies Accessibility Regulations 2018
• if you are procuring hardware or software, you will need to meet the EN 301 549 Accessibility Requirements for ICT (Information Communications & Technology) products and services
• if you are working to the GOV.UK Service Standard, you will also need to meet the Government Accessibility Requirements
• the Web Content Accessibility Guidelines (WCAG) list of criteria your website or mobile application needs to meet to be legally compliant

There are  many government resources available to help you to design and build your survey and survey materials in an inclusive and accessible way. These include the Government Service Standard, the GOV.UK Service Manual, and Government Design Principles, among others.

Point 5 of the government Service Standard is to, ”Make sure everyone can use the service”  including people with disabilities and people with other legally protected characteristics.

The standard highlights that, “Government services must work for everyone who needs to use them, so there’s a legal duty to consider everyone’s needs when designing and delivering services”. The standard can be used by anyone working in the public sector and requires you to think about the user’s wider journey, and the work you have done to make sure the transaction you’re building joins up with that wider journey.

The GOV.UK Service Manual can be used to help you meet the Service Standard.

You can also use the GOV.UK Design System Patterns to adopt best practice design solutions for specific user-focused tasks and page types.

The government Design Principles help to establish and maintain good design practice. Principle 6, ‘This is for everyone’, is about developing inclusive and accessible products and services.

You should adhere to these principles to ensure you design survey products that follow good design and meet the needs of the users.

The ONS has also developed Survey Strategy Research and Development Principles (SSRDP), which are value statements that set the standards and ways of working for everyone to follow. The ‘Everyone counts’ principle focuses on inclusivity and accessibility.

You should take a Respondent Centred Design (RCD) approach to the design of your survey. RCD is defined as “learning about the needs of those who will use your service and designing it to meet them”. RCD enables inclusivity and accessibility to be embedded from the start of the research process, as it puts the needs of the respondent at the forefront.

In the context of survey design, RCD is an approach that brings together best practice from the Government Social Research profession and user experience design. By being inclusive and accessible, you then reduce burden for all users, and not just those with disabilities. RCD also involves understanding respondent experiences and needs so that you can identify any friction points or barriers and design to overcome these. You can use the RCD Framework (RCDF) to learn more about how to implement RCD in your survey design.

RCD also aligns with Agile Delivery principles, which include focusing on user needs and delivering iteratively. Iteration is key when implementing RCD and links to point 5 in the Government Design Principles (‘Iterate. Then iterate again’) and the ‘Everyone counts’ principle of the ONS Survey Strategy Research and Development Principles.

By conducting research to understand the needs of respondents, you can ensure the questionnaire and any associated survey materials meet the following goals:

• reduce respondent burden
• are highly accessible and usable
• use language that respondents understand as intended
• results in a positive experience for your respondent
• gathers data of high quality

Stakeholder engagement is an important step of survey design and can help ensure that your survey is more inclusive and accessible, as it will:

• help you to accumulate knowledge and expertise
• build shared understanding of different needs and requirements from a wide range of groups and individuals with different experiences and perspectives

Stakeholders can help you to better understand the purpose of your survey, set priorities, encourage participation from target respondent groups, and communicate findings. You should educate stakeholders on how different people use technology so that your stakeholders understand how this can be used to improve accessibility. Involve them in your research, share examples, and highlight user quotes.

Stakeholders might include:

• policymakers
• expert groups and charities
• gatekeepers
• data users
• funders
• survey developers
• community members

More information about how to identify and map your stakeholders can be found in guidance on stakeholder mapping created by the Engagement Good Practice team from the Engagement Hub at the ONS.

Stakeholder engagement should be an iterative process, which is ongoing throughout the development of your survey. Engagement should start early, to ensure that you achieve maximum benefit.

You should check with your organisation’s communications and research teams, to ensure that any stakeholder engagement is aligned across your organisation before you get in contact with stakeholders. Your organisation may have existing contacts and relationships with those who you would like to contact. It may also be beneficial to agree as an organisation a set of user groups or user personas for research projects to ensure that you are covering the right groups of stakeholders and to make sure that no-one is being missed.

How you engage with your stakeholders will depend on your goals, and the input expected from each group or individual. Engagement methods may range from workshops, focus groups or advisory boards, to briefings and newsletters, and this will depend on what you want to achieve from each relationship. Mapping your stakeholders based upon their level of interest, influence, and impact will help you to develop an effective engagement plan.

You may also benefit from involving Experts by Experience (EbE) in your survey development. EbE are research advisors that share characteristics with research participants, for example homelessness, or a certain health condition. They can help shape the research and the research process.

Whilst the level of involvement of EbE may vary, typically they may provide input and consultation on aspects of the research such as your methods, materials, findings and outputs. This may help to maximise relevance and help to determine an optimal development approach, building trustworthiness, and ensuring the interview is a meaningful experience for participants.

Examples of working with EbE can be found in the following projects:

• Experiences of displaced young people living in England
• Survivors of domestic abuse and their lived experiences with temporary ‘safe’ accommodation in England

You should engage with EbE in the early stages and continue to work together throughout the survey development. You should work this into your timelines at the beginning of the project, to make sure that you have enough time to engage with EbE in a meaningful way.

Involving EbE in your design can have many benefits, including:

• ensuring your survey products are user-centred (that is, that they are fit for purpose and meeting user needs)
• having a means of quality assurance, as EbE can be used to ‘sense check’ findings to ensure researchers’ interpretations are seen to accurately represent experiences of the population of interest
• creating solutions to challenges that may be more difficult to overcome without involving EbE
• empowering EbE, working towards their feeling more supported, respected and listened to by acknowledging their lived experience as valuable expertise.

In addition to the general guidelines, some survey-specific considerations include:

• designing for screen readers: use the phrase “select the button”, rather than “click the button”, for example
• designing for respondents with colour blindness: consider the colours used in both your questionnaire and survey materials and make sure they are suitable for colour blind respondents. You should avoid communicating information using only colour and also avoid giving instructions that rely on colour. The Government Service Manual has guidance on colours that will help you to ensure consistency across your products. Your organisation may have its own requirements, so make sure to check with your content design teams
• designing for those with anxiety: ensure the steps the respondent needs to take are easy to understand and easy to follow
• designing for dyslexia: The British Dyslexia Association has a useful style guide that helps to ensure that written material, such as survey questionnaires and survey materials, consider the difficulties experienced by some dyslexic people
• designing for those with visual impairments: provide text alternatives for images so that screen readers can convey the content to users.

Other considerations relating to the layout and formatting of your survey products include:

• using the ‘pyramid structure’: put the most important content first, followed by important details, and then add any other general and background information
• using descriptive headings to help users scan your content
• considering your sentence structure, for example in materials such as invitation letters. Is there a logical flow to the content?
• Having only one question per page in a questionnaire to avoid the need to scroll and to help avoid cognitive overload for the user

There are different ways in which you can ensure your survey products are compliant with accessibility standards. These include:

• using accessibility tools to help you determine how the technology you are using will work with different assistive technologies
• utilising accessibility applications (programmes designed to perform specific functions) to support the end user complete your survey. For example, an accessibility application may give users the option to use a text to speech function, which would allow users with vision impairment to hear the survey questions read out loud
• working with your department’s communications team to create and publish products that meet requirements.
• You may wish to work with an accessibility company, who can work with you to ensure that your products meet best practice in accessibility standards and legislation.

Remember, accessibility and inclusivity should be considered from the very beginning of your survey development and be regularly reviewed throughout the project lifecycle.

Principle 5 of the RCDF states that you should use language, readability and tone that is suitable for your reader. This is key to reduce ‘burden’ on the respondent and increase the accessibility of your content.

By using complicated, unfamiliar language you risk burdening your respondent, which can reduce engagement with your product. Choosing the right tone for your survey products is important and should be established through research. For example, using a formal, authoritative tone can be off-putting, whereas an overly friendly tone may raise concerns about legitimacy. It is important to learn about users’ expectations of tone from your organisation before creating products. Inappropriate language and tone can have adverse effects such as reduced response rates, lack of inclusivity and lack of compliance with Government Digital Service standards.

Reading age

The average reading age of people in the UK is nine years old. Therefore, all content should be tested for this reading age, including both questionnaire and survey content and materials. There are various free online tools available to test readability, though you should take care not to put official sensitive information into these tools, or anything that is not public, such as qualitative data.

There may be instances where achieving full readability will not be possible, for example when discussing specialist topics or when the message could become compromised by adapting messaging. In these cases, try to make your content as simple as possible.

In general, your content should use:

• plain English
• simple vocabulary, avoiding vague and complex terminology
• short sentences with 25 words maximum
• avoid colloquialisms (words or phrases used in a specific geographical regions), for example, “apartment” (in America) vs “flat” (in UK)
• avoid abbreviations

English as an additional language

Your survey should be representative of the group you are studying, and often this will include groups with limited understanding of English. By using overcomplicated language, you risk alienating respondents and potentially excluding a key demographic of the population. This could result in poor quality data that is not representative of the population.

As well as following the guidelines to meet the reading age of the UK, you should also consider:

• using words that translate directly. Consider the different meanings of the words being used in other languages and ensure the wording is read as intended
• avoiding contractions. A word that combines two or more other words in a shortened form, for example, you should say “is not”, rather than “isn’t”

To ensure you are including people who speak other languages you should, where possible, be conducting research in these languages. Where this is not possible, consider providing translation services. User testing is also particularly important when designing products and services for non-native speakers. Usability testing will help you to establish the content users have trouble understanding. This should not only include your survey questions, but also any other participant materials such as consent forms. If participants have difficulty responding to questions, they are more likely to select inaccurate responses, choose the “don’t know” option, or time out of the survey completely. Make sure that you work with these groups from the start and keep them involved throughout the process.

Where there is a legal obligation to provide the same service for those who speak another language (for example, in Welsh) any products created must also be available in that language as standard.

Before you commence your project, make sure your organisation has a translation protocol in place. This should include agreeing:

• the methodological approach to translation that you want to take
• how to resolve differences between conflicting translations
• a consideration of risks and mitigations

Many households may have a very competent English speaker who can assist the person in completing the survey. Whilst this may be the easiest way of gathering data, it is not ideal, as there is little control over how the questions are translated, and how responses are interpreted. This may be particularly problematic if the survey is sensitive in nature, as participants may not wish to respond via a friend or family member.

Why is this important?

Government products and services must work for everyone (as mentioned in the Service Standard). Inclusive, accessible services are better for everyone, whilst also improving the quality of the data collected. By using appropriate tone and language, we minimise the risk of excluding individuals or groups and maximise understanding and participation.

Methods and modes of data collection are an important part of the questionnaire process. The two main methods in which surveys can be administered are interviewer and self-administered. Each method has modes, or specific techniques, used to collect the data.

Interviewer-administered methods include:

• face-to-face
• telephone
• video

Self-administered methods include:

• paper
• web
• email

You may need to provide a mixture of modes to ensure you are being inclusive. For example, if you are developing an online survey, consider also having paper versions available for those unable to complete the survey online. If paper versions are to be made available, highlight this within any correspondence, making sure it is clear to participants that these materials can be requested.

When designing for multiple modes, we always design the self-complete mode first (whether this be online or paper) and then move on to develop the other modes. We do this because if a design can be easily used and understood in a self-complete mode, then it stands a high chance of being easily used and understood in an interviewer-led mode. Additionally, this aligns with the government’s digital and data strategy. The Government Service Manual provides several different patterns and components that you can use in your survey, which have already been tested and are accessible.

It is also important to consider the device a respondent is using.  A webpage or questionnaire will look different depending on the device therefore, it is important to use adaptive design when building webpages and questionnaires. Here, adaptive design involves designing for ‘smartphone first’ as this will have the least amount of space and ensures that your product is accessible on all device types. You should design and test across modes to ensure comparability of the data.

You should then take an ‘optimode’ approach to design, where you optimise and adapt to the mode being used. This means the respondent materials and questionnaire should be tailored for the mode that the survey is being administered in. This can be achieved through ’cogability testing’, which is a combination of usability testing and cognitive interviewing. This will improve respondent experience and data quality.

Why is this important?

By designing in this way, it can help respondents give us the data we need by reducing respondent burden. It also creates products that are more user friendly to our internal users, including interviewers and call centre staff.

By tailoring each product to the medium it uses and the specific needs of the users in that mode, it helps you get the data you need whilst improving data quality.

To ensure that your product is inclusive and meets your respondent’s needs it is important that you test it with respondents.

Testing designs through cognitive interviewing enables researchers to quickly learn what is and is not working so it can be altered and tested again. It can increase the understanding of question validity (the extent to which your questionnaire measures what it is supposed to measure) and response error (the return of false or subjectively modified information from survey respondents). It does this by tapping into participants’ mental models, learning about their thought processes, how they arrive at their answer, and so on.​

For self-completion modes, you should conduct ‘cogability testing’, which is the combination of cognitive interviewing and usability testing in one session. Cogability testing is often used in survey design as it can help capture a participant’s understanding of the questions being asked, and their ability to navigate the survey questions and the survey’s format.

Cogability testing will often use a ‘think aloud’ approach where participants are asked to talk through their thoughts and feelings as they go through your survey questions. This helps the researcher to understand how the respondent interprets and comprehends the task, and the thought-processes behind their decisions.It can be beneficial to research these two aspects together because of the interactions between question wording, instrument design, and the response process.

Whether you use cognitive testing, or cogability testing, you should make sure to include a wide range of user groups to test your survey. This will help to make sure that your product meets the needs of these different groups. Testing should be conducted throughout the design process, from beginning to end.

Ahead of your recruitment activities, think about how to make your communication inclusive and outputs accessible to your participants.

Here are the steps you can follow to plan inclusive testing:

• layout in a participant-centred way the steps and interactions participants will need to complete during the testing project
• use your survey design user personas to identify pain points, blockers in the accessibility of your methods or materials
• plan for solutions taking your pain points in consideration, innovate and utilise existing resources, such as communication teams, available tools or expertise
• track how well your solutions are performing, recording each against participant feedback or effectiveness; Share your solutions log with the team and others studying the same target population.

Why is this important?

By testing questionnaires and other survey materials with respondents, it ensures you are designing and building the right thing. By building the right thing (such as, the right survey), this leads to:

• reduced respondent burden
• increased data quality
• reduction in costs

Digital inclusion includes having access to the internet, being able to use digital services (such as computers) and having services that are accessible and meet the needs of users.

There are many barriers that people may face when accessing digital services. These include:

• access: not everyone is able to connect to the internet
• skills: not everyone has the skills to go online and use online services
• confidence: some people are fearful of online services, or lack trust

It is important to design survey products that address and overcome these barriers. Otherwise, there is a risk of excluding certain individuals or groups in your survey.

Here are some examples of how you can make your survey digitally inclusive:

• make the survey accessible to users with disabilities
• make the survey accessible for people who lack digital skills or internet access, for example, by providing digital support or offering alternative modes for completion
• use an adaptive design (think ‘smartphone first’) and test on different operating systems, but ensure the display can adapt to different screen sizes

Why is this important?

It is important to be digitally inclusive to ensure you are recruiting a representative sample in your survey. If a survey is not digitally inclusive, it has a risk of collecting poor quality data which does not represent the entire target population.

Some surveys may be administered by interviewers. These are the considerations that the survey designers need to be thinking about to make sure that the survey experience is inclusive and accessible for interviewers.

The considerations that need to be made will depend on the survey and the interviewers conducting the survey, so research should be carried out to identify what the inclusivity considerations are. You should explore interviewers’ needs and then decide how to best meet them.

Here are some ways in which you can identify interviewer needs:

• conduct focus groups with interviewers: identify any issues or concerns they may have with the day-to-day conducting of the survey. These findings can feed into the first iteration of testing with the public
• keep open communication with interviewers: organise a regular working group with your survey designers and interviewers, where progress, ideas and challenges can be discussed
• create a document summarising your recommendations and protocols for interviewers to refer to. For example, providing interviewers with clear protocols for how to handle situations that are common blockers

Why is this important?

Interviewers can be considered users of a survey just as much as the respondents taking part in the survey are.  As such, the needs of the interviewers conducting the survey are an essential part of inclusive survey design.

Failure to consider the needs of interviewers can result in:

• decreased interviewer job satisfaction potentially leading to higher staff turnover
• decrease in the quality of data collected
• increase attrition (that is, dropout) of respondents

Guidance for questions is additional information that is provided with a survey question to help the respondent answer that question.

Survey questions should be as self-explanatory as possible. If a question needs guidance to make sense, this puts more burden on the respondent. Quite often, respondents do not read, understand, or adhere to guidance that is made available to them. Where there is a problem, in the first instance you should always consider whether the question can be simplified. For example, this may mean splitting out one question into two or three simpler concepts.

Where, as a last resort, guidance is included, it is important to think about where it is placed. For online survey modes, the ONS Service Manual recommends using question guidance panels to give the user extra information that they need to answer a question.  A question justification, or information about why it is important to provide an answer,  should be positioned at the bottom of the page. Care should be taken to understand where a respondent needs and expects the guidance to be placed, and the need and placement should be informed by research, as is the case with the ONS Service Manual.

To identify whether a survey question needs guidance, and what that guidance should be, you should:

• hold cognitive or cogability interviews with a group of people who are representative of your survey’s target audience
• hold focus groups with a sample of interviewers who will be conducting the survey

Why is this important?

Guidance can be used to assist the respondent to interpret the question correctly and therefore respond with the desired data. Guidance can therefore support the improvement of data quality. However, we should always try to design better questions first, only using guidance where all other options are exhausted.

Harmonisation is improving the comparability, consistency and coherence of data and statistics. The GSS Harmonised standards include definitions, survey questions, suggested presentations, and information for data users. Producers of statistics can use these harmonised standards to align with others, which will increase the usefulness of their statistics. There are more than 40 harmonised standards and guidance available. Being harmonised is about aligning with others, not necessarily being identical to them. As such, some of the harmonised standards can be tailored for specific situations.

In some circumstances, it is not appropriate to suggest harmonised questions and definitions. In these cases, harmonisation guidance explains the topic landscape to help users understand where they can and cannot compare data.

Why is this important?

Harmonised standards provide a convenient and often off-the-shelf solution to help with data collection. Work on the standards has been developed by topic groups, after wide consultation across the Government Statistical Service (GSS) and beyond, ensuring that they are well-researched and have proven success. This can save time and money, whilst avoiding duplication. It can also help to improve coherence and comparability.

If you are using images or case studies in your respondent materials, it is important that these are diverse, that is that they include or reflect the variety of people in society otherwise, you run the risk of people with certain characteristics not feeling important or included in your survey. The more you represent diversity, the more engaged your respondents will be, making your survey more inclusive.

Why is this important?

Without diversity in your images or case studies, you risk alienating people from your survey. If people do not feel important, or included, they are less likely to take part. You then miss hearing from these individuals in your survey responses.