Respondent engagement for push-to-web social surveys

Document downloads

Respondent engagement for push-to-web social surveys: qualitative research report (DOCX 2MB)

Templates:

• ONS survey prenote – letter template (DOCX 87KB)
• ONS survey invite – letter template (DOCX 79KB)
• ONS survey reminder – letter template (DOCX 74KB)
• ONS Labour Market Study prenote – letter template (DOCX 95KB)
• ONS Labour Market Study invite – letter template (DOCX 90KB)
• ONS Labour Market Study reminder – letter template (DOCX 84KB)

These files may not be suitable for users of assisted technology. If you require any of these files in an alternative format, please email: GSSNet@statistics.gov.uk

Acknowledgements

This research has been completed over several years and has involved input from numerous researchers and experts across the Office for National Statistics (ONS).

Main contributors; Natalia Stutter, Laura Wilson, Sophie Nickson, Tara McNeill

Design; Thais Faria, Grace Ellins, Andy Budd, Rachel Price

Research support; Alex Nolan, Charlie Hales, Emma Dickinson, Vicky Cummings, Emma Timm and their teams

Additional thanks to our colleagues in Social Survey Transformation; the Survey Operations Research Team and the Materials Strategy Advisory Group.

Executive summary

This purpose of this report is to:

• share the learning from our qualitative research so that it can be used and recycled by others
• provide advice and recommendations for those developing respondent communication materials for their own push-to-web surveys

Over the last few years, the Social Survey Transformation’s Research and Design Team, as part of the Census and Data Collection Transformation Programme (DCTP), have conducted a series of qualitative and quantitative research activities to develop a suite of push-to-web respondent materials. Online completion is offered in the first instance as the primary mode of data collection.

In 2012 the government introduced a digital by default strategy which supports the design and delivery of digital services. Alongside this, standards and guidance on how to design digital services were released. Following this guidance, the approach to this work is respondent (user) centred.

Introduction

The ONS collects data through running large-scale household surveys. Traditionally, many of these surveys are conducted in person, or over the telephone. In line with the government’s digital by default strategy and as part of the Census and Data Collection Transformation Programme (CDCTP) , the ONS has been exploring online first data collection (also referred to as a push-to-web approach) alongside alternative sources of data, such as admin data.

A push-to-web approach to data collection means that respondents (referred to in this document as ‘users’) will be first asked to complete the survey online before any other mode is offered. If the user is unable to complete online, an alternative follow-up mode of data collection will be introduced such as face-to-face or telephone.

In the push-to-web approach the first touchpoint users will have with the ONS is through the materials they receive in the post. This means that compared to traditional data collection methods, additional reliance is put on the communications to convince people to take part.

Understanding who our users are

The users of ONS social surveys are potentially anyone living in the UK. Most of the ONS’ surveys are for people aged 16 years or over, but sometimes children are invited to take part too. It is important that the household surveys are representative and unbiased, to do this, addresses are sampled from a database, usually the Postcode Address File (PAF) or Address Base. This means that households are selected at random and nothing is known about the person or people who live there. Therefore, it is important that initial contact materials are inclusive, accessible and meet the needs of a range of different users.

Although the materials are an offline part of the respondent’s journey, they are asking user to complete an action online, if they are able. The UK Government accessibility regulations mean that all public sector organisations have a legal duty to make their websites accessible for everyone, including those with disabilities. We have extended our approach to accessibility to consider the end-to-end journey, including both online and offline touchpoints, and in collaboration with our graphic designers applied the principles of accessibility to printed products. The ONS Style Guide includes guidance on web accessibility.

We explain more about how we have designed our materials for accessibility later in this report.

What we need people to do

ONS needs households to take part in voluntary online surveys so that they can collect data that are used in the production of accredited official statistics. The aim is to get as many different people from sampled households across the UK to take part in these surveys to help produce quality statistics. Although users can complete on behalf of another person in their household (known as to ‘proxy’), the best outcome of a household survey is where each individual aged 16 years or over in the household completes their own section.

Unlike other GOV.UK services which a person might be obliged to engage with, such as obtaining a driver’s license or paying tax, all ONS social surveys are voluntary.
In practical terms, for a user to take part we need them to:

1. Open the envelope
2. Read the letter
3. Go to the website
4. Complete the survey
5. Return for a subsequent wave, if taking part in a longitudinal survey

This research explores the role of communications at and between all five steps and the different user journeys that might be involved in reaching the completion step.
This document outlines the several years of research and development carried out to understand user needs and create respondent communications that are integrated as part of the whole survey experience.

Given our wide target audience and feedback from ONS field interviewers, many people asked to complete an ONS survey are unlikely to consider it a priority. In acknowledging this potential lack of propensity to respond (and evidence of current response rates), we set out to explore different strategies to try and ‘nudge’ people into acting as soon as possible in their user journey. We aimed to learn about their motivations or barriers to taking part and came up with solutions to address them.

This report explains the qualitative research carried out to develop each of the respondent engagement products in the user journey. This includes the envelopes contact letters, between wave engagement and interviewer stationary used in mix-mode data collection. Each piece of material or set of materials is referred to in this document as a ‘product’. The research conducted for online respondent materials follows the three agile development phases:

• discovery
• alpha
• beta

This is the approach used in the development of products by the Government Digital Service (GDS).

Discovery

The purpose of a discovery is to understand the problem. In this research we have used a discovery to start understanding user needs and conducting conceptual exploration of topics. Typical discovery research activities include desk research, workshops staff, focus groups with the users and pop-up testing. These insights are used to inform the next stage of research and development; alpha.

Alpha

In the alpha phase, research involves testing out prototype products and ideas that have been developed using the insights gathered and identified in the discovery research. It allows you to explore some assumption you might have around the best way to meet using need. The alpha allows you to try things out and drop or re-iterate if you find they don’t work.

Beta

Product development in the beta phase involves taking the findings from alpha and rolling out a product into a practice live environment (private beta) and exploring whether the product meets user needs in the way that you expect. It allows you to assess how the product performs in this environment and identify where improvements may still need to be made.
The qualitative research conducted through discovery and alpha phases was used to developed prototype products to be trialled in large-scale quantitative tests as part of a private Beta.

The high-level results of these trials will be covered a in each product section, as well as links to the full reports. This report will begin by discussing the main contact letters.
Although these products have been developed for push-to-web survey, many of the learnings from our work and design features in final products can be applied to any mode.

Summary of findings and recommendations

The user centred design approach taken in this research revealed not only the consistent user needs, described throughout this report, but also important features that can help address these needs.

1. Branding: including official logos and any supporting branding, such as the Royal Coat of Arms adds authenticity and legitimacy to the letter, providing reassurance for user. We would advise thinking about how your users interact with your brand across their journey and how you can ensure consistency at different touchpoints.
2. Tone: in our research we explored different tones, from overly friendly to authoritative. For ONS, users expected the tone of our communications to be friendly but professional. This will depend upon the type of organisation you are and the image you already have, so it is worth exploring in context.
3. Icons: using iconography in the letters next to the text was found not only to make the letter more visually engaging but helped skim readers find the information they needed quickly. Icons also help guide users with low levels of literacy, breaking up the text and indicating what the adjoining paragraph is about. Although helpful, icons should only be used as support tool, and not to replace text to convey information.
4. Sub-headings: using sub-headings to divide up text helps users navigate the text and understand what to expect in each subsequent paragraph. This allows the user to assess whether they want to read the section or not.
5. Infographics: are useful when helping to explain the survey process. For example, the three step instructions for completing the online survey and the five-step diagram on our leaflet include both pictorial elements and text. In our research, infographics were found to engage users by making the process look simple and drew attention to the action required by the users.
6. Layout: in the research we found that ‘less is more’. Users told us that they would be put off by having too much information or being faced with a wall of text. Some users also mentioned a reluctance of turning the page or opening a leaflet. To minimise any potential friction in the users’ experience, we suggest limiting the content to what is needed by the user at each point in time. This will help create more white space, providing breathing room for the text.
7. Sign off: there was an expectation from our users that the letter is signed-off from a named person with a signature. This provides degree of authority and legitimacy to the letter.
8. Leaflet: when presented alongside a letter, the leaflet is considered as secondary material by users. It should be used for any supplementary information that has not been prioritised for the letter. The content should still be user-focused and assist in meeting the needs identified in section 11.

As a result of this research we have produced a suite of generic materials, these can be found on the report web page, alongside three letter templates that can be used as a starting point for creating your own letters.

Contact letters

Discovery

Due to the lack of an address register in the UK with population contact details, at first contact, the only way to initially communicate with people to take part in an online survey is by post. The letter therefore needs to be easily understood and persuade people to take part.

Note: After first contact we can resort to other mediums as we will have collected alternative contact details, such as email. These alternative approaches are integrated into our transformed communication strategy and are detailed later in this report.

Exploring and experimenting with contact strategies for push-to-web surveys is not a new concept. There are many past studies in the UK (Taylor and Lynn, 1998; Lynn, 2016), the USA (Bandilla, Couper, and Kaczmirek 2012; Dillman, 2011; Messer and Dillman, 2011; Dillman 2017; Kaplowitz, 2012) and Australia (personal correspondence from Australian Bureau of Statistics) that illustrate the role that communications have in influencing engagement with surveys.

However, each of the studies and strategies explored are context specific to the sample population, country, culture, organisation, survey topic and mode. Therefore, if possible, we always recommend conducting your own research to mitigate contextual risk.

Furthermore, the ONS’ position as an independent government department is unique to the UK and we know from internal research by our communications department engagement is likely to be influenced by audience perceptions and attitudes towards, politics and government and largescale data collection more generally.

The move towards push-to-web in the context of household surveys also presents new challenges (Couper, 2000). It is easier for a user to ignore a letter or exit a survey than it is to avoid answering the door to an interviewer at their home or asking them to leave mid-interview.

The Discovery phase of this work started in 2016 during ONS’ Electronic Data Collection programme (EDC) where initial contact letters were developed alongside the questionnaire. The letters went through multiple iterations of cognitive and user testing. This section summarises the research and insights from this Discovery work on those initial contact letters. The EDC insights were later used to inform the development of the later CDCTP communications work.

Main findings

The Discovery research involved focus groups and one-to-one interviews (reports available on request). In the Discovery research across several iterations of the materials, the main user needs identified for the initial contact letters were:

• understanding what the data is used for and by who
• knowing what to expect when completing the study and how long it would take
• making it clear that the study was being offered online
• clarifying which types of device the study could be completed on
• providing reassurance about data protection and confidentiality
• offering a telephone number to call for help
• a website address to complete the study
• clear login details and instructions

In addition to the main user needs, the following insights were obtained:

• navigation from the letter to the web should be as simple and straightforward as possible to enable those with low digital skills or other accessibility requirements to access the study independently
• URL and login codes should not contain ambiguous characters for example ‘l’ and ‘I’ or ‘O’ and ‘0’
• when text was bolded it stood out and should be used to highlight important messages
• short paragraphs of information rather than bigger paragraphs were easier to understand
• the inclusion of the ONS logo and Royal Coat of Arms stood out to users
• using colour in the letter design made it look more friendly
• quick reference (QR) code stood out but didn’t necessarily mean users would use it
• users felt that it was important to know that their contribution was valued
• using “Dear Sir/Madam” to address the reader was as expected (see Contact letters  – Alpha) for a later update on how we address the letters)
• the use of “Dear Resident” was considered spam or junk mail, and users would prefer the letter to be personalised with their name
• users felt that including signature on the letter made the letter more personal
• the purpose of the prenote was understood to be an introduction to the study and encouragement to take part

Following these high-level insights, there was also feedback on specific aspects of the prototype letter:

Headline “Play your part”:

• the words “play your part” made respondents feel that taking part could help and this was encouraging
• it also made users feel important and gave the letter a personal feel

Sub-heading “Your household has been selected”:

• this phrase had mixed reviews from users, including some negative comments relating to it sounding a bit “bogus” and “a scammy type thing”

Sub-heading “We value your contribution”:

• users felt positively towards this subheading because it promoted a feeling of having real input which was found to be encouraging
• users felt that this was important to include

The phrase “a few days from now”:

• users understood the sentence and expected to receive another letter
• it was important to users that context was given before reading this information about what would happen next

The phrase “Each year about half a million…”:

• initially, users felt that this was a lot of people, but then compared with the UK population seemed small
• the large sample for the survey indicated legitimacy and suggested to users that the results would be reliable
•  because half a million people take part some users felt that it was less special to be taking part and this took away the importance of their participation

The phrase “The ONS is the country’s largest provider…”:

• users felt that the short introduction to ONS was important to have at the beginning of the letter
• the reference to the Census was appreciated to help users understand what the ONS does

The insights gained from this Discovery work during the EDC programme informed later design phases and laid the foundation for the CDCTP materials product work. Some of the EDC wording was re-tested and re-developed in the later phases of this work to reaffirm the findings still held.

The ONS’ CDCTP approach to transformation follows the Government Digital Service (GDS) standards and principles which is underpinned by human-centred design. In this approach to design, the user’s needs direct the work and content. Insights are gathered through user testing which considers the whole end-to-end journey for the respondent. This joined up user centred way of working is a more holistic and inclusive way of considering survey design. It includes engagement contact strategies alongside the data collection instrument for different user groups and needs. All legal and ethical commitments are retained and met by the content.

Alpha

To invite users to take part in a study it is traditional to send an invitation letter. In face-to-face or telephone modes this is often referred to as the ‘advance letter’. In the online context, research has shown that for online surveys, prenotification letters can be beneficial (Dillman, Smyth, and Christian, 2014). A prenotification letter is a letter used to prepare users, sent ahead of the survey going live for data collection. In developing the suite of letters, these three products; prenotification, invite and reminder, were the primary focus.

The EDC Discovery insights prompted us in the CDCTP work to think more about the psychological tools that could be used to encourage response. Recognising recent successes of behavioural insights within policy and government, this is what we turned to next.

Behavioural insights research

Behavioural insights or ‘nudge theory’ is a psychological technique used to encourage behaviour change. Traditionally used in health interventions, nudges involve implementing small changes to try and produce a desired outcome of effect. In 2010 the Nudge Unit, later renamed to the Behavioural Insight Team (BIT), was set up by the Cabinet Office with the aim of applying behavioural economics to public policy. Successful trials carried out within UK government, for example HMRC sent tax letters stating, “most people pay their taxes on time”. This social norming nudge helped significantly increase payment rates (Halpern, 2015).

The framework used to develop these techniques or interventions known as ‘EAST’ which is based around the principles of make it: easy, attractive, social and timely.

We used this framework to think about how behavioural insights could be applied to online social survey letters. Below are some of the ideas from the behavioural insights literature that we suggested could be explored in the ONS survey context.

To apply the make it easy principle:

• ensure text is clear and simple
• avoid complicated or specialist language
• reduce volume of text and remove clutter
• people are more likely to believe statements written in bold

To apply the make it attractive principle:

• use the right messenger for the audience; the ‘messenger effect’ can add credibility and build in personalisation and reciprocity
• use the word ‘you’ to grab attention, promote personal responsibility and reduce diffusion of responsibility
• use the word ‘our’, for example, ‘our society’ to increase relevance
• frame the user’s decision to not respond to the initial ask as an oversight, allowing their participation at the next touchpoint to be perceived as an opportunity to do something positive (i.e. make it attractive) and their decision not to take part conversely, as unattractive
• loss aversion – frame chance to take part as something to be lost e.g. ‘don’t miss out on your chance to be counted’
• explore envelope colour, white envelopes with personalisation have been shown to be effective at improving engagement with mail
• give a reason for taking part i.e. ‘because’ increases compliance
• incentivise behaviour as a friendly thank you, not as a payment for doing the right thing
• make it salient by personalising or referring to local area

To apply the make it social principle:

• use descriptive social norming – what people actually do versus what they should do, for example ‘most people answer our surveys’
• diffusion of responsibility – focus on the importance of their selection and avoid mentioning sample sizes
• skewed estimates of prevalence influence behaviour in a self-reinforcing way – avoid specific figures in favour of terms like ‘most people’
• people can be influenced by behaviour of others who are similar or from same area, draw on this, rather than people in general
• encourage a ‘good mood’ to improve open mindedness – preface with an incentive (for example a gift) use pleasant graphics, photos and colours
• include a photo of the messenger to increase compliance; a photo can also increase reciprocity

To apply the make it timely principle:

• appeal to hyperbolic discounting by offering incentive up front and mention the time cost as soon as possible
• temporal decay – reward participation upfront (also making it attractive), describe how long the task will take and ask people to plan the time into their schedule later
• use the endowment effect, which means that when something is ‘yours’, it has greater worth than the same object belonging to someone else – use this to make the survey and its responses belong to the participant – ‘your responses’ makes them seem valuable

Focus groups with public

After desk research reviewing previous Discovery work and the behavioural insights literature, three prototype prenotification letters using different tones and techniques were developed and tested with members of the public in focus groups (see below). These focus groups were conducted in England, Wales and Scotland.

All letters were printed on the ONS social survey business as usual headed paper which is important because we wanted users to experience the letters as they would do if selected to take part. It also allowed us to gather feedback on the current branding. The footer is printed on white paper and with icons integrated into a turquoise banner including a heart to indicate health, piggy bank to indicate savings, a shopping trolley, car, mortarboard, nuclear family, aeroplane, wheelchair icon, in the same colour or navy blue (see Appendix 1 of the Word version of this report DOCX 2MB).

Alpha main findings

The letters developed and tested as part of this Alpha phase of research were for the online Labour Market Study (LMS). However, some learnings from this research are generic and could be applied to any online household survey.

The main high-level user needs that were identified from this research that users said they needed to know about in their decision-making process, include:

• what the study is about – to help users gauge relevance
• how long it would take them – so they could plan when to do it
• whether users would be “good at it” – to determine whether to take part or not
• how they could do it – mode options for participation
• the study deadline – to plan and avoid the study sounding indefinite.

All the letters developed for push-to-web social surveys at ONS aim to include this information at least once across the ONS-user touchpoints.

Alpha summary

The research conducted in the Alpha phase used qualitative methodologies and therefore gathered qualitative insights. One of the challenges of this type of data is trying to avoid research and acting on what people say they want or will do versus what they do when faced with the situation in a natural environment. For example, we didn’t pursue a photo on the letter because of the strong feelings it evoked with some users and associations with scam mail, which we wanted to avoid. Whereas feedback about having a potential unnecessary letter didn’t play on those same sentiments and negative emotions and was therefore explore further in beta testing as part of a larger-scale quantitative trial.

Private beta

Private beta: Quantitative test summary – contact strategy

The first private beta of the Labour Market Survey (LMS) was an online only quantitative uptake test of different engagement strategies to determine which was more effective and cost appropriate.

Using the insights gained from the qualitative testing the following conditions were trialled:

1. Prenote, invite, reminder
2. Invite plus reminder
3. Invite plus two reminders

The other conditions in this study included posting on different days of the week (Wednesday versus Friday), brown versus white envelopes and branding on envelopes (see Envelopes). No incentives were given to respondents during this trial.

To find out more about this private Beta experiment download the report authored by Ipsos MORI [DOCX 2.2MB].

The optimum strategy identified in this quantitative research was:

1. Invite plus reminder plus reminder = 21.4%
2. Prenote plus Invite plus Reminder = 20.2%
3. Invite plus Reminder = 17.9%

Although condition one produced a higher response rate, the addition of a prenotification letter led users to complete the online study sooner. Given the importance of timeliness in the collection of labour market data, this is the strategy that ONS has taken forward in subsequent private betas and the early launch of the live LMS beta.

At the ONS the graphic design team has grown since the work began, and with-it knowledge and expertise. In addition accessibility became a legal standard for digital services and as an end-to-end experience it was important the all parts of the user journey was joined up. In order to deliver quality products, it was necessary to work across disciplines, forming a multi-disciplinary project team.

The ONS print designers established their principles and helped us understand the main criteria for ensuring materials are accessible:

• contrast of colours
• font type and size
• alignment and styling of images
• creating white space
• writing in plain English
• providing access and options for alternative formats

We worked closely with the print designed to incorporate our research findings and develop the final suite of letters.

Summary: Letter research

The beta phases of research and development of these materials has shown us how to make our respondent communications more accessible. This was achieved through re-iterating, re-testing and re-validating our user needs and making decisions on what to keep and remove based evidence.

Here is a list of nine tips to consider when developing a respondent letter for a push-to-web survey:

1. seek to understand your users’ needs
2. test and re-iterate your design and content
3. only include information your users need
4. keep to one A4 page
5. identify the tone suitable for your organisation
6. use branding and official logos
7. break up the text and create white space
8. write in plain English
9. use minimum font size 12 in a legible font

For further advice, download the templates on this page.

Leaflet

Discovery

In our face to face surveys a leaflet has traditionally been sent alongside survey invitation letters to provide additional information for users. In the ONS Electronic Data Collection programme two types of leaflet were developed and tested with users.

Focus group and cognitive interview insights

The first leaflet tested was the ‘1 in 60 million’ leaflet, which included generic content as it was used across all the social surveys. The front page said, “you have been selected from 60,587,000 to help shape Britain today and tomorrow”. The inside of the leaflet included the title “Make sure your voice is heard…” and provided information for respondents about the ONS and why they had been selected. The second was an instructional ‘Getting Started’ leaflet, which explained how to log on to complete the online survey.

The ‘1 in 60 Million’ leaflet – main findings

On the front cover, users were engaged and liked the use of the people within the numbers. However, respondents questioned the message and associated it with the lottery and thought it was related more to a sales approach.

Feedback on the visual design and layout was as follows:

• users thought the leaflet had the opportunity to be more aesthetically and visually pleasing
• the inside was described as a “wall of text” and an “old fashioned way of presenting information”
• users thought the layout of the text was cramped
• users suggested taking the colour away to make it seem more official and there were association made with a glossy takeaway leaflet

Through the analysis, this research revealed users need to know:

• more information about ONS
• what will be done with statistics and why the study is important
• how they take part
• what the study is for
• why they should take part
• when they need to do it
• more about security and confidentiality information (for example use a padlock icon)
• about the process for completion and what their role will be (for example diagram of the process respondent goes through)
• about the stats we produce (for example, infographics, ‘did you know?’ type facts or provide current data on the topics of the survey)
• ”should you wish to see the results they will be available in X months time here”
• how to get to more information for example, include QR code taking respondent to website or social media
• a QR code for the Twitter and Facebook links

The recommended actions from this discovery research was to:

• remove the phrase “you have been selected from 60 million”
• review the title “make sure your voice is heard”
• review wording on first subheading – “Please take part…” to avoid sounding like pleading
• review the ‘why have you been chosen section’ which talks about address but not person, to include more information about importance of study and why the household has been selected
• amend the sentence “your address has been selected from the royal mail list of addresses…” to explain that this a publicly available resource and not just the government to reduce suspicion
• change the line “if you are not in when we call…” as does not apply to online completion
• review the language and use a warmer tone to explain why the respondent is important and cannot be replaced other than for sampling reasons
• review the mention of the history of the wartime survey
• at this stage the specific study is not explained, and the user will not know if they are taking part at this point, to address this, change: “If you have questions about the study you have been chosen for, go to our website www.ons.gov.uk and click on “Taking part in a Survey?” on the right-hand side. Or “ring us on 08000 298 5313” to a simpler “For more information, go to our website www.ons.gov.uk/surveys or give us a call” address the fact that at this stage the specific study is not fully explained.

The ‘Getting Started’ leaflet – main findings

(see Appendix 3 of the Word version of this report DOCX 2MB)

Icons and visual design

The follow observations we made about the visual design of this leaflet:

• users responded well to the front cover
• users noted the lack of consistency in the icons between the letters and the leaflet and said this affected the legitimacy of the process
• the fact the disabled icon was present on the letters but was missing from the leaflet was picked up
• the blue colour pallet was considered modern and formal

The recommendations following these insights were to:

• continue to use the green icons to ensure consistency with other survey branding (for example; the online questionnaire design banner)
• investigate possibility of including the disabled icon in the survey specific icons

Basic steps

This leaflet provides instructions for accessing the online survey, in the testing:

• it was suggested to change ‘Have you got your letter, this contains’ to ‘..it contains..’ or the letter you have received contains..’ or ‘inside you’ll find’
• there was some confusion over the use of ‘XXX – XXX …’ to indicate the enrolment code in Step 1
  • instead of a dummy code, ‘XXX etc’ was used in the testing to avoid users mistaking an illustrative alpha numeric code for their login details
• users liked the basic steps but thought it was unclear about what they should do when they had entered the questionnaire
• users that were not used to using computers said it gave them the confidence to go online and follow the steps to log on to it
• the size of the images was observed to not be big enough for respondents to discern the content clearly

The recommendations following these insights were to:

• change the text in Step 1 to make it clearer
• change the design of the example enrolment code
• provide some further guidance and investigate possibly including a next step explaining what they do when code is entered
• reduce the green space in the margins by having Steps 1, 2, 3 to the left rather than above the detail allowing the images to be enlarged

QR Code and Frequently Asked Questions (FAQ)

This leaflet included a QR code, in testing:

• users noticed the QR code on the reverse but were unclear where it would go
• the use of the QR code in Step 2 was suggested for those using a mobile device
• users suggested having a link to the outputs of the study and when the results would be available
• users suggested including information on what they should do if they do not have access to the internet

The recommendations following this testing were to:

• explore if the ‘getting started’ leaflet would be questionnaire specific and if so, continue to use colours and icons and include a link to the outputs and possible include some statistics
• include information in the FAQ on reverse of leaflet to address what they should do if they do not have internet access

Note: At the time this research was done (around 2014) a recommendation was made to continue to use the QR code within the basic steps if questionnaire compatible with tablet. More recent research (around 2020) however, has identified that whilst QR codes and the technology that support their use have become more common in recent years, they should be used with caution. Qualitative insights suggest they should be used only to support and not in replace of any crucial information. Users expect to have all the important information in the letter and do not want to have to take an extra step to find it.

Following the testing of the two leaflets, they were re-iterated and taken out for further user testing. The ‘1 in 60 million’ leaflet became the ‘Your household has been chosen’ leaflet, adopting a more informal approach. Although the ‘1 in 60 million’ concept worked when it was originally introduced, it had become outdated and didn’t fit with expectations of government branding. For the ‘Your Household has been chosen’ leaflet the amount of information was reduced, infographics were included, and a greater amount of colour used.

The ‘Your household has been chosen’ leaflet – main findings

Initial observations:

• users usually read the leaflet straight after the letter
• some users left the leaflet in envelope when they pulled out the letter and needed prompting to find it during probing about the letter
• the information in the leaflet provided answers for many of the questions that users had after reading the letter, about why they had been selected
• users were drawn to the leaflet because it was brightly coloured and was “pretty eye catching”, and some users first noticed and commented on the colour of the leaflet
• other features that were salient included the box at the bottom of the page that included reference to statistical releases

Front cover observations:

• the title was considered appropriate for the leaflet and sufficiently explained what would be found in the leaflet
• the word ‘chosen’ made users feel special. It was also mentioned that the word chosen made the respondent “want to get involved”
• users favoured the use of ‘chosen’ over ‘selected’, because “chosen has many positive connotations” and “implies you’ve done something good”
• it was suggested that ‘selected’ seemed more formal whereas chosen was a “a bit softer”
• the title reminded users of junk mail that says you have won something, and ‘your household has been chosen’ was compared with the phrase ‘you’re the lucky winner’
• the title of the leaflet was also considered threatening and was suggested that we add: “…to take part in a survey” to the title to clarify what the user has been chosen for
• users thought the subheading ‘here is what you need to know’ was clear and informative: “it’s very informal but very clear I know that when I open this it is going to talk me through what is going on, which is great”

Headings feedback:

• users were attracted to the headings within the leaflet and stated that they were all clear and easily understood
• users thought the headings covered all of the main points and were presented in a good order
• use of acronym in ‘who are the ONS’ was not liked by some users

Infographic feedback:

• users responded positively toward use of the image and thought it was better than using just text, as it “makes it less boring” and is helpful
• there was some confusion about what the infographic was supposed to be representing
• it was not clear what the first part of the infographic was showing and the faint icons behind the figure were difficult to see and often unnoticed
• it was suggested that an image of what the survey would look like could be included in the section about confidentiality
• users understood the images on the right as different household types that the ONS talk to for their studies

Feedback on the phrase “why has your household been selected?”:

• most users understood that their household had been chosen randomly
• ‘at random’ was not always noticed and was suggested that we change ‘why has your household been selected’ to ‘your household has been selected at random’ to make it clearer
• users commented that this section was meaningless and did not fully explain why they couldn’t use another household close to theirs if they were not willing to take part, questioning “what difference does it make?”
• the reference to the Royal Mail was thought to be misleading when attempting to understand the sentence about not being able to replace their household
• some users would like more information on why they had been selected, and the Royal Mail section was referred to, however this would not largely affect respondent’s decision to take part
• it was mentioned that the size of the list that users were selected from should be stated
• there was also a query about the age for participation

Feedback on who is ONS and what do they do:

• most users knew who the ONS were and said they were satisfied with the amount of information that was provided
• if they wanted to find out more information users said they would use the website or telephone number
• users liked the mention of the weekly shopping basket in this section of information

Feedback on descriptions of how data is used:

• some users had a clear understanding of who the data was being used by, mentioning government, charities, and members of the public
• some users were unsure exactly how their information was being used
• users mentioned that if the information was helping people, the community and services this would encourage participation
• however, not all users were concerned, and thought they had enough information: “it doesn’t actually tell me what you’re going to do with it” but the user was not bothered by this

Feedback on the use of the phrase “we are not a commercial organisation”:

• this information was often missed when the leaflet was read
• users suggested that the phrase was missed due to its position in the fold of the leaflet and because the text was too small
• this information was considered essential as without this knowledge users may think information was being provided to political parties and may be discouraged from taking part
• it was suggested that this information should be made more prominent and could be part of the main paragraph ‘who are the ONS?’
• other users thought the sentence should remain in the sub-box, but the box should make up a whole line rather than two thirds of the space

Feedback on the phrase “your answers will be treated confidentially…”:

• there were mixed views over the importance of confidentiality and how much information should be provided
• users were generally reassured by the information about confidentiality and thought no further content should be added
  • this included some of those that were most concerned about the protection of their data, but not all
• there were concerns that users’ data would be sold on to other companies, specifically sharing with third party marketing organisations
• there were positive responses to the use of bold text and the padlock icon
  • these features drew user’s attention to the area and made them more willing to take part

Feedback on the phrase “our statistical releases are available…”:

• users were happy with the information provided about statistical releases and thought it was “useful to know”
• some users were surprised to learn that they could access the statistics online and said it would be interesting to see, although there was uncertainty about what they would find online
• this sentence was missed by some users and was noticed on the back of the leaflet before it was seen at the bottom of the page inside

Feedback on the back page of the leaflet:

• users liked the information about how their data is used and were surprised to find out that it wasn’t only the government that uses the data but charities and universities too
• some users mentioned that they liked the use of the informal tone
• some users disliked certain parts of the sentence that referred to being ‘rich’ or poor’ saying that is was “quite offensive” and could be considered “slightly depressing”
• a lack of ethnic diversity was picked up in terms of representation and should be addressed
• another change suggested was that ‘living up north’ could be changed to ‘living in the north instead’, or thought that maybe it should say north, east, south or west
• users found it helpful to include the contact details and opening times on the back
• it was suggested that it should be mentioned that the enquiry line is a free number

Views on visual design:

• users liked the colour used in the leaflet, commenting that it “catches your eye”
• users also mentioned the contrast of the text on the white background
• users commented that as leaflet was split into chunks it made it easy to read
• users were happy with the icons for Twitter and Facebook and understood what these were showing, although not everybody was prepared to use these websites

Length and amount of materials:

• as there is limited space to provide information, a decision should be made on what respondents should be most aware of, such as how to complete the survey and the return date
• the most important information should be highlighted, bolding text where necessary
• large chunks of text should be avoided and make use of infographic to improve the visual appeal

Alpha

The next steps for developing the leaflet for online surveys came alongside the work developing the contact letters as part of the DCTP (see Contact letter – private beta). Using the insights gained in the prior discovery work for the EDC programme, iterations we made to the leaflet and further testing was conducted.

Beta

GDPR requirements

In May 2018 the General Data Protection Regulations (GDPR) were introduced. This meant updating respondent materials with the relevant information.
Whilst ensuring we met the requirements of GDPR by including compulsory information to satisfy business needs, we needed to ensure that the new content was clear and understood by users too.

The changes to the leaflet were minor adjustments to the confidentiality statement, signposting users to the Data Protection Officer (DPO) and the Information Commissioner (ICO). All this information is presented together on the back of the A5 ‘What you need to know’ leaflet.

One-to-one cognitive testing was carried out on the leaflet and the main findings were:

• the confidentiality statement was scanned, rather than read line by line
• users felt reassured by the presence of the confidentiality statement
• the lock icon was interpreted to mean the information was going to be kept secure
• the confidentiality statement was considered clear and easy to understand
• users understood their data would be anonymised and used for statistical purposes
• general trust in how their data will be used because of ONS being an official organisation
• users understood that they could contact the DPO if they had further queries

The leaflet in isolation does not meet GDPR requirements there are a combination of factors across the suite of materials which ensure our compliance.

Envelopes

Discovery

As outlined in the introduction (section 1.0) of this report one of the main barriers to engaging users with an online survey is getting them to open the envelope. As this is the first step in the user’s journey, we wanted to explore what needs users had at this stage and what we could potentially do to encourage response. To start with we looked back at what had been done in previous ONS survey work.

Insights from Electronic Data Collection (EDC) Programme

The envelope used in the EDC testing was a plain white generic envelope except for the Royal Coat of Arms and the words “On her Majesty’s Service”. They were printed in black across the top, and the ONS logo in the bottom right hand corner. In testing it was found that these official markers made the letter look important and encouraged respondents to open the envelopes.

Other discovery activity included looking at the testing conducted on previous Census envelope designs.

In this research the findings showed that:

• images, particularly maps were favoured by respondents
• users responded positively towards the idea of being ‘counted’ because it makes them feel included
• some users were put off by forceful language such as ‘act now’
• users didn’t always know what ‘ONS’ meant or ‘Census’
• users in Wales responded positively to regional designs

On one of the envelopes the phrase ‘Act now – login inside’ with an image of a post-it note with ‘Census Day! 21st March’ was included in the front bottom right hand corner. On this, users felt the tone was a bit forceful and the inclusion of a post-it note made it look unofficial.

Another envelope included the environment nudge “Save money, save time, save the environment… do your Census online”’. Users feedback that the call to action was too long winded and something snappier would be better. The use of the saving the environment was not universally appealing to users and it was suggested that saving money or helping the community would provide more motivation.

Using these insights in combination with the desk research, further testing was conducted as part of the alpha phase.

Alpha

Pop-up testing

Pop-up (also known as Guerrilla testing) was used to gather feedback on several prototype envelope designs. The aim of this research was to gather people’s reactions to the different envelopes. Users were asked ‘which envelope jumps out at you?’ and the researcher explored the reasons why. Further follow up questions were asked about envelope for example colour, design, layout and how the presence of the Royal Coat of Arms made them feel. The way the envelopes were presented alternated to provide a counterbalance.

Four types of envelope were included in the testing:

1. plain envelope
2. various regionally branded envelopes for England, Wales and Scotland
3. a ‘word jumble’ envelope containing survey topics
4. a call to action ‘play your part’ envelope

Pop-up testing helped identify which envelopes to take forward and which to drop. For example. the map of Wales was dropped from further testing because it was found that only people born in Wales recognised it. However, the evidence showed that there was merit in regional branding so other options were later explored.

Focus group testing

Using the pop-up testing insights, envelopes were further tested in focus groups. For brevity the feedback across two iterations on each element tested, is summarised.

Private beta

Quantitative test summary – envelope testing

The first private beta (in this instance a large-scale quantitative experiment with using prototype materials) of the Labour Market Survey (LMS) was an online only uptake test of different engagement strategies. Two elements of the trial involved envelopes; these tested:

1. White versus brown (manila) envelopes
2. Regionally branded envelopes versus non-branded envelopes

Using the insights obtained in the qualitative testing, both white and brown envelopes sent out across England contained:

• the Royal Coat of Arms logo
• the phrase ‘On Her Majesty’s Service
• a slogan at the bottom-right hand corner saying “Play your part in shaping the future of the UK”
• the ONS return address was printed on the back of the envelope

In Wales and Scotland the envelopes adopted regional branding in addition to ‘OHMS’ and the Royal Coat of Arms. In Wales, the slogan was ‘Wales, make sure you are counted’ (in English and in Welsh) and included a greyscale image of a dragon. In Scotland the envelope said ‘Scotland, make sure you are counted’ and contained a printed picture of a map.
The other conditions in this study included posting on different days of the week Wednesday versus Friday, and different types and number of contact letters (see Contact letters). No incentives were given out during this trial.

Conclusion

The results of the envelope trial showed that brown envelopes achieved a better engagement rate with the survey, but the difference compared to the white envelope was not statistically significant. In terms of regional branding, the Welsh envelope appeared to improve response but the results were not statistically significant. The same was not seen for Scotland, although it’s worth noting this experiment was conducted around the time of the Scottish referendum.

To find out more about this private beta experiment download the  report authored by Ipsos MORI [DOCX 2.2KB].

Interviewer calling cards

Discovery

The LMS Statistical Test was a mix-mode (online with face-to-face follow up) study requiring interviewer stationary. In the event that the user didn’t respond to the invitation to take part or follow up reminders or if they could not take part online they would be followed up by an in-person interviewer visit. If the door is not answered, interviewers will often leave a calling card to show that they have been tried to make contact and offers away for the respondent to get in touch. To develop the calling cards for this mix-mode test we first sought to understand the current processes.

Desk review

To develop these materials, the first step was to understand what items interviewers currently use on the doorstep in the face-to-face surveys and understand which items were most useful. This involved looking at metrics from our operations division to evaluate which items to focus on. These were found to be:

• called today cards
• appointment cards
• broken appointment cards
• impediment to entry letters

These items were reviewed to establish their role in the user journey and explore how we could bring them in line with the letters and leaflets to ensure a consistent look and feel across the suite of materials received by potential respondents.

We also looked at research by the Behavioural Insights Team (BIT) to see how we could apply learning from the Census 2017 test which trialled different types of nudges on two different

• calling cards – these applied the concepts of:
• implementation intentions – the cards asked respondents to tick when they would complete their Census Test questionnaire. Research shows that people are more likely to complete an action when they have planned to do so.
• local – the cards referred to their local census officer
• monitoring – the cards said that we noticed that they hadn’t filled in their questionnaire, indicating that they are being monitored
• reciprocity – the card said “we designed this reminder to help you out” invoking a sense of reciprocity

The second card made use of the above and the additional behavioural nudge:

• interviewer personalisation – to personalise the card, the field worker had space to sign their own name and referred to the individual as ‘neighbour’.

The two cards designed by BIT were trialled, using the existing ONS calling card as the control group in the 2017 Census Test. The results showed that both BIT calling cards outperformed the control, but the addition of the interviewer personalisation did not significantly differ from the one employing only the implementation intention. For more information about this trial contact the team at behavioural_insights@ons.gov.uk.

These learnings were applied to create a ‘nudge’ calling card as part of an operational test for the LMS. The test was only conducted with small sample, but findings indicate that they helped improve engagement with survey by approximately 10% compared to those who did not receive a nudge calling card (no statistical testing was applied due to small sample size).

In addition to the nudge calling card, traditional calling cards were developed with interviewers and members of the public.

Interviewer workshops

Users of the calling cards include interviewers as well as the recipient (respondent), so it was important that feedback was obtained from both types of user. Research with interviewers on this product sought to find out about current behaviours, for example how and when they use the each of the available calling cards. It also provided an opportunity to get feedback on the current cards to see what ideas and suggestions they had for any potential improvements. We carried out two research activities, firstly a face-to-face workshop with interviewers and secondly, we circulated the calling cards via interviewer managers to collate feedback via correspondence.

In this research, we found:

• interviewers don’t necessarily all use the cards in the same way, with the same frequency or at the same point in time
• not all interviewers include the time on the card they leave
• there was an appreciation for wanting to standardise as much as the card as possible to minimise the amount of ‘write-in’s’ required on each card in order to save time and provide consistency.
• use of the word ‘study’ was preferred over ‘survey’ to convey importance
• interviewers need the card to be clear it’s not a cold call
• users responded positively to and recommended a polite and friendly tone, over more authoritative approach
• that the ‘Sorry I missed you’ was deemed appropriate  for the called today card, with mention of interviewers using that already or writing it on the current card.

The actions taken away on the prototype cards we shared with interviewers included:

• creating more space for interviewer’s details
• adding space for day of the week on the appointment card
• linking to ONS website to provide legitimacy
• simplifying some of the language used
• ensuring consistency across layout and design of different components across the three cards

The next step was to test these cards with the public.

Alpha

Public perception testing

After re-iterating the designs for the calling cards based on user feedback from an interviewer perspective, we then took them out for further testing with members of the public. It is important that the calling cards also meet the needs of the public and are not perceived negatively. The findings of this round of testing is summarised below.

Main insights from the ‘called today’ card (see Appendix 5 of the Word version of this report DOCX 2MB):

• users didn’t like the thought of being approached in person to take part as they would feel under pressure
• users were surprised that they would be followed up face-to-face if they had not responded to the online survey invitation
• users understood why they were receiving the card
• users said they would call or recognised they should phone the interviewer either to arrange an appointment or to opt out
• users expect that they could still do it online at this point if they’d prefer
• users demonstrated a mixture of preference over phoning and going on the website for further information
• users did not always appear to notice the website straight away, even if it would be their preferred mode of finding out more.
• users felt reassured by the ability to call up and verify the interviewer identity

Main insights from the ‘Your appointment’ card (see Appendix 6 of the Word version of this report DOCX 2MB):

• the appointment card met users’ expectations
• users found comfort in the fact that the option to call and re-arrange or cancel is included
• some users mentioned that it was helpful as they would put it on their fridge as a reminder
• other users said that they would prefer something electronic
• there was a need to know how long the appointment would take by a handful of participants
• fewer mentioned about being given a heads up of what they might need to prepare or have available whilst doing the survey
• users understood that the appointment would take place as scheduled and the interviewer would come at the agreed time (unless they had called to cancel or re-arrange)
• there was some expectation from users that a reminder message would be received the day before, for example a text or email
• users said that they would call the interviewer to tell them if they could no longer make the appointment
• there was mention of sending a text or email, but it was clear that the card was asking them to phone the interviewer

Main insights from the ‘broken appointment’ card (see Appendix 7 of the Word version of this report DOCX 2MB):

• users understood why they received the card and felt the onus would be on them to call the interviewer to re-arrange, apologise or cancel
• in the ‘I’ll try again’ box users expected it to be filled in with another date and time and if they couldn’t make it, they would call to say so and re-arrange
• however other users said they would prefer for it to be on them to initiate contact if they’d missed their appointment, rather than have an appointment re-arranged without their consultation
• users were happy with the tone of the card; it didn’t appear to make them feel bad for missing the appointment and users empathised that the interviewer would probably be feeling frustrated by this point

Interviewer Calling Card: Alpha

A version of these cards was also used in the LMS statistical test by our delivery partner. The cards were adapted so that they were dual branded with the third-party contact details and logos included.

Calling cards summary

The calling cards developed for the met the needs of both field interviewers and the public. The next step is to trial the calling cards as part of a live Beta to asses their effectiveness at a larger scale with ONS interviewers and seek to identify any improvements to be made before rolling out.

Between wave engagement (BWE)

Discovery

One of the main challenges with voluntary longitudinal surveys is retaining sample and minimising attrition (drop off). One strategy which can potentially help with this is to engage with users in-between the waves of the survey to keep their interest (Cleary and Balmer 2015; Lynn, 2017). However, it is important that the content adds value for the respondent and meets users’ needs and expectations. To develop this content a series of research activity was conducted.

First, questions were added to the current business as usual LFS survey to establish what users might want to receive, if anything. We also spoke to previous ONS social survey respondents to find out their thoughts on engaging with them in-between waves of the survey. Following the insights from both these activities and some desk research, prototypes were developed and tested over a series of iterations in focus groups and then as part of end-to-end cognitive testing.

The findings from each of these activities will be discussed in turn.

Dress rehearsal BWE questions of the business as usual Labour Force Survey (LFS)

In the 2017 Labour Force Survey (LFS) dress rehearsal, respondents were provided with information about the survey and asked questions about being followed up between waves.

The results of the survey found that of the total sample of 730, 49% said ‘no’ to being followed up, 30% said ‘don’t know’ and 21% said ‘yes’.

Of those who said ‘yes’ (n=149) in answer to an open text question, most people were keen to hear about: findings from the study, use of their information and when the study had contributed to news headlines.

Respondents were also asked about the specific items they would be interesting in receiving.

Options included:

• a thank you
• a reminder about the time of the next interview
• some findings from the survey
• how the information is being used
• a way to tell us about moving house
• newspaper headlines of things we have done
• information tailored to age group

112 wanted to receive findings and 108 how the information is being used (see figure 1). Over half of respondents said they would prefer this information via email with post being the second most popular choice (see figure 2).

Figure 1:  Items respondents would be most interested in receiving between waves

Click on the image to view full screen

Figure 2:  Preferred communication methods between waves

Click on the image to view full screen

Of the people who said ‘no’ to being contacted (n=360), 148 said they were not interested, 59 cited too much post and 176 suggested other reasons. The main themes were that it was unnecessary and that they didn’t have time or were too busy.

A full summary of the results is available on request.

Discovery summary

In the discovery phase of this research there were clear themes identified. The insights suggest that communication about study results and how the data is used is the most appropriate content to provide users, rather than promotional items that were not meaningful. Email and post were considered suitable forms of contact and what users might expect from the ONS.
In the next stage of the BWE development these insights were taken forward, concepts and content were tested, and products were prototyped.

Alpha

The research objectives of the alpha phase of research were to find out:

• what information users would find most engaging
• establish the best ways of presenting the data
• explore the best medium for disseminating the information

The qualitative insights would be used to inform a trial containing three experimental conditions exploring the impact of an offline and digital BWE against a control group, which would receive no BWE.

Concept development and testing – public focus groups

The content developed and tested in focus groups and one-to-one interviews was based on data from the current Labour Force Survey. Labour market data was used as this is what makes up the core of the online LMS. The concepts of employment, unemployment and those not looking for work were broken down into different categories to show how user’s data are used in the production of statistics.

In developing the prototypes some considerations that we made include:

• accessibility of content for online and offline
• content should be the same across email and postcard
• timeliness of data release
• no ability to personalise email

In the focus group several different concepts were explored, using a postcard as the medium to display the information. As part of the exercise, users were asked to rank the different ways of presenting the data on a scale. The first scale they were given was around clarity – with ‘clear’ one end and ‘unclear’ the other, along a horizontal line. An additional vertical line was added to this, to create an axis. The vertical scale was labelled ‘interesting’ and ‘not interesting’. In groups, users were asked to sort the different types of data presentation and feedback where and why they placed each one on the axis.

The findings from this research are summarised below:

Summary of public focus group

The research showed us that often the simple ideas work best, such as representing the UK at just 100 people. In the next stage of one-to-one testing, we took forward this concept and alongside the idea of regionalising the statistics.

[Tab title=”Front of the postcard: ‘Shape tomorrow’ versus ‘Thank you’” tag=”h5″]

In the first iteration of the postcard prototype we recycled the ‘Shape tomorrow’ branding from the tote bag incentive on the front of the postcard. However, this did not work in this context. Users felt that they would have already shaped tomorrow by taking part in the survey and saw it as a call to action. Although users found the style of the ‘Shape tomorrow’ engaging, particularly the images that represented parts of everyday life.

The comments on the alternative postcard front which simply read ‘thank you’ were that it was more appropriate and enabled them to connect the dots between what they had done, i.e. the survey, and the postcard they were now receiving.

Incorporating feedback from both designs, a ‘thank you’ was developed in the same style as the ‘shape tomorrow’. This allowed us to make the communication more engaging for users and join up the branding between different touch points in the users journey over multiple waves.

[/tab]

Between wave engagement (two)

The next stage of development involved testing content for a second between wave engagement, continuing the work conducted for wave one and exploring regionalisation of content further.

Findings suggested that people in London wanted the national picture, whereas users in different parts of the country, for example the West Midlands, Wales and Scotland preferred hearing about statistics more local to them, even if at the country level.

Summary

The qualitative research conducted through the alpha phase of develop, informed what products were tested in the quantitative LMS trial that explored attrition between waves. The results of which are summarised in the next section.

Live Alpha: Quantitative test summary – attrition test strategy

The LMS attrition test was online only and consisted of a three-wave experiment exploring the impact of a communication between waves of the study.

In the trial design respondents were allocated at random to receive either a postcard, an email or no between wave engagement after taking part in each wave of the study. In addition to being allocated to one of two incentive groups (£5 unconditional voucher or no incentive).

The uptake rate at wave one was 29% (25% completion rate). At Wave two we saw 62.5% of households return to the survey to complete (56.4% completing the whole survey). The breakdown of the materials groups with no incentive at wave two was as follows:

• 58.5% completed who received a postal communication
• 62.6% completed who received no communications between wave
• 66.4% completed who received an email between wave

In the analysis, the email group was found the be significantly more effective at retaining respondents than the post and no communications group at wave two.

At wave three we saw an overall uptake rate of 73.8% with a 69.9% completion rate. The breakdown of the materials groups with no incentive at wave two was as follows:

• 70.2% completed who received a postal communication
• 70.2% completed who received no communications between wave
• 74.0% completed who received an email between wave

The analysis shows that email was significantly more effective than post at retaining respondents in wave three.

The final versions of the between wave engagement communications rolled out and a full write up of the results of the study are available on request.

Helpful tools and guidance

Designing products for everyone is an important principle underpinning this work to ensure as many people as possible who are selected to take part can. There are several tools we recommend to help assess the accessibility of your materials.

Hemingway app

The Hemmingway app asses the reading level of text and makes suggestions on where content could be improved to achieve the recommended reading age of 9 years old which is considered best practice in content design.

Plain English Campaign

The Plain English Campaign offers an A to Z of alternative words [PDF 174KB] and guides for writing in plain English.

Collins English Dictionary

Collins English Dictionary allows you to search a word to see how frequently is used. The higher the frequency, the more familiar it is likely to be.

Google Trends

Google Trends allows you to explore and compare different search terms, this can help with deciding the best way to refer to something in a way that is recognised.

Seeing AI

Seeing AI is a multi-purpose tool, a recommended feature includes the ability take a photograph a document which can be converted into a file that can be then read out loud.

Accessibility posters

Home Office Accessibility Posters offer guidance on do’s and don’ts when designing for accessibility.

Simulation glasses

Simulation glasses are physical tools which attempt to mimic the effects of visual impairments. They can be used as a basic and initial assessment of how your product may be viewed by users with different visual impairments. However, every individuals experience will be unique to them.

These tools shouldn’t replace getting feedback from real users. There are professional centres that offer accessibility review of products such as the Digital Accessibility Centre.

Many government departments now have user research teams and content designers embedded within the development of services, so reach out to them if you need support in this area. Or visit user research pages on GOV.UK.

User needs

The research detailed in this report has consistently identified a selection of user needs over the course of developing these respondent materials. These needs are summarised below as user stories. These stories may be helpful to recycle for developing respondent communications other push-to-web studies.

Users stories help frame the need from the point of view of the users you are designing for. You can read more about creating your own user stories in the GDS service manual.

User stories

As a potential survey respondent, I need to know:

• what I’m being asked to do so that I can decide whether I want to take part
• why I should do it so that I understand the importance of taking part
• who in my household needs to take part so that I can complete it on their behalf or ask them to complete their own section of the study
• what is in it for me so that I can decide if it’s worth my time
• how long it will take so that I can plan an appropriate time for me to complete the study
• how I can take part so that I have the right equipment
• when I need to do it by so that I can make time to complete the study before the deadline
• that my data will be treated confidentially and handled securely so that I trust my data will not be misused
• how I can find out more information so that I can make an informed decision about whether I want to take part
• what happens next so that I know what to look out for or what action I need to take

We have met these needs through the design and content of our letters. Our user-centred design approach put users at the heart of this, but we also need to consider the business needs, such as GDPR requirements.

References

1. Bandilla, W, Couper, MP and Kaczmirek, L ‘The mode of invitation for web surveys’, Survey Practice 2012: Issue 5, Volume 3, pages 1-5.
2. Cleary A, Balmer N. ‘The Impact of Between-Wave Engagement Strategies on Response to a Longitudinal Survey’, International Journal of Market Research 2015: Issue 57, Volume 4, pages 533-554.
3. Couper MP ‘Web surveys: A review of issues and approaches’, Public Opinion Quarterly 2000: Issue 64, Volume 4, pages 464-494.
4. Dillman DA, Smyth JD, and Christian LM ‘Internet, phone, mail, and mixed-mode surveys: the tailored design method’, 2014, John Wiley and Sons.
5. Dillman DA. ‘Mail and Internet surveys: The tailored design method (second edition)’ 2017, John Wiley and Sons.
6. Dillman DA. ‘The promise and challenge of pushing respondents to the web in mixed-mode surveys’ Statistics Canada, 2017.
7. Kaplowitz MD, Lupi F, Couper MP, Thorp L ‘The effect of invitation design on web survey response rates’, Social Science Computer Review 2012: Issue 30, Volume 3 pages 339-349.
8. Lynn, P ‘Targeted appeals for participation in letters to panel survey members’, Public Opinion Quarterly 2016: Issue 80, Volume 3, pages 771-782.
9. Lynn P. ‘From standardised to targeted survey procedures for tackling non-response and attrition.’ Survey Research Methods 2017: Volume 11, Number 1, pages 93-103.
10. Messer BL, Dillman DA. ‘Surveying the general public over the internet using address-based sampling and mail contact procedures’, Public Opinion Quarterly 2011: Volume 75, Issue 3, pages 429-457.
11. Taylor, S and Lynn, P ‘The Effect of Preliminary Notification Letter on Response to a Postal Survey of Young People’, Market Research Society Journal 1998: Issue 40 Volume 2, pages1-11.