Introduction
When designing survey questions we advise carrying out cognitive interviewing and usability testing.
Cognitive interviewing is a method used to explore the ways in which respondents mentally process and respond to survey questionnaires. Cognitive interviewing can highlight shortfalls within a questionnaire such as missing response options, incorrect terminology and incorrect routing for respondents.
In usability testing a researcher will ask a participant to perform tasks while the researcher observes and records the participant’s behaviour (such as frustration during completion). The researcher will also listen out for any feedback. It can be used to explore problems or confusion when respondents are completing a survey.
Both cognitive and usability testing are normally done in a face to face setting. However, due to researchers currently being advised not to meet participants face to face, we have had to think how this testing can be adapted to work remotely.
This guidance aims to help researchers adapt some of their methods to remote working.
What we mean by 'remote research'
Remote research is a method of research where the research participant and the researcher do not interact in person. It is usually conducted via a computer or telephone.
A downside of remote research is that it is more difficult to observe non-verbal cues and body language, which can form a valuable part of the picture in face to face research.
However, remote research is better than no research at all. It also provides us with the opportunity to research with participants in a variety of diverse geographical locations.
There are two main types of remote research: moderated and unmoderated. Ultimately, as with any research method there are pros and cons to both types
Unmoderated remote research
This type of remote research does not require a researcher to attend each research session.
One way of conducting unmoderated remote research is using a software application which provides instructions to participants, records their actions and may ask them predetermined generic follow up questions.
Participants can complete the task or answer the questions at a time they want, at their own pace and in the location they choose.
Pros
- It can allow you to capture large amounts of data in a short space of time by inviting many more participants to participate simultaneously without the need for a facilitator.
Cons
Issues related to the fact that unmoderated sessions are unsupervised and the task is set beforehand:
- the participant is unable to ask follow up questions – this could result in participants who need assistance feeling excluded
- respondents dropping out if they don’t have a researcher to assist with issues – this could cause response rates to fall
- researchers not knowing how a session has run until it’s over – e.g. participants may have skipped tasks or completed the wrong task
- little flexibility in getting the participant to elaborate on any comments or interesting points they make during the session
Also, in general, unmoderated research sessions tend to be shorter than moderated sessions which can limit the amount of rich data you can gather from each session.
Moderated remote research
In a remote moderated session, the researcher is present and able to observe participants complete tasks while communicating with them in real time using web conferencing and screensharing technology.
Participants are encouraged to think out loud as they work through tasks and provide immediate feedback on their understanding of the processes they are going through.
Pros
- Moderated research sessions can be longer than unmoderated sessions. This allows for in-depth exploration of topics or questions and therefore richer data.
- As the researcher is present, they can ask follow up questions and spontaneously probe if needed.
- Having a researcher present can be more helpful for participants who come up against technology issues.
- Moderated testing is nearer the face to face method than unmoderated, so it is the best option for conducting cognitive interviewing remotely.
Cons
- If you are using some of the more advanced technology that allows for screensharing or video conferencing, then you may be introducing bias to your research as it relies on participants having a certain level of digital ability.
Choosing between moderated and unmoderated
When deciding on your method of remote research think about your aims and the sample. To ensure that you have comprehensively met all of the sampling criteria, a mixed approach could be suitable. For example, if you were carrying out research on groups that may need more assistance then you may consider gathering in-depth feedback from moderated research in the first instance, before moving onto un-moderated for the next iteration.
Top tips for running a remote research
Once you have decided on the type of remote research to carry out, here are some top tips for running the sessions:
- Planning is everything.
Write your task beforehand and think about how you are going to present the participant with the task. In a normal face to face setting you could perhaps show them a link or provide them with a piece of paper but think about how you will do this remotely.
- Test run technology
Have a test run of technology with the participant beforehand. This will ensure any audio or screensharing issues are ironed out before the actual test.
- Run a mock session
Doing this beforehand helps you practice probes and tasks and lets you see how the session will run.
- Consider your consent process.
If you would normally have a participant sign a consent form then you could send this to them beforehand. Alternatively, if you are happy with verbal consent then you could cover this in the session and record it.
Remember, the consent form will also need to reflect whether you are now recording the participants face and screen as well the audio.
- Think about where the interviews are taking place
If you are running moderated sessions and video recording them, take extra care to protect the privacy of both interviewer and participants by not including any personal aspects such as family photos.
- For moderated research consider the human effect of not running the session face to face, for example:
How can you build rapport with participants virtually right at the beginning of the session?
How can you mitigate for the fact it is harder or impossible to pick up non-verbal cues?
Think about the cues you would normally give as an interviewer and perhaps adjust them. For example, if you normally nod your head to encourage anxious participants you may need to be more vocal in your encouragement if you are now running telephone interviews.
Often in face to face interviews there are a few minutes at the end where the researcher is packing up and preparing to leave which can give the participant time to think of any further questions. This is probably not the case when running research remotely so it can feel like the ending is more abrupt. Perhaps consider subtly preparing the respondent for the end of the session.
Ethical considerations
There may be topics of research which ordinarily cause little concern for participants but have become more sensitive due to recent events. As researchers we should explore the ethics procedures our own departments have in place. If this is not a well established process then there is also a self-assessment process provided by the UK Statistics Authority to review the ethics of your research projects.
Software considerations
There are many different tools available for remote research.
Some things to consider when looking at your options:
- What are your research aims?
- It may be that the software is not suitable for those aims or you may not even need software if telephone interviews can meet your research aims.
- Do your participants need to download any software?
- If they do, is that a burdensome process and will it exclude less digitally able participants?
- How does the software tool work with assistive technologies?
- Does the software tool meet your departments requirements for managing personal information?
- Seek advice from your department’s IT security team to ensure that any software you would like to use is also protecting the data rights of your participants.
- What technology does your participant have access to?
- Some software tools require a laptop or desktop for use but some participants may only have access to a mobile phone or tablet.
- Do you need to screen share?
- Do you need to audio or video record the session?