Ethical user testing and avoiding bias in the usability testing process

You’ve just created a website that you think looks amazing. There are bright colors everywhere, your logo flashes in the middle of the screen, you’ve installed one of those hamburger menus so the main menu is hidden and you’re ready to go. But you find that you aren’t getting the traffic you expected with a web redesign. What did you do wrong?

Well, besides putting your logo in an unexpected place, potentially using distracting colors and imagery, and hiding your menu, i.e. how your users will know where to go and navigate your website, you didn’t test how your intended future user likes your website, how they use it, and what feedback they have.

Of course, most new website redesigns and development won’t be as extreme (or use outdated methods) like our fictional website, but user-testing is quite possibly one of the best investments you can make in your website (besides the whole design in the first place). User testing, even just one round, can give you invaluable insight that you just wouldn’t have without because CEOs, company employees, and web developers are just too close to the work to see the forest for the trees.

But just like you wouldn’t test your website on an audience with prior knowledge of your company (i.e. your grandma and your neighbor), you want to ensure that you’re conducting non-bias and ethical user tests. Let’s get into how to conduct ethical user testing and avoid bias in the usability testing/user testing process.

What is usability testing/user testing?

In web design and development, a user test or usability test is designed to test a system in question. A researcher (often called a ‘facilitator’ or ‘moderator’) takes a user, generally an individual who fits the buyer persona for the website or fits their usual customer base, and asks them to perform a task using an interface (usually a website). Despite the term ‘user-testing’, the researcher is only observing the participant’s use of the interface, listening for feedback, and taking notes that could be useful to the web developer and designer.

The terms of usability testing and user testing are often used interchangeably. On a related note, you may often see the terms UI/UX floating around the internet which refers to the user interface and user experience design. Generally, a web designer and developer wants the interface and the experience to be positive and easy-to-use.

What is the purpose of user testing?

User testing tries to create an environment (in so far as you can control factors) as close to the intended audience’s environment as possible and test the interface in those conditions. In web design, the goal is to identify any problems, discover opportunities for improvement or places where you can better meet the user’s needs, and learn about the target audience’s behavior and preferences.

The researcher will gather important information to report back to the designer and developer. Generally, the web designer and developer will continue to work on the site, conduct more rounds of user-tests, until they feel the design is ready for release.

User testing varies in complexity and some designers and developers conduct single-testing sessions and others conduct multiple smaller tests.

Read our pieces on ‘The importance of behavioral research with usability issues‘ and ‘How to test prototypes for usability issues‘.

What are ethics in user testing?

You might have seen that one Black Mirror episode called ‘Playtest’ where Wyatt Russell’s character (aka the son of Goldie Hawn and Kurt Russell) signs up to participate in a video game testing session. The catch is he has no idea how his data will be used, what the risks are involved, who is observing him, and he’s purposefully manipulated. (Oh, and spoiler alert: he dies.) Whether you’ve seen the show or not, this scenario is the ultimate example of unethical user testing.

No matter how experienced you are at conducting user tests, the biggest consideration is ‘research ethics.’ Research ethics ensure that you preserve the rights, well-being, and dignity of your participants. Nothing you do should violate, harm, trick, coerce, or manipulate your test subjects in any way. 

How do I observe my users ethically?

A more subtle way you can preserve the ethics of your testing is to create the proper test environment, free from distractions, and ensure that your questions are worded carefully and thoroughly. It’s easy to word a question in a way that will confuse the user and subvert what you’re actually asking them to do. Make your tasks simple to follow and allow the user space, time, and comfort to go through the tasks. Researchers should be in a position to observe their subjects out of their eye line and their personal space so that the users are not distracted or nervous by your presence. 

Take notes quietly and as effectively as possible. Note anything that seems important such as any mistakes the user makes in navigation, any system errors or error messages, their click-paths and navigation sequences, any strategies or tools the participant uses or shares, the search terms, and results, any interesting quotes they make, and any suggestions, questions, and comments you’d like to make.

Anyone who comes into contact with participants should understand the code of conduct, have research-ethics training, have guidance documents on how to collect consent and approach researching certain topics, have standardized consent forms and information sheets, and understand the data policies for user research, including understanding how the data will be used, how it’s stored, and how it’s handled.

How do you avoid human bias in user testing?

Humans are naturally curious and often have preconceived notions which can lead to bias. When conducting a user test, it’s important to eliminate the bias in the participants. Don’t engage in any conversation before the test, outside of explaining the necessary information to the user. Don’t offer advice, correct them, or answer any questions as it can invalidate your test results. If you give any cues, verbal and nonverbal, to your user, you’re not recreating the optimal environment, which is to create a situation in which the user can navigate your website without any prior knowledge. If the user asks a question, you can respond politely that you will gladly speak to them after the session.

Remove bias by trying not to fidget, sigh, distract participants, express facial expressions, and so on. Try not to go in and out of the room and respect the testing space.

Need usability-testing services?

At Key Medium, we conduct usability testing on your website, whether you want small or large-scale testing, creating optimal non-bias and ethical test spaces so you can get the most from your research. Get in touch today.