Usability testing – this weird, wonderful technique/skill/process/method that can reveal insights to how your users really use your website, prototype, product and so on.
But what is usability testing?
Teaching you everything about what usabiltiy testing is, to go from newbie to competent in one article, is next to impossible.
But the next best thing is a guide that provides you actionable information and resources to achieve usability testing expertise on your own.
This is that guide.
Before we can guide you, we must define what it is…or are we already guiding you by defining it?
Defintion:Usability testing refers to the process of evaluating a product, website or prototype’s user interface ease of use and learnability, through a means of testing.
The end goal is to discover usability problems, collect qualitative data and measure the participants ease of use, learnability and satisfaction with your website, prototype product etc.
Layman terms: you find out how people use your website, product, prototype and so on.
There’s this misconpcetion of when conducting usability testing, if participants can’t complete tasks, use your product or navigate your site, it’s the particpants fault.
Unfortunately not.
Usability testing has nothing to do with testing the user, but everything to do with testing your product, website or prototype.
Click to Tweet
So, why does usability testing matter?
In the book, Cost-Justifying Usability, Clare-Marie Karat, Ph.D. from the IBM Watson Research Center referenced a study where:
"$20,700 spent on usability resulted in a $47,700 return on the first day, the improvements we implemented and $68,000 spent on usability on another system resulted in $6,800,000 return in the first year."
Who wouldn’t want those kind of returns?
Now, user tests aren’t just important so you can increase financials – that’s a bi-product from the insights you derive from testing.
However, usability testing is important for the following:
Check if your product is meeting user’s expectations
Identifying and removing bugs from your website or product
Revealing user frustrations in completing their tasks with your product
Matching your business decisions with real-world product use.
Ultimately, it matters to your business and your users.
What is the purpose of usability testing?
The purpose of usability testing is to test the ease of use of your product or website – to reveal the learnability behind your product and the satisfaction a user (person) has when using your product or website.
A successful round of usability testing should explain the why behind your analytics data.
As an example: you run an ecommerce site and have a 60% drop off from add to bag to checkout. A usability test should reveal the why behind that drop-off.
There are a couple of usability testing methods that can reveal the why behind your what.
Usability Testing Methods
UX researchers and devoted practicioners in the UX (user experience) industry, have developed many techniques and skills in order to validate or in-validate what they thought they knew about their user’s behaviour.
Some of the most popular techniques are usability testing, card sorting & A/B testing:
1. Usability Testing
Often referred to as “user testing”, usability testing provides a way to understand how real users experience your product or website. It is one of the few methods that provides flexibility to what information you can gather, due to it’s multiple use cases e.g.
Test how users would interact with your prototype
Reveal how customers find information about delivery
Compare competitors to see who has better navigation or search
There is often a debate between whether you should use moderated usability testing or unmoderated remote usability testing – the decision should be made on per situation a basis, as each method has advantages and disadvantages.
Moderated Usability Testing
Mostly used during the design phase of a website or product, moderated usability testing allows a moderator to sit live with a participant – faciliating tasks, asking questions and see live reactions/actions as a user users a website or prototype.
Tip: It’s recommended to have two moderators when testing – one to ask questions, the other to transcribe insights.
Advantages:
Control: you’re able to control and guide the participant during a test. As an example, the participant may say or do something you find interesting – you’re able to probe deeper by asking further questions around that behaviour.
Time with participant: as you’re with the participant, you’re able to set the expectation that the test will run for 45mins to an hour. Likewise, incentivisation in MUT is generally higher and therefore, participants are comfortable with providing an hours worth of feedback.
Participant recruitment: with MUT, you usually recruit your own participoants or use a recruitment research firm. This allows you to specify exact criteria that represents your real users e.g. recently bought X product, shops on our site 3 times a week, AOV > $100 etc.
Disadvantages:
Expense: generally speaking, MUT is expensive compared to it’s counterpart, URUT. When you factor in the time to setup, travel expenses, testing labs, incentives and time to analyse the feedback, your testing session can be on the other side of £1,000+.
Time: compared to URUT, there is a lot of time involved to gain user insights e.g. time to plan session/s, schedule participants, run session and then analyse and report on findings.
Bias: we’re human and all have natural biases. Unconscious body language, miscues and non-vebal cues can lead to false-positives in participant results.
Location: it’s quite common for MUT to take part in some form of “testing laboratory”, unless you have on-site facilities. These “testing laboratories” are, unfortunately, unnatural and are unlike the participants real environment e.g. their home, in a shop etc. The unnatural environment can hinder test results and feedback – causing errors and non-realistic feedback.
When to use moderated usability testing?
“Moderated usability testing is best to be used during alpha, beta and live phases, to test prototypes or the service you’ve built. You can also use it in the discover phase to learn about problems with an existing service.” – Gov.uk.
Where to learn?
Learning how to do moderated usability testing best comes from practice. The more tests you run, the better you’ll get.
However, I won’t leave you out on a limb…
Here are some resources on where to learn moderated usability testing:
"My main recommendation is to be prepared and relaxed. If you're relaxed, then the user will be relaxed. You're more likely to talk less and the user will talke more, which is the purpose of the test."
Dr. Giles Anderson // User Experience Designer / Researcher, Packt
"Expect users to get things wrong, therefore, be as explicit as possible with intructions to minimise possible confusion."
Anna Livingstone // UX Researcher, PhotoBox
"For in-lab MUT - keep all tasks written in some peices of paper and deliver them to your userm by reading aloud at the moment before each task. This action will help you to be consistent through the different sessions of user testing, once you are not explaining the task in your own words, which could vary between the sessions. Asking the users to read the task aloud, you can check if they understand the task."
Marcella Medeiros // UX Researcher Freelancer
"For remote MUT - keep an eye on your intonation while reading the tasks. While the participant is performing the task, minimise the interruptions. Although you can encourage him/her to think aloud, your aim is to understand the user's mental model, through their thoughts, rationale and frustrations."
Marcella Medeiros// UX Researcher Freelancer
Unmoderated Remote Usability Testing
As the name implies, unmoderated remote usability testing, is usability testing that is completed with the participant in their location of choosing, without a moderator.
Due to having no moderator and participants being able to complete tests remotely, unmoderated remote usability testing offers, fast, accurate and inexpensive insights.
Advantages
Cost: tools such as Userpeek.com, allow you run tests with 5 testers with a small budget. Due to the low cost, this allows you to afford more tests and in turn, conduct more.
Insights: gaining more insights is a bi-product of the cost of URUT – lower cost, afford more tests, run more tests, have greater insights into the behaviour of your target audience.
Time: once you’ve created your test and put your test live, participants will begin their test in a matter of minutes. Reducing your time to receive insights from a day, to minutes/hours.
Location: participants peform remote testing in their natural environment e.g. home, office, supermarket, resulting in relaxed participants that are representative of your real users. This also alleviates pressure from the tester and results in more accurate test results.
Disadvantages
Test control: with unmoderated usability testing, you have to predict where the participant will be on your site/prototype, at a given task. This can lead to mis-match of tasks and questions, confusing the participant. Likewise, you don’t have the back-end-forth conversation you can have in MUT.
Time on test: as tasks are completed when the participant decides, tasks may end prematurely, resulting in inaccurate time-on-task data.
Recruitment: while you can recruit more participants from larger demographic panels, you are sometimes limited to just demographic data. Demographic data isn’t a full representation of your actual users.
When to use unmoderated remote usability testing?
URUT has the privilage of having numerous use cases – practically never a bad time to use it. Having said that, URUT is best used when:
You need answers to short, focused research questions
You have budget contraints – you can derive the same level of insights from URUT as you would in MUT
Easily examine competitors and benchmark the competition
Time is of the essence and you need quick feedback
Need large sample sizes
Have a global user base and need to test many partcipants at once
Increadibly helpful for identifying how users classify information on your website, participants in a card sorting session will organise topics into categories that make sense to them – often referred to as evaluating information architecture. Tip: Before beginning a card sorting session, think, what am I going to do with the results? Hypothesise the best, worst and most likely scenarios for your results, and then imagine what you would do with each.
There are three main types of card sorting, which are: 1. Open Card Sort: participants sort cards into categories that make sense to them, and label each category themselves. When to use? Use open card sorting when designing the information architecture of your website, or aiming to improve it. This method is more generative than evaluative; providing you with ideas on how to structure and label your categories.
2. Closed Card Sort: participants sort cards into categories you label. When to use? Use closed card sorting when you want to evaluate your current site structure.
3. Hybrid Card Sort: participants sort cards into categories you label, but are also given the option to create and label there own categories. When to use? Use hybrid card sorting if you want to generate ideas for grouping your information and/or need clarity on less certain groupings that occurred in earlier open card sorting sessions.
"Where possible ask your questions in the form of a narrative; this helps create a sort of state dependant mindset in the user, which means they are more likely to use the kinds of thought processes they would use in the real-life equivalent scenario to solve your card sort."
Rich Murphy // User Experience Designer, The Hut Group
"Give a visual representation of the item whenever possible, to avoid misinterpretation of the cards."
Anna Livingstone // UX Researcher, PhotoBox
"Be clear what the purpose of the card-sorting is and what you are going to do with the results. For example, in closed card-sorting you may be biasing users to categorise in a particular way, while in open card-sorting the number of groups created may not be suitable for your service."
Dr. Giles Anderson // User Experience Design / UX Research, Packt
"I would also suggest sense-checking your card names before you commit to a large scale user test, especially if it is unmoderated. Something that makes sense to you as a researcher, or to someone familiar to your business, might have a completely different meaning to a user without that context, and therefore bias your results."
Rich Murphy // UX Design, The Hut Group
"Be organised. During the sessions or data analytics, it's easy to feel lost among a number of card. Some good practices will help you in these moments: take photos after your user arranges and categorises the groupds, number the cards in the bottom or the back, use a spreadsheet to organise and analyse your data. This will help you to visualise a big picture of the dat collected."
Not usually thought of as a method to test the usability of something, but A/B testing can provide you a way of comparing two pieces of functionality and which seems to be the most usable…unfortunately, A/B testing won’t tell you why.
Tip: conduct two URUTs with A/B test variations to see how users use the functionality and reveal why one is better than the other.
When to use?
A/B testing is best used when you want to test and change the user behaviou. As an example, you see from your data, those who use search, convert at twice that of those who only use your navigation. Now, you can do a couple of things and change the user behaviour:
Test a variation of search e.g. make it larger, more bold – get more people using search
Test a variation of your navigation – make it easier for people to find what they’re looking for, restructure your categories and products
Whenever I’m asked, where do I begin when conducting a usability test?, my answer is always the same.
Start with a plan.
A goal without a plan is just a wish.
Click to Tweet
DISCLAIMER This part of the post won’t go into much detail on conducting a usability test from start to finish (that is for another time). We will cover in detail, how to write a testing plan.
Starting with a test plan has a couple major benefits over preparing your designs, finding your participants etc, including:
Test clarity: Writing a test plan, that covers your testing objectives, the type of participant you’re after and the testing script you’ll use, allow for clarity. Anyone in your team can pick up the test plan and know exactly what’s going on.
Test revision: Planning your test design before you begin recruiting or executing the test, will allow you and stakeholders to make any necessary changes, without wasting time and money. Planning ahead of time, will give you greater flexibility later on.
Script writing: Defining the objectives for your test, allows for easy script writing later on. By defining your objectives, you know what questions must be answered in order to achieve the objectives, and in turn, you’ll know what tasks to provide in your test script. As an example, an objective for a usability test on Amazon may be: Identify usability issues when finding a book. A question that needs to be answered is: What functionality of the site did the user’s use to find a suitable book? So, your test script may include a task, such as: Imagine you’re searching for a book about Google. Using this site, find a book about Google. Feel free to use the navigation of search to find your book.
Participant recruitment: Within your test plan, you’ll define the type of participant that is suitable for your test, e.g. shops online 5 times a week, buys technology items from Amazon. Defining your ideal participant early, allows changes to be made and recruitment services enough time to find the participant for your test.
How to write a usability testing test plan?
One of the first steps when conducting a round of MUT or URUT, is to construct a test plan. The test plan should document:
What are you going do?
How are you going to run the test?
What is your objective for this test?
What and how many participants are you going to test with?; &
What testing script will the participants use?
Start
Specify what you are going to do: what are you testing? when are you testing it? what device/s are you testing on, how many participants do I need and what type of participants?
Define your objective for the test and what questions must be answered, in order to achieve the objective/s. For example, our objective may be: Identify usability issues when finding a book on WHSmith. The question you would need answering is: What problems did the users encounter when finding a book?
Write your testing script and keep in mind your objective and the questions that need answering to achieve your objective. This will allow you to keep your test central to what you’re trying to achieve/wanting to know.
Torsten is CEO and founder of Userpeek. He is an old stager in the online business with 20 years of experience as an online marketer, conversion rate optimizer and UX strategist.
Thanks for your article!
Let’s check carefully the usability testing when creating new applications