In user testing, one of the most important questions to ask is, how many users do you need?
This question is very important when it comes to planning to conduct a usability test. Most companies are reluctant to do usability testing because they are unsure of the number of users to test and most of them overestimate the cost of the entire usability test.
In this article, User Testing How Many Users Do You Need? we will cover and discuss the number of users you should conduct a user test with, including the following sub-topics below:
When it comes to the number of users to test, the common answer you will get is 5 users. Testing with 5 users has been the standard in UX research. For some, this has become the rule of practice for any type of user research, whether it be field studies or diary studies.
Today, we will discuss further why the 5-user test should not be the standard and UX researchers should look at the kind of user research they conduct before they decide on the right number of users.
Why this is important? The sample size plays an important part in UX because it impacts the learning curve as well as the decision-making process. Unable to determine the right size of users can likely cut you short of valuable insights.
He further argues that he still recommends a 5-user test since he started promoting the “discount usability engineering” in 1989.
This applies to all kinds of usability testing like websites, intranets, PC apps, and mobile apps. Nielsen highly recommends the 5-user testing to be enough number of participants when it comes to getting close to usability testing’s maximum benefit-cost ratio.
The benefits of 5 users
Cost-effective and practical
One of the obvious reasons for a 5-user test is its practicality and cost-effectiveness that every business can benefit from when it comes to finding usability problems and insights.
Another benefit companies can get with this is they can do a comprehensive round of user testing in one day since there are only 5 participants.
The more users you test, the more days you need to acquire and more resources to get as well.
Sufficient insights
Just as Jakob Nielsen suggested, having five users is enough number of participants to let you get as many usability problems as you would in more test participants.
That having smaller groups can make report writing effectively as well. And in comparison, more participants mean more analysis and video compilation time.
The five users test avoids this kind of unnecessary work for the UX research team.
The usability problems in using 5 users in user testing
There are quite several debates on why the 5-user mantra of Jakob Nielsen should not be the standard number of users in usability testing.
For one, it has been found out that this is out of touch with the reality of what modern Agile development is about and that this strategy is hurting more teams than it is helping.
The 5-user sample size is repeated by so many UX companies and agencies and because they have seen poor results and usability problems from using it, a lot are questioning its effectiveness. Here’s why:
A lot has changed since the 1990s
Jakob Nielsen’s 5-users mantra is rooted in his discount usability engineering where it can be traced back to hostile engineering environments.
But UX today is more than that and it has a strategic advantage today. The UX industry is now more valued than it was back then.
The UX professionals’ goal is to learn more about its users and making quick generalizations leads to poor decisions. We should carefully conduct our usability testing, including knowing the right size of our participants because all UX efforts are based on usability testing.
Having said this, the five users of the Nielsen Norman Group are a bit small sample sizes to get the right insights.
Factors that influence sample size choices:
Varied segments
The 5-user sample only considers having one type of audience. This will not meet the need of organizations that have numerous customer segments or user types.
Low fi or high fi
The 5-user testing can only be applied to low fidelity style of test (non-working code). If the prototype is considered higher fidelity, we recommend you test 10-15 users.
Formative or summative
If your design is a concept on the formative side (directional insights), then often, you can test with fewer users. If your design is published and/or you are looking for more solid metrics to quantify user experience, then you need to test more users.
Other factors to consider when using a small sample size
Like we previously said, sample sizing is a huge factor in UX because this impacts learning and decision-making. However, small sample sizes can be jeopardized by the following factors:
Recruiting issues
Users who are poorly recruited can result in inaccurate insights. You cannot just grab random people. These people are not your market. Instead, you want to test actual customers or your end-users who can relate to the pain that you are focusing on.
Moderation
Another factor that can lead to inaccurate data is poor moderation. For example, you do not expect users to answer truthfully when asked what they expect here and there. The best form of moderation is listening in any type of UX research.
Observation
You may likely get poor pattern recognition by making generalized insights from 1 to five users, which leads to poor decision-making.
Low insight users
If you only have 5 or fewer users for UX research, your insights may be diminished as not all users can provide you with good information that you can use to make decisions. Some people simply do not have enough words to say.
Thus, usability testing with 5 users is a gamble. If this small tests does not yield insights or sustain patterns, then you may get thin on actionable data pretty fast.
What is the right sample size to get the right insights?
The answer is it depends. The right sample sizing for participant test depends on your approach. So if you want to follow Nielsen Norman Group advice on using 5 users for participant testing, we suggest you do 5 users per customer segment, 3x with enough time to iterate in between the number of usability tests. For example, if you have 3 customer segments, you need to test 15 users, then iterate that in 3 rounds.
The most practical answer to the question, how many users do I need in usability testing should be based on your many test on 5 users for segment. You need to determine the number of iterations after you conduct the initial first experiment with quite a good sample sizing.
Also, you need to take note that for optimal UX, conduct a field study first to understand who and what you are designing for before you do the number of usability testing.
In summary
As discussed, there are clearly some exceptions to the rule of five that UX researchers should keep in mind.
If you are usability testing a website or an app with multiple functionalities, you should user tests per each function. The same holds true if you have more market segments. You need to test five participants per user segment.
Some UX research needs completely different user group with different behaviors. For example, on a property website, there could be buyers, renters, and sellers. So, in this case, you need to experiment 5 buyers, 5 renters, and 5 sellers to get a good understanding of their behaviors.
Another thing you need to know is that the rule of five may also be broken when stakeholders require statistical proof from usability testing.
While this is not really an ideal practice in a qualitative method, this situation is quite unavoidable.
For usability testing, you are free to follow Nielsen's advice on test with 5 users whenever this is possible. You can assess 5 users or less and then do many iterations of usability tests.
But if you have multiple user segments and you do not have time to do 3-5 iterations, then do user experiments of 8 to 10 users for prototypes, 10-20 users for finished products. If you can, we suggest you iterate a second round of testing for better user insights.
For field studies, interviewing users at home or work using diary studies, you need 15 to 40 users. Take note that having fully understood your user needs in a field study reduces the need to do many testing iterations.
