In this article, What Is Preference Testing In UX, we will look more into preference testing as part of user research, including what is it, when is it used, how it is different from A/B testing, and its disadvantages in user research.
What is preference testing?
Preference testing is a type of test where users are provided with two different choices and they are expected to choose which they prefer more. This is a valuable UX tool as the users go beyond just choosing an option.
In preference testing, it goes beyond just what the users want as it is used to look for more qualitative feedback. This feedback helps make a larger and bigger picture that is geared towards a better understanding of the best design for users.
The designers use this approach to test two different design concepts. For example, in a website design, you can show users two different styles of navigation. The users’ feedback will not only tell you which designs they prefer more but also their feedback on why they chose it. The users’ feedback will provide you with some valuable information about your design, wherein you can make changes until you get into the right design.
When do you conduct a preference test?
We use preference testing to know what users prefer and why. This user test is conducted in the early stages of the design. This helps get proof of concept before there is a commitment of time, effort, and resources put in developing the finished product. By conducting this kind of test early on, you will have a better understanding of what your customers want and why.
What to look for in preference tests
Like we previously mentioned, preference testing is more than just knowing which design your users prefer. While the question may start from which one they prefer, a huge part of what you are looking for will come from observing their interactions with the design and hearing their reasons “why”.
What is more important is gathering the qualitative data, which helps you do a better design based on what your users prefer.
Common questions asked in preference testing
You can use the common questions below if you are unsure what you should be asking. These questions are a good start-up guide for your test setup:
- Which design do you prefer?
- Which design looks more trustworthy?
- Which design looks more modern?
- Which design looks easiest to use
Getting quantitative feedback in preference testing
You will have the number of participants who preferred each design shown on the results page. This is a straightforward comparison of designs and from this, you can calculate the statistical significance of the result.
By statistical significance, this is defined as the likelihood of the best design is the one chosen and is not outperforming the other designs by random chance.
The level of significance you obtain will vary on the sample size. Of course, the larger the sizes will give you a greater significance. Additionally, it will depend on the degree of difference between the design’s performance with a large difference in performance giving greater significance.
Getting qualitative feedback in preference testing
Your qualitative feedback is from asking the participants why they have chosen the design they did. You may categorize their feedback into smaller groups to get a better view of the number of participants who had similar feedback.
Qualitative feedback is very important because it allows you to discover new areas for design improvement, which helps you with future design decisions.
Preference testing vs A/B testing
You may ask and wonder about the difference between preference testing and A/B tests, which is both geared to knowing which design stands out best for users.
However, A/B testing is not the same as preference tests. For example, you have two designs and the goal is to find out which one they prefer better. This is not AB testing, but this is preference tests.
The book titled, Think Like A UX Researcher, it discussed about three kinds of UX research pieces of evidence:
- Strong evidence- is from participant doing tasks or engaging in some activity relevant to the product being designed
- Moderately strong evidence- is from studies that include carrying out tasks
- Weak evidence- comes from methods that are flawed or a bit better than guesswork
Preference test as discussed further in the book is considered a form of weak evidence. In contrast, A/B test is under the strong evidence category.
There are four reasons why preference test is under the weak evidence category:
It does not reflect the real-world usage
In the real world, people are not instructed to choose between two different designs. Instead, people use a product to achieve a goal. Thus, asking people to choose a better design for them, which they have not used will just lead to decisions that are less important to the design.
People are less invested in the outcome
Indeed, people can easily choose between two things when asked because it is an easy decision. However, for the qualitative data, people taking preference tests may not be able to provide better answers to the why questions because they are unsure as to why they like the design better. Additionally, they can provide answers that are based on fickle opinions as they are mostly questioned on the spot.
It asks people to predict the future
Preference test is about asking people to imagine a future where there is a better design that exists and lets them predict which one they will use and why. However, some behavioral studies suggest that people are poor at predicting a wide range of future behaviors.
These behaviors include the time it takes to complete the tasks, the prediction that they will have wide long and happy relationships, and how well they perform in exams.
It is further explained that the best predictor of user's future behaviors is their past behaviors. So rather than asking people what they prefer, it is best suggested to learn which they perform best with.
It mixes research questions with interview questions
The research question is the purpose of your research. This is your guide to your design study and asking whether design A is better than design B is a valid example of the research question.
However, a research question is something that you cannot ask your participants in a study. Normally in user research, you simply ask participants to do the tasks and you observe in the background and record their activities.
In preference tests, asking participants with direct questions can lead to answers that may not be useful enough for your design study.
What did we learn?
Although there are flaws to preference test, still, this is one of the simplest ways we can get feedback from participants when it comes to design study. This is most useful when you want feedback on certain design aspects such as colors, logos, layout, and many other sections of user experience.
The benefit of conducting preference test is you can get feedback from participants in the early stages of the product design cycle. By running multiple preference tests, you can refine your product to be its best design and function before you can invest more time and money into its production.
However, preference testing is not enough testing method to get the best product design. Since this type of testing has its flaws, it is better to also conduct A/B testing to make sure that the design is better looking and more functional in the real-world setting.
Related Posts:
