It took Unilever 45 generations and 449 designs to create a perfect nozzle which would dispense washing powder.
5,127 prototypes of Dyson’s cyclone technology were created before James Dyson believed the technology was fit to go into a vacuum cleaner.
Over 3 million years of trial and error have occurred for you and I to evolve into what we are now.
Testing, iteration and trial and error.
It’s what differentiate’s the “good” from the “bad” and the “great” from the “good”. Yet, we don’t seem to take this approach when it comes to user experience.
This is what this article is about: user experience testing and measuring.
What is User Experience Testing?
User Experience Testing definition:
User experience testing is the process of collecting qualitative and quantitative data from the user, whilst the user is subject to all aspects of a service or product. Qualitative or quantitative data is collected from the user to improve ux and is done using various methods and usually with the aid of a tool or service.
What is user experience?
A quick summary, provided by Don Norman Jakob Nielsen, defines UX as: “All aspects of the end-user’s interaction with the company, its services, and its products”.
Likewise, Peter Morville and friends from Semantic Studios developed the user experience honeycomb to encompass 7 facets of the user experience.
In more detail:
- Useful: Does your site, product, service have a practical purpose? The greater the use, the greater the experience.
- Usable: Is your site, product or service able to be used for it’s purpose? Is it complicated and confusing or easy to understand?
- Desirable: Adding emotion within your design, through the use of imagery, identity, branding and other elements of emotional design.
- Findable: Can user’s locate what they’are after? Whether that be a piece of information or a product. Information and objects need to be locatable.
- Accessible: Proper use of alt text for images, organised headings for structured content, correctly labeled forms etc. (Top tips for making sites accessible)
- Credible: Will the user trust you? Design and design elements influence user’s trust and belief in you.
- Valuable: Can the user derive value from your site, product or service? Do you improve a user’s satisfaction?
How to test user experience?
User experience testing can be broken down into several methods. Each method is used to uncover insights from one or more of the facets of UX.
Even though the methods are being categorised under “user experience testing”, there are other phrases used to describe these methods, including: UX testing, user testing and usability testing (which actually is only one part of UX testing).
User Experience Testing Methods
User testing is the process of testing a product or experience with real users of that product. Users who complete the “user test” are presented with a combination of scenarios, tasks and questions with the screen of the user’s device recorded for later analysis.
“If you want a great site, you’ve got to test. After you’ve worked on a site for even a few weeks, you can’t see it freshly anymore. You know too much. The only way to find out if it really works is to test it.” – Steve Krug – Don’t Make Me Think
There are two main types of user testing:
- Moderated user testing
- Unmoderated user testing
Moderated User Testing
Moderated user testing is when the user completes set tasks on a product or service, whilst being observed by a researcher (moderator) in real time.
During a moderated user test, you have the ability to guide users through tasks, probe and dive deeper into problems and receive real time feedback. Moderated user testing has the advantage of being able to explore a user problem in real time, but, has the disadvantage of biases and facilitation.
The most influential bias during moderated user testing is the Hawthorne Effect.
- Ability to probe users during the test for more information
- Elicit real-time feedback and emotional responses
- Good sample quality with tight control over user activity
- Can be expensive trying to hire a lab
- You may not have the equipment to conduct in-house mod testing
- Time to recruit, pre-screen, run test and analyse
- Limited to few locations (unless conducting remote moderated)
Unmoderated User Testing
Similar to moderated user testing, unmoderated user testing is the process of assigning participants a series of scenarios, tasks and questions, usually being completed in a remote setting. E.g. the participants home.
Unmoderated user testing has some great benefits over moderated user testing, for example:
- Large geographic diversity, with the ability to recruit from around the world
- Easy to run with large sample sizes
- Low operating costs due to low compensation and no requirement for faciliation
- Timing is quick!
- Known upfront costs
However, there are a few disadvantages:
- Sample quality can be poor when recruitment is done incorrectly
- Can’t probe users during the test session
- If a user gets stuck, you can’t direct in the right direction
Five Second Tests
Five second tests are best used to answer the following questions:
- What does/is this site used for?
- Who is the company?
- What benefit does this service or product provide to me?
How to create a five-second test ?
- Create a design of what you want to test. In most cases, a homepage, landing page or product page. You can use a Google chrome extension to take a screen shot for you.
- Upload the design to a testing service, such as UsabilityHub.
- Set your test questions. What do the users recall?
Five second tests are a great way to deliver qualitative insights as well as quantitative data to produce results.
If there is an imbalance between the intent and the results, you most likely have a clarity (copy-writing) problem. Back to the drawing board…
The goal of card sorting is to understand how users find information, navigate a system (e.g. using a menu), label information and organise the information provided.
To conduct a card sorting session, you can use:
- Real Cards
- Pieces of Paper
- Sticky Notes
- Or Online Card Sorting Software
Open and Closed Card Sorting
Learn how users group and label content or learn how users sort pre-defined content items? Card sorting can be split into two types:
- Open Card Sorting: Participants are free to organise topics of content into groups that make sense to them. These groups of topics are then labelled by the participant, without pre-determined fields.
- Closed Card Sorting: Like open card sorting, closed card sorting asks participants to sort topics of content into pre-defined categories.
Used in the same way as card sorting, tree testing (aka reverse card sorting) is a ux testing method for evaluating the “findabililty” of topics in a website.
Using a simplified text version of a site’s structure, user’s navigate a series of top-level topics, choosing a heading and then a list of subtopics. User’s complete the task when they are satisfied they have chosen the correct topic.
How to do tree testing?
- Create your “tree”. This is the site’s structure of your website.
- Write a series of “find it” tasks. For example: You bought a new watch and need a protective case. Where would you look to a buy a protect case?
- Recruit participants for your test. More times than not, you want to recruit user’s of your website or product.
- Have participants complete your tasks.
- Analyse the final results.
Also to refine the structure of the site, tree testing can also be used to quantify the improvements. Testing can be done on the existing site as a baseline which can be compared against the outcomes of each stage of testing. – Stephen Byrne
A session replay is a non-audio recording of a user’s journey and interactions on your site. They include user’s scrolling, clicks and form interactions and are usually anonymous.
Great for identifying user behaviours and trends and has an advantage over user testing, as users do not know they’re being observed. Therefore, users display their real on-site behaviour.
Surveys and Polls
Working in tandem, surveys and polls are great for collating qualitative and quantitative data from your users.
With survey and poll tools allowing behavior triggering, you can gather data from your users at the point they displayed a certain on-page behaviour
- A user just purchased an item: send them an incentivised survey to learn how hard or easy the buying process was.
- A user spends 5 minutes on the one product page: fire a poll to ask if there’s missing information.
The beauty of polls and surveys is they allow you structure a variety of questions.
Surveys can be a really useful UX tool to provide input for the design process. The key to a successful survey is establishing the objectives and information required from the study up front, then making sure the questions asked cover them. – Chris Gray, Director, Nomat
Great for revealing user behaviour on pages, heat-maps are a graphical representation of where a user or users have clicked, scrolled to, mouse hovered or looked at, on a page.
Show where a series of users have clicked on a specific page. The heat-map usually identifies high-level click areas as red, with lower levels of interaction ranging from orange, yellow, green and blue.
Based on a percentage of the page’s traffic, a scroll heat-map shows the percentage of users who scroll to a level/area on a specific page.
Reveal where your users are hovering their mouse/moving their mouse.
Eye Tracking Heat-Maps
Similar to how click heat-maps work, eye tracking heat-maps show where users look most on a webpage. Likewise, the heat-map reveals the path the user’s eye took when browsing on the page.
Mainly known as “voice of the customer“, intercept question is the ability for users to provide feedback on the user experience as and when they feel the need to.
Popular tools such as Hotjar and UserSnap, offer the ability to add a widget to your site, which allows user’s to select an area of the site they wish to provide feedback on.
This can allow the user to be open and honest about certain elements of your site, that impact the user experience. However, the feedback received isn’t in-depth as user testing or session replays and can vary from page to page, limiting the sample size.
Even though sample size may be small, you can gather insights to tone and use of language used by users and in turn, influence your own copy-writing to relate to the user.
A/B Testing – the best way to accurately measure if your changes are positively affecting the metric that matters for your user experience.
If you’re new to A/B testing, ConversionXL have this comprehensive guide for you to follow.
Being able to truly understand how a user feels during the user experience, is UX researcher’s dream.
Are they frustrated? Do they enjoy this? Are they engaged of uninterested?
Being able to accurately understand a user’s feelings during an experience, helps us optimise and improve our UX.
Biometric studies encompass numerous research methods, including:
- Eye Tracking: The ability to track a participants eye movement – what they’re looking at, the eye path, how long they’re looking etc. Kind of like click-heat map but for the eye.
- EEG (electroencephalography): Measuring electrical activity in the brain via electrodes. Great for on-the-fly representation of emotions and tracking of cognitive activity.
- GSR (Galvanic Skin Response): Electrical conductivity is measured in to the small amounts of sweat generated by the pores on your skin. Provides insights into your emotional states.
For a complete list, with information on each method, check out this blog post.
User Experience Testing Tools [List]
User testing: moderated and unmoderated
Five Second Tests
Surveys and Polls
- More of a service rather than a tool
- More of a service rather than a tool
- CoolTool sell eye tracking equipment
Conclusion: A User Experience Testing Checklist
- Start by identifying the outcomes/goals you want to achieve by testing the current user experience.
- Ask what questions need to be answered in order to achieve your outcomes/goals.
- Choose a ux testing method that will reveal the answers to your questions. Likewise, be sure to pick a method that aligns with your time, cost and experience.
- With the aid of a tool or service, conduct the ux test and gather qualitative and quantitative data.
- Analyse the data, design, iterate and repeat.