A pilot test is essential in user research and usability testing. In this article, Pilot Test UX, we will discuss why this is an important step prior to user research and test.
The purpose of this article is to determine any issues with your current pilot testing techniques, and at the same time, get a hold of the best practices with the procedures or process involved to apply in a product, application, or website.
We will also cover the following topics below:
Pilot testing can go a long way to alleviating problems that can happen during user research or user testing. In a pilot test, the team runs through an advance session or sessions within a small group that prepares for the full study or survey, making sure everything is in order. The point of the user test on pilot is conduct a test of the study itself, running advanced sessions to users to ensure that the actual study will go as smoothly as possible
Pilot testing in user experience research
Pilot testing is a quality control measure for marketing research and testing. It’s a dress rehearsal conducted to ensure the instructions are clear, the equipment is working properly, and there are no errors in the script.
By making a dry run through your testing protocol with just a few representative users, you can catch any problem areas before bringing the entire group on. In essence, the practice test makes sure the path you want testers to take is free of obstructions.
Pilot test vs beta test
Here’s the key difference between beta testing and pilot testing:
Pilot testing is performed within only a handful of users
It’s aimed at getting the bugs out of the testing system prior to its full deployment to collect feedback on its ease of use and performance.
Beta testing requires a lot of users to test a system that pilot testing has cleared
You know the big mistakes and points of confusion have been cleared. Now you’re fine-tuning the process.
Benefits of a pilot testing in user research and user testing
Advantage to newbies in usability study
Better to have your first try with a session that you can throw away if needed.
Helps in conducting test in an unfamiliar area
If this is your first test of a site aimed, for example, at rocket scientists or nuclear physicists, and you aren’t an expert in the area, pilot tests can help prepare you. (Also do work ahead of the study to familiarize yourself with the topics and terminology, for instance by meeting with subject-matter experts as well as designers and developers.)
Beneficial to remote and unmoderated usability test
Any time that study instructions need to stand on their own, they need to be tested to try to limit possibilities for misinterpretation or the prototype. If you are running a data diary study or recorded video, unmoderated online or video study, or study conducted via email, all communications, from recruiting information or data to confirmation emails to study instructions to follow-up questions, should be pilot tested. These instructions need to stand on their own, as no one will be there to answer questions or make clarifications if a participant runs into a problem.
Beneficial to quantitative study
Larger scale studies are typically done so that statistically significant results and feedback can be calculated. In such studies, each session needs to be run the same way. As such, a solid script must be created and tested.
Helps in testing high-visibility projects
Even if you’ve worked in usability for years, there may be the occasional high-visibility project that requires some extra care. Maybe the results and insights are going straight to the CEO. Maybe the test is on the company’s premier product. Of course, every usability project is important tasks, but some may be a bit more important than others.
Helps in one time research project
If you’re doing many rounds of iterative testing, then the damage is limited if you make a few mistakes in the first test. The second users data study will be better. But if a design project only gets a single dose of user research, you want to get it right the first time, because there won’t be a second time.
Experienced UX experts can benefit from running pilot tests
The longer you work in the field, the better you can get at writing strong tasks and instructions in the first place, but it never hurts to run a test plan past a participant in advance. Does every usability study require a pilot test? No, but it’s extra insurance that the study will run smoothly, leaving the team able to focus on the tasks results, rather than the study itself.
Pilot testing checklist
A pilot study to get the real test in order would seem like a no-brainer. There’s a tendency, though, to assume that testing sequences have already been tested by the project designers.
The design team is too close to the issues they’re testing and too familiar with the insights and results they expect to see. That leads to an unavoidable bias.
At The Good, we find most companies who think they’re running a pilot test aren’t.
Here are the following checklist to make sure the pilot testing prepare the main test instead of adding more confusion.
The subjects should qualify for the primary UX test subjects
They should fit the desired profile, and they should never be members of the team that designed the test you’re piloting.
Pilot testers should not be the test subjects in the main test
You don’t need a lot of pilot participants. Most of the time, one to three subjects are plenty.
Divide the test into logical sections and observe the time required to work through each
That will help you pinpoint trouble areas and adjust where needed. You don’t want participants to feel rushed. You want as much task-centered, focused feedback as you can get.
The purpose of the pilot study is to test the test
You’re looking for buttons that don’t work properly, instructions that aren’t clear, type that’s difficult to read – anything that hinders subjects from concentrating on the tasks they’ve been given to complete.
Do not conduct the pilot test immediately prior to the actual testing
You want to provide a buffer of time to allow test designers to make necessary corrections, and there are almost always changes required.
Keep pilot data study participants in the dark about being pilot participants
They should be given the same instructions and treated in every way just like the “real testers” will be treated. Avoid bias as much as humanly possible.
When selecting the pilot participants
For some studies, you can use just about anyone as a pilot test participant, such as a coworker, friend, or relative. For example, if you’re doing a pilot usability test of a general, consumer-facing website, such as a travel booking or video site, you could use just about anyone. They don’t have to have any special knowledge or experience.
But for some tests, it’s best to get a pilot test participant who closely fits the profile of the actual participants you’ll be using. For example, if you’re conducting usability testing of software used by tax accountants, it’s ideal to get a pilot test participant with the domain knowledge that’s needed to use that application. Even for something that seems simple, such as a study of people using a car buying website, it’s best to have a pilot test participant that fits the user profile of people who are considering buying a car in the next three months. If your questions have a lot to do with the motivation of car buyers and how they use the site, you’ll get very different answers from someone who doesn’t fit the profile. In these situations, if you use a pilot participant who doesn’t fit the profile, you won’t get a good sense of whether the questions are relevant, produce interesting discussion or research, or how long the session will take.
Why the first session cannot be the pilot session
If you don’t specifically conduct a pilot user test to a website or app, then your first session becomes a pilot test. That’s where you’ll find the problems with your method, the tasks, and the questions. The problem with waiting until your first participant to find the problems, is that you won’t have much time to make changes before the next session. You may be able to make minor changes in the brief time between the first and second testing research session, but it will be too late to make major changes.
The right pilot study
Ideally, conduct the pilot research test at least a day or two before the first session. Recruit an extra participant, who matches all the screening criteria, to perform the pilot test. If everything works out well, you can keep that person’s data. If you find research problems, you can correct those before the official research sessions begin.
Conduct a pilot study on new research task
Pilot testing is always useful, but it’s especially important when you’re going to try a new technique. For example, I once conducted a group workshop in which we had the participants divide into small groups to illustrate their work processes on post-it notes. We found that it was too difficult for them to do this together as a group activity. They ran out of time, and it wasn’t the best experience. For subsequent workshops, we had each participant work individually to illustrate their own work process, which was much easier and more effective. In effect, our first workshop was successful as an unintended pilot test research. However, it would have been much better if we had planned it as a pilot test with a smaller group of users or participants using the app or website.
Here’s one final thought if you think there is no use to do a pilot tests study:
With pilot testing, you do not only improve the website or application insights and feedback generated from a project, but it also improve on the process the the UX team uses to conduct the users research. Its reflective and iterative nature will fast forward the development of any skills of a UX researcher.
Having said thus, pilot testing is an investment of both time and effort. However, this is a significant investment that will deliver great insights and results on all of your incoming UX research projects and testing studies.