In this article, Formative vs Summative UX Research, we will explore the meaning, differences, and purpose of the two usability research methods, how to perform them including how and when they are utilized during the product design cycle.
Before we dig deep into the differences between the two, formative and summative evaluations are both considered a form of evaluative usability research.
Both types are made to understand how a product performs compared to benchmarks in the product's history. Regardless if the research is formative or summative, the goal is to identify and resolve usability issues in order to improve a product’s design.
This method is mostly conducted during the early stages of the design and development process. It identifies the issues with user interface design and provides solutions to solve those issues during the first stages of the design and development process.
The formative method is one of the excellent tools when it comes to determining which design features are useful and which are not. This method heavily influences the design decisions during the product design process. Normally, you get a team of 5-7 users, with the goal to design a good user experience for the product.
The discovery-based process moves the iteration process forward as the testers discover more and more issues with the design of the product. The data collected during formative user testing sessions are observational in nature and deal with the quality of the design, hence the name qualitative usability testing.
We just discussed that formative usability testing has a strong influence on design decisions, which is why it is considered to be an excellent tool to figure out which design features are useful and which are not.
We recommended conducting at least two formative usability tests:
The first one is during the early design concept phase with a wireframe prototype that has no working functionality. This will allows for a validation of the workflows and the initial decisions around navigation, layout, and terminology used.
The second formative usability test should be conducted on a Minimum Viable Product (MVP) with some functionalities ready just before the development phase.
If we don’t conduct formative usability testing during the design and development lifecycle, we might end up designing something that will simply never be a usable solution.
To summarize, the formative usability testing:
Is carried out during the first stages of the design and development process
The goal is to improve the product design
Can be conducted anywhere and can utilize both remote moderated or unmoderated
Is performed on wireframes and MVP designs
Is Conducted with 5-8 users
The benefits of using formative evaluation
When it comes to [product iteration (without launching it), the formative is the way to go.
This method requires much less rigor compared to summative testing evaluation
The formative evaluations are done rapidly in succession to an iterative process
In formative evaluations, you only need the design team or the stakeholders
The formative usability evaluation is less expensive and less time-consuming than the summative evaluation
Summative user testing provides an indirect assessment of the design usability of the product. The group of testers in this situation is given a task and based on their performance, the usability quotient of the design elements is measured.
Summative user tests are done after the product has been launched in the market. It is usually conducted by a group of more than 10 testers.
The summative testing data collected are more quantitative data than qualitative data. Thus summative user testing is also called quantitative user testing.
Why conduct summative usability testing?
Summative evaluation provides us insights into how usable an interface or a product is. If we don’t do summative usability testing before a product is released, then we won’t find out if an interface or a product has any problems or not.
Summative usability testing is utilized to obtain measures to establish a usability benchmark or to compare results with usability requirements. The metrics recorded in summative usability testing reflect what actually happens during the test and not about the user's perceptions or feelings.
The usability requirements should be task-based, including task completion rate, time on task, error rates, and overall user satisfaction. It’s about measuring effectiveness and ease of use. These metrics include:
pass/fail of user tasks
the average time it takes to complete tasks
counting the number of clicks
users’ errors or system errors
In summative testing, the volume and speed of tasks undertaken, and the pass/fail metrics are more important than the quality of the observation or user narrative.
To summarize, the summative testing:
Is conducted at the end of the product development stage (before product release)
The goal is to validate the usability of a product against usability metrics
Is to establish a usability benchmark
Is utilized to compare competitor products
Is performed with fully functioning prototype
Is conducted with 15-20 users
The benefits of summative evaluation
Summative evaluations is best utilized when a product is about to be released and the design team needs valid satistical proof that the new design performs better than the previous design.
The audience for summative evaluations is ultimately the product’s users (and its reach could be millions of customers).
It is usually more expensive and time-consuming compared to formative evaluations.
Formative and summative goals
We already discussed the difference between formative and summative evaluations when it comes to the questions they answer and when they should be used.
Thus, it is given that when it comes to goals, these two tests have huge differences when it comes to goals.
Based on the tests used and what questions they answer, the difference between the goals of formative and summative usability tests is pretty easy to understand.
The first step of the redesign process is Summative evaluation, here the design process starts with the evaluation of the existing design. Then once the design elements are evaluated and the data is collected, the summative evaluations of that data will reveal insights into the issues, including where the product users are struggling the most.
Based on this data, the second step will perform qualitative testing to better understand the new design elements added to the platform.
Once the platform is designed, the two tests are repeated to guarantee the new design works well.
The whole process helps us understand and clarify the goals of summative and formative evaluation tests.
So in that sense, formative testing helps to quickly collect data during the first stages of the design process vs summative testing, which helps to collect an extensive amount of data and evaluate the design.
The method is probably the one factor that clearly differentiates the two types of usability testing. While both methods deal with user experience, the way they collect data is what makes all the difference.
Both testings user research tests use 3 pillars in their methodology:
the number of participants
the flexibility of study conditions
the protocol used
Qualitative or Formative usability research and test
The number of participants: limited, preferably until 5 members only
The study conditions: extremely flexible. There is no need to strictly maintain a singular kind of testing atmosphere during Formative tests.
The data: is collected through a think-aloud protocol, wherein the users when interacting with a product or service during the testing process give their feedback verbally.
Summative usability research and test
The number of participants: usually more than 20 users performing a single task.
The study conditions: strictly maintained with each session happening in the same predetermined atmosphere conducted by the development team.
The data: is collected through quantitative testing and usually, there is no think-aloud protocol. Data collected are much more precise than formative user testing.
Formative and summative usability outcomes
Qualitative data helps the product developer to look for the strengths and weaknesses of the design. However, this is not accurate when it comes to usability issues.
The outcomes of formative heuristic evaluation are the result of an irregular test done among a small number of people who may not even be able to represent the whole user demographic.
In addition, the outcomes of qualitative heuristic evaluation are based on the expertise and knowledge of the product developers and how they interpret the user’s struggle when it comes to user interface elements.
This is why the data gathered from formative testing is useful during the initial stages of design, but they are still not very reliable.
On the other hand, the quantitative study can provide concrete outcomes. The process involves a large number of users, which represents the entire demographic compared to formative testing.
In this method, the data is not collected arbitrarily. Instead, data is collected through statistical methods, which helps the product development team to understand the error margin of the product design and where most users are struggling. Using this method can take better action against the lack of design usability of the product.
Conclusion
Formative and summative research evaluations both serve their purposes in the product design process. We hope that in this article, you fully understand the differences between these two usability test methods in terms of how they are used in the field of user experience.
In summary, formative research is more about formulating knowledge about a product before it is even launched in the market. On the other hand, summative research is about testing the acquired insights from the formative research method using the pass/fail criteria.
In UX, the products must always be usable enough to pass final testing. The UX designers determine the tasks when it comes to design support, however, when it comes to product performance, it is always the users who will best decide which product or features perform well for them.
