As someone working in the UX field, I could say that one of the most important factors for good user experience is knowing your user’s behavior.
The good news is, there are multiple research methods and tools available. However, it is not always practical to observe these behaviors directly such as when and where these behaviors are occurring.
And this is where reliable digital tools come to play. There are now a variety of remote UX tools that will help you collect behavioral data. These tools use an established method called diary study.
Today’s article, Diary Studies: A Step-By-Step Guide for Beginners will help you get started in setting up your custom diary.
Diary studies are the type of research method that collects qualitative data like behaviors, activities, and user experience.
Just like a journal, diary studies contain data entered by the participants. These are self-reported data longitudinally structured. This means the reportings take place from several days to months, or even longer.
During this study period, the participants are asked to log to their diaries at a specified time to enter the information needed for the study.
Since this is a remote method of testing, users are prompted several times to help them remember to fill in their diary.
Diary studies are defined by two factors: the context and the time spent in gathering data. These factors are what make this methodology different from the rest of the user-research methods that we know.
This method is also known as the "poor man's field study" as they are unlikely to provide information that is detailed and complete compared to field studies. However, diary studies have some advantages that are important in specific cases.
What are the advantages of running diary studies when conducting user research?
There are several benefits to using diary studies when it comes to understanding user experience behavior. This research method often surfaces topics that the UX team does not know exist.
Like for example, in a more controlled type of research, participants may not provide information or phenomenon in an interview or survey, simply because these data do not come to mind at that particular moment or time.
Since diary studies are longitudinal, they are likely to get different information from participants that did not surface in other study methods.
Another advantage of conducting diary studies is its ability to collect data in its natural form. Since it is minimally intrusive, participants can provide more accurate diary entries on their behavior.
Diary studies are efficient as you can conduct the experiment to 20, 30, or even more participants at the same time. On top of this, it does not need a team to commit to a very intensive kind of research.
This type of user behavior study is beneficial in terms of getting to know the market first before conducting more extensive and in-depth research.
Diary studies prepare the UX team to get prepared and to know their respondents first before heading out for a face-to-face observation, which can help them gain helpful insights and come up with more meaningful observations.
When to conduct diary studies
While going out to participant locations and observing them first hand is typically my first choice when striving to study and understand behaviors, there are several reasons why diary studies can sometimes be a more appropriate choice. For instance:
We already discussed the importance of diary studies, specifically why this method is more appropriate than the other user research methods that use direct observation.
In addition, diary studies are essential in the following instances:
When you want to know about the moments and actions that affect the decision-making process.
If you are curious about how people make a decision or take an action on something as big as moving to a new home or buying a car, then diary studies can show you how these daily moments lead to a major decision or action.
If you are curious about the different experiences, behaviors, and perceptions of users and how these factors change over time.
Observing the same participants over a period of time lets you track the different events, moments, and moods that can have an impact on their decisions.
It can also show how participants learn a new system or product, which can teach you something about customer loyalty.
Overall, this study method exposes users' habits on a daily basis rather than just a particular moment or time.
If direct observation can affect user behavior.
This is true for activities that are private or sensitive to talk about. Having a barrier between the researcher and the participant can influence data and can hinder you from getting more transparent answers. These personal topics are more appropriate in the diary studies setting.
When you want to know what motivates the user to act.
For example, why would someone prefer to ride the Uber than the train? The diary study method can get you with closer behavioral data and depending on the length of the study, you can get good answers about the users' thoughts that impact their decisions.
When you want to know the behaviors that happen occasionally.
Not all behaviors happen on schedule. The best way to know occasional activities is to conduct diary studies. For instance, you want to make sure when parents worry about their kids or when people look for decor inspiration.
A survey method or an interview method will only get you with a glimpse of these experiences. A diary study method, however, will get you more contextual diary entries like "once in a while" or "10x a day".
When direct observation is not an option in observing the sensitive market.
This has something to do with access, privacy, and location. A diary study, for instance, is a more viable option for teenage participants who prefer to record their activities on their mobile phones rather than talking to an observer.
When there is not enough time for you to conduct the research.
Sometimes things get so busy that you do not have time to conduct your study. Diary studies are the easiest option. Just set-up the parameters and you can work on a more immediate task first. Then after a few weeks, you may get back to check the results.
When you need to impress investors with real and more personal data.
If you want to make a good impression, then a diary study method can help you collect "human moments" such as videos of your participants showing their thoughts and feelings. This type of qualitative data is a straightforward way of making an impact in front of your investors.
What are the topic scopes covered in this methodology?
Depending on the topic, the focus of a diary study ranges from broad to targeted. Below are the topic scopes often covered in diary studies method:
Product/Website - understanding the interactions with a site over a period of time. Example: Intranet
Behavior - gathering of information about user behavior. Example: use of smartphones, student web visitation patterns
General activity - understanding how users complete general activities. Example: online shopping, social media shares
A specific activity - understanding how users complete specific activities. Example: buying a new house, planning a vacation
How do you set up a diary study?
Rather than giving you a list of tips on running your own diary studies, we at Userpeek came up with a sample case and a step-by-step study process that you can use as reference in conducting your diary study.
One of our clients runs a news application that collects, aggregates, and distributes the news.
The UX team is tasked to plot new features based on what the user needs. We decided to run a study that gives us some broad, accurate idea of how our target users stay up-to-date with the current events.
However, the team needs to travel across the country to trail the participants until they click on a headline.
Since this is not a viable option, we decided to conduct a diary study. This method is cost-efficient in understanding better ways the news app may provide more value in their services that can easily fit into the participants day to day needs.
The step-by-step guide
Step 1: The preparation
Here's a checklist of what you need to prepare:
Define your research questions to suit the diary studies structure
There are two factors to consider here: the "in-the-moment" and "immediacy" factors.
You need to create questions that take advantage of the diary study structure. because your goal is to capture the "insights" of that moment.
Getting participants to log their responses as soon as they read or take action unveils the new, the interesting, and the nuances of those actions and in return, you will get a closer understanding of their behaviors.
Additionally, when you write your research questions or hypothesis, you need to decide on a corresponding trigger that tells your participants when to document.
Set your start and end dates
Ask yourself these questions: "how long should I run this study?" "how much time do I need to get valuable information?" You do not want your participants to lose interest in logging in for some time.
Choose the right tool based on your goals, budget, and timeframe
The tool used matters a lot in your research because the right tool will help you properly collect data. You have the following options:
The manual method
You can send a diary to your participants and they can send the diary back to you.
The downside is, it's surely going to be a challenge to read and sort all information from each of the participants. Plus, it can be hard to read that awful handwriting.
Everything will be done manually including waiting for that mail to arrive and manually importing the data.
There's also a possibility that the participants may misinterpret your instructions if they are unclear. If this happens, you can end up with unusable data.
Having said that, we do not recommend you to use this method unless you have a very small budget to start with.
The DIY method
This is doable for platforms that your participants are already familiar with such as email, Facebook, SMS, Whatsapp, and the likes.
Depending on the platform you use, you may collect multimedia data and you can also send reminders. You may have an automatic search feature as well.
This approach is not that perfect and may leave you some challenges like data import can be a problem and some of your participants may be wary of giving their information over social. Still, if you have a limited budget, this is a viable option.
Use the available paid tools for research
These tools may not be free but they offer all the nice features that you need to run a successful dairy studies research.
Some of the features for paid research tools include digital diary features with the option to login multimedia entries, set reminders for participants to log their entries, and it works on mobile too.
When choosing the right tool, look for something that allows easy sorting, tagging, and exporting.
Recruit quality participants
Whatever your research needs are, always get solid recruits. This means you should aim for fewer but better participants.
Screen and segment your participants carefully, considering the conclusions that you want to make.
Do not worry about having only several recruits. Even with a few but quality and engaging participants, you will still be able to get a truckload of information.
Choose the right props or stimuli
Remember that props can be mental. If you have an item or product for your participants, get that in the mail as well.
Decide how you want your participants to log in their entries
You can check below some techniques as a guide:
Interval-contingent protocol: predetermined intervals for the participants to report. Example: The participants can take notes every 3 hours throughout the day. Where to apply: This is best suited for behaviors that are not situation- dependent, for situations that do not occur on specified moment and time, or for day-to-day activities.
Event-contingent protocol /in-situation logging: whenever an event occurs, the participant reports. Example: When a participant gets distracted or anxious, he logs in and answers a set of questions.
Signal-contingent protocol: a set alert or alarm that tells the participants to report.
Our preparation
Our research team considered these factors in coming up with our questions and set-up for our remote study:
We considered the length of time a person would be willing to try the news app before it falls out of habit.
We spent time on research about our target market's use pattern, schedule, and daily activities.
We considered the best times our participants will interact with the app.
After looking at the factors above, we have decided to put our participants on a single workweek timeline for logging.
We have set some days for any potential post-study reflection sessions.
We have enabled participants to send in multimedia files so we can show these to our stakeholders.
We have decided to use a diary study tool as deemed suited for our needs above.
We have set a single trigger for the participants: They should log right after any active steps taken in using our news app.
We have set reminders in the diary study tool so test subjects are prompted throughout the day. In addition, they are prompted every 7 p.m. to reflect on how informed they feel and how many times they have tuned in to the app.
Step 2: The Process
Congratulations! You made it to the application stage. This is where your planning stage comes to life.
Here are the questions that may arise once you are already running your study research:
How often should I communicate with the participants?
As often as needed. There is no hard rule here but you should have an idea of what you expect. Do not forget to send them reminders, guidance, and acknowledge their entries.
You should sincerely talk to your participants by being human first, and researcher second.
Giving your participants frequent feedback will also encourage them to provide better and more detailed responses.
Be understanding all the time. Your participants deserve more than just "thank you". Saying something like "we are sorry to hear that and we sincerely appreciate your response" whenever they are having a bad day will go a long way.
What should I do when there is a lot of data coming in?
Do not procrastinate. Take notes as data comes in. And by doing this, you will have a good idea about your follow-up questions. You will also be able to see the participants that need more pushing and encouragement on their digital diary entries.
Our process
We have assigned members of our team to spend at least several minutes a day reading the participants' entries, responding to them, and reminding them if they missed an entry.
We also bookmarked all insights that we thought are good so we can pull them easily anytime for synthesis.
For also take note of all engaged responses as well and flag them as a focal point for our analysis.
We also spent time responding to test subjects who were upset by the sad news story. We took empathy with them and sincerely thank them for providing their honest feedback.
Step 3: The follow-up
In this step, you must have already known your participants and you have a good idea of what they have submitted.
But once you get to know more about them, most likely you still want to know more insights that you like to dive deeper.
In this case, here are our suggestions:
Schedule a follow-up interview with your most engaged participants or users that have unique insights
This helps you confirm your haunch, close a gap, or provide you with the needed insights. With just a 30-minute debriefing after the diary studies research can go a long way with your research.
Ask for feedback
You can ask your participants questions like: Did the study go well? What did you learn from this experience? What are your suggestions that you think can contribute to better study research?
All these questions can lead to exposing biases and improving your future study.
Improve your qualitative data with quantitative data
You can make a chart analysis or run A/B testing with the data you have gathered. Quantitative data may show better and clearer study data that can be used for new hypotheses.
Our follow up
As our team completed the data, we realized that many of the participants' engagement with the news is passive --- they usually just check the news links sent to them by their family or colleague, or they just click the headline they see on their news feed.
Few people check the app or newsletter for news. Only a few people stay loyal to a particular news source.
We have also found out there are 12 participants who consistently provide engaging insights. We have picked our top 3 from these participants for follow-up questions.
Step 4: The analysis
After you have gathered and completed your data, you need to set a time to sit with the data, track possible patterns, and synthesize everything.
Here are several data analysis tips to take note:
Identify your star participants
These are participants that represent a larger pattern or have given you particular insights. You can tag them as "star participants" and use them as leads in your synthesis and presentations.
You can add quotes and videos from other participating members, which are your supporting cast.
Share insights with investors
You can schedule your presentation and discuss the results to anyone who you know can benefit from an in-depth user insight. This means going beyond just the design or product team.
If your research goals and diary study design were successful, then you can use these data for your company to present results that help investors make sound decisions.
Our analysis
Our team gathered together for a brief synthesis. We looked at the raw data to discuss the possible conclusions.
We have decided on a few important ways that these data can be used.
We recommend new integrations that can work with other discovery mechanisms that the existing market already uses.
We also enlist some jobs to be done based on the motivations and pain points that we have gathered from the study.
To sum it up!
We've learned that there is a wide range of situations where we can use a diary studies method. This method does not have to be any more difficult or time-consuming than other usability tests.
Even without experience, you can try to set-up your DIY diary studies using the manual method or with the help of the right automation tool.
Have you done any diary studies before? Did you find it extremely useful in terms of gathering data? Share your experience with us. We love to know 🙂
