Period: May 2020 – August 2021
- The UX maturity in the company was not satisfactory
- A diary study uncovered a lot of UX issues
- Presenting the diary findings in video format increased the empathy towards users and improved the UX maturity across the company
The user research had a low impact across the company and colleagues did not see its value. Many times, it didn’t lead to concrete actions.
The company worked in product areas, where each area focused on specific part of the product. As a result, the end-to-end user experience was unknown.
Overall, user empathy was not enough to drive a consistent user-centered approach across the product teams.
A survey among the UX team revealed that our company was at the Emergent maturity level. This was far away from our goal to be at least Integrated and, ideally, User-driven.
Role and process
Our UX research team, consisting of 3 researchers, initiated the study. The idea came casually while going for lunch one day and discussing the lack of visibility of the end-to-end user experience of Skrill. I said:
The best thing we can do is a longitudinal study, like a diary study
A diary study would provide visibility of the complete end-to-end user experience for a more extended period. We discussed the idea and decided to try it. The other 2 user researchers and I took part in each stage of the study process.
- No diary tool: How will users record their sessions?
- Hard recruitment: participants had to be willing to use Skrill for 1 month with their real money.
- Covid-19: the lockdowns denied the opportunity for an in-person study.
- Lack of time: the UX research team schedule was already full of projects.
- A lot of data: the produced report will likely be so big that stakeholders won’t have the time to attentively go through it.
Considering the above challenges, we decided to leverage the UserTesting.com platform that we had already on-boarded. As this was not a standard study, we had to make a request for custom research. We wanted users to use their existing payment methods to increase the validity of the study. However, following discussions with Usertesting.com representatives, we decided not to record moments where personal information was displayed. Participants used their own money during the study that we reimbursed at the end of the study.
A video diary study
As everybody in the company was busy, no one had the time to go through a potentially 100+ pages of the user research report. To solve the problem we decided that instead of creating a giant pdf report, we would create a video series showing how customers use our product. This way, stakeholders would be able to view our users in action and observe how they feel and behave during the different stages of their user journey. Additionally, this will also be much easier to digest than an intimidating 100+ page report.
The idea was to:
- Recruit 10 participants that were willing to use our product for one month
- Explain the procedure to participants in a kick-off interview
- Have participants complete predefined tasks and record a video of them doing so each week
- Conduct a final interview with each participant
- Create video episodes for each stage of their user journey
Each session with participants was video recorded. The friendly UserTesting.com staff agreed to find us 10 targeted participants. They managed all the recruitment and complex consent requirements, since users had to use their own money.
Creating the episodes
Thanks to the UserTesting Highlight reel feature, we were able to cut specific parts of the user sessions. Unfortunately, it took a lot of time to watch and cut the important moments from the user sessions. The length of all user sessions was more than 20 hours. Finally, we compiled 13 episodes, each around 20-30 minutes long. We even included cool background music to make them more engaging.
Finally, 13 episodes were produced in what we call “The Skrill documentary series.” Each week, an episode was played at the weekly meeting where the latest customer research is usually presented. Product owners, managers, customer support agents, directors of all kinds of departments and even C-levels, usually attended the meeting.
The 13 video diary episodes covering the end-to-end journey of our users
The summary of the diary study findings is as follows:
The Skrill wallet is very easy to use, with a very aesthetic and delightful design. However, the high fees ruin the overall user experience.
We uncovered issues that were previously invisible during the testing of individual areas of our product.
The results were unexpectedly good. Stakeholders were much more engaged when viewing and listening to users, rather than reading reports. This significantly increased user empathy across the company.
“If people don’t see something with their own eyes, it is hard for them to believe in it”
We’ve never seen so much discussion and enthusiasm in these weekly research meetings. Everyone got excited, suggesting ideas and sharing experience on certain product issues.
Here is a list of the positive outcomes of the diary study:
- We uncovered UX issues that we were previously unaware of during testing of individual areas of our product
- Stakeholders emphasized considerably more with users, which led to taking action on user issues
- Created productive discussions among colleagues.
- Fixed user issues that improved user-retention
When our colleagues witnessed how users struggled to use our product and how they felt during that time, they were able to emphasize with users much more than before.
The video diary study has considerable potential to increase the UX maturity across any organization. Although challenging, the outcomes produced are definitely worth it.