How to Gain Important Insights from User Testing

In July of 2022, our studio manager Udi Salant gave an enrichment presentation to employees of the Tipalti company at their offices in Galil Yam. The Salant ID studio has been working with Tipalti for over three years. Tipalti is a leading unicorn in the field of fintech not only in Israel but also around the world. The lecture dealt with user testing / usability testing and emphasized how to conduct user testing correctly, as well as what important insights can and should be derived from the process. The methodology Udi presented reflects the procedures carried out at the SalantID studio, and this article summarizes the main points of the lecture.

When we work on a system for a client, regardless of how well we know the content of their field, one of the most important principles that guide us is the knowledge that although we have extensive experience and knowledge, there is also the modesty that we do not know better than users themselves. Sometimes we produce a system that we believe in, that we gave our best to create an optimal system, but then when we go and test the product in front of potential users and receive completely different feedback than expected. Essentially, the guideline for success for any product will always be from users themselves, and if auser fails to use the system correctly, the system is not accessible enough for them. As a result, we will make changes to the system and check it again.

We are all human, and we are all affected quite a bit by inherent cognitive biases. One common human bias is called “the false consensus effect” – a built-in bias where we think that what we believe in is a common belief among others as well. For UX and UI designers, this bias creates a feeling of “falling in love” with our ideas and makes it difficult for us to step into someone else’s shoes and understand that maybe they perceive things differently. Therefore, it is very important to listen to our users in the tests and consider all the important conclusions that emerge during the sessions.
To avoid the false consensus effect, do not conduct focus groups in a common room since users will influence each other
User tests done by the SalantID studio

So What are User Tests and What is Done in Practice?

User tests consist of two groups of people – researchers and participants. The researcher gives specific instructions to participants of the user test, and those participants then go and complete pre-written tasks. While the participant operates the system and begins completing those previously-mentioned tasks, the researcher monitors participants’ behavior as well as listens to them and summarizes their feedback.

User testing has many advantages:

  • Monitoring problems: user tests make it possible to find problems with a system that we as UX and UI designers did not notice and as designers did not notice.
  • Uncovering opportunities: Sometimes opportunities emerge that we hadn’t thought of before.
  • Learn about the potential users: it is always important to notice how users behave and what they prefer during user tests.

User tests can be divided into four different categories:

  • Moderate synchronous user testing is when a researcher holds a meeting with the participant and guides them to act according to a pre-written list of tasks. This can be done in a physical meeting or remotely. The experimenter can change and direct the participant as needed.
  • Asynchronous and non-moderate user testing is similar to synchronous testing only that the instructions are not delivered by an experimenter but by the computer into which the various tasks have been entered. From there, the user can perform them in their free time. The performance can be saved and recorded and viewed at a later stage. This is sometimes problematic because the user does not always properly understand the task or conveys irrelevant information that later costs valuable time for the analysis of the test findings.
  • Guerrilla user testing involves having colleagues at work or in the field examine your UX and UI designs and give feedback on them from a usability and visibility standpoint.
  • Testing with existing customers: is when you examine a platform or system in the real world with real customers who actually use your product.

Types of users and the different levels for examination

The users of a system can simultaneously be a very diverse population or a very limited one. Users can differ in basic parameters such as gender, age, nationality, religion, socio-economic status, and more. They can also differ in their personal preferences such as interests, level of technological mastery, and a variety of interpersonal differences that exist in each of us. Each person is different and our goal is to create a system that will be easy and accessible in the best way for each and every user. Of course there is no way to characterize and design a system that will perfectly fit the needs of one hundred percent of your users, but we will create a common enough denominator to meet the needs of most users, as well as create a support system for those who have difficulty (just like training wheels on children’s bicycles).

User tests can be done at several levels or layers depending on the needs and according to the mental and financial investment of a company. Systems can be tested on colleagues at work or on company employees outside of the product team. Potential users can be gathered, given tasks, and have their performance tested. Moreover, companies with higher levels of products can conduct user tests on beta versions and their systems, as well as real-world testing or performance testing of customers already using their systems.

Types of user testing and their distribution in effectiveness and recruiting efforts
The graph above comparing insights to number of users tested show that most insights were gathered from testing around five subjects

How Many Users Do You Need to Test a System?

Nielsen Norman, the godfather of all things user experience, sums it up in the most clear way – “The answer is 5, except in situations where it is not the case”. Google, for example, will bring in between 5-8 users, depending on the situation. In general, there is a general agreement in the field that this can and should be a small, single digit number of users. The idea behind these numbers is that 80% of the problems that are in a system will be fixed in user testing with these five participants, and the remaining 20% of problems will only be fixed after many users have used the system, so in relation to the investment versus the outcome, the most cost-effective way to improve the system is by conducting user testing at a smaller scale initially, with at least five users.

The Main Questions We Ask Ourselves in User Testings

  •  Value Proposition: Are you contributing something that your potential users actually want or need and is not adequately addressed elsewhere in the market? 

  • Relevance: Does the system convey what the potential customers expect? Does it match their needs and wants? 

  • Clarity: How easily is your value proposition communicated? It is reflected dynamically in your product with images, microcopy, and calls to action?

  • Urgency: How do you meet and utilize the internal motivations of your potential users versus creating external incentives.

  • Distractions: Is your platform pulling users away from the main purpose they came in for?

  • Anxiety: Are there processes that create uncertainty or fear among your users when they go to perform actions that you want them to perform?

What Not to Do in User Testing

So after we talked about what user tests are and how to perform them, let’s dive into important guidelines and steps for conducting user tests so that you can make the best out of them:

  1. Don’t examine users in groups – instead, examine users one at a time. We as humans have a tendency to imitate other people’s behaviors and agree with arguments they put forward. Therefore it’s important to conduct user testing one on one to limit any biases. By doing this, we can form a full picture of the problems and malfunctions we have in a system since each user will have genuine, unbiased feedback while interacting with the system.
  2. Don’t (inadvertently) add stress to the subject by sitting in front of them . Instead, we recommend you sit next to  subjects to help them feel as comfortable as possible during a user test. You should create a pleasant and welcoming environment for them that feels comfortable. Keeping user tests one-on-one can help with that (one participant and one facilitator), as can sitting next to the participant rather than across from and especially not behind them.
What not to do in user testing? Several experimenters in one room sitting in front of the participant, giving them a feeling of being tested
The optimal user testing environment includes a quiet space free from distractions with the examiner and participant sitting side by side

How Do You Prepare for User Testing?

User testing is not a trivial process. For user tests, you need to organize and prepare a script in advance in order to avoid variables that may interfere with receiving feedback. It is extremely important to carry out a similar if not identical procedure between all users, for example by asking the same questions and providing the same setting for all users being tested. These are the steps you need to follow in order to build user tests the right way:

  1. Preparation and assessments:
    1. Prepare a test plan document
    2. Set up a user profile
    3. Build user scripts for the tests
  2. Administering the tests:
    1. Create an interactive prototype for the tests
    2. Conduct a conversation with the subject
    3. Collect findings during a session
  3. Analysis and findings:
    1. Consolidate findings into a unified document
    2. Analyze findings
    3. Draw conclusions and create an action plan for improving your product
The testing environment must be a quiet environment, in which only the subject and the experimenter are present. You should give an initial briefing to the subject and explain that the product is being tested, not the user’s performance. Let the user read the various tasks from the script and ask clarifying questions. A method that greatly contributes to the understanding of the subject’s behavior is to ask them to speak what is going through their mind at any given moment during the test (known as the think-aloud method).

Who Should Do the User Testing?

Many large, well-known technology giants have entire departments whose job it is to conduct user tests, gather the findings, and pass them on. This role is often defined as a UX researcher and is performed by people who specialize in user testing specifically.
In smaller workplaces, the role of user testing always falls between several roles and anyone can take on the responsibility – starting with the digital manager, through the product managers, and ending with the UX and UI designers.
Many people can perform user testing depending on the need and organization

Example 1: We examined why the “about” page diverts users from the main flow of filling out a referral form

Instead of sending the desired form, we saw in the tests that users tend to click on the “about” button and actually exit the process of sending a referral form
In the design of the new page, we provided an answer to this problem which increased the conversion percentage by 34%

Example 2: Opinion rating on a business is stuck in the process

A combination of qualitative insights from user testing (red) and analytical data gathered during the process (blue) resulted in an improved characterization and design. Some of the problems that arose during the test were related to navigation, visibility, calls to action, and feedback from the interface.
The new design answered the various problems found during user testing and resulted in a significant improvement in the conversion ratio

Want to learn more about projects where we conducted user testing? Look at the Effectivate or ZoomEngage projects from our portfolio.

Interested in hearing a little more about user testing workshops at our studio? Contact us and we would be happy to tell you more!

פוסטים נוספים שאולי יוכלו לענין…