TOP 5 usability mistakes when doing usability tests with users

What are the most common usability mistakes that people do when testing users during usability studies? We list our top 5 picks and ideas how you can avoid making those same usability mistakes.

There is a number of posts on our website already describing how to plan the tests with users, how to perform the tests, how to measure, etc. What we haven’t yet wrote about is what are the most common usability mistakes that people do when testing users and evaluating usability of your product or service..

Not every mistake appears every time, of course, and there are also other ways to screw up your tests. Top 5 usability mistakes we are listing here represent the most common usability testing mistakes that also can have a significant effect on the overall goal of the testing, which is improving the product.

We’ve had a fair share of usability mistakes on our own, especially those listed here. I hope this article helps someone avoid making these mistakes. So, here they are…

1. Not choosing the meaningful scenarios to test – one of key usability mistakes

One of the first questions that we ask ourselves when planning tests with users is what exactly do we test. We can’t just go through the site and keep asking users “do you know what this button do?”, “do you know what this is for?”, because there’s no context around that question and whatever the answer we would get, it’s not valuable.

So we need a context, a story, that will make users get to that button on their own. This story is the scenario that you are testing. An example is “Using the XYZ service, find the nearest Mexican restaurant and make a reservation for tomorrow night”. Choosing a scenario that isn’t all that frequently used is an obvious mistake. But the less obvious mistake is setting a too broad scenario in order to cover more of the interface.

TOP 5 usability mistakes when doing the usability tests with users
The scenario should be defined in such a way that when the user hears it, it makes sense, and it neither seems incomplete nor it seems overwhelming to memorize. That’s when the user can do the things naturally and the feedback has the greatest value.

2. Wrong persona tested

When doing the test with the user, it is important to get the person that would have normally used your product. And this is not because they will might how to perform tasks, while other people won’t. It’s because they have certain expectations from the product, while those people who usually do not need your product, will only evaluate what they see.

For example, let say you are testing a banking system and you want to see if users can learn about the housing loans that the bank is offering. If you are testing this with the person who either already has a loan or has already planned to get one, they will be interested in tools like calculators (e.g. I say how much I earn, and spend on food, clothing, etc… and it tells me what kind of loan should be ideal for me), simulations, or something like that. Someone who has never seriously thought about getting a bank loan might simply go through the scenario and not think about what might be missing.

You should carefully look for the people you bring to the tests, in order to get the best possible feedback.

3. Not getting the most from the user

This is about how you facilitate the test. The most common mistake is assuming you know why someone said or did something, and simply moving along from there. If the user has paused for longer than you would expect, or has done something that is unusual, don’t assume you know why that happened and then ask “Did you do this because…?”. Never ask closed questions (those that have YES or NO answers). You should even be careful about asking a question like “Why did you click that button?”, because it kind of implies that it was a mistake, and users might start thinking they did something wrong.

Simply remind them to share your thoughts with you, and guide them to getting a clarification of what they are currently doing or expecting to happen. You should never give the users any indication of whether or not what they are doing is the right thing to do. They should figure that out on their own. You should only gather as much as information and thoughts as possible, without interfering with their thinking.

The most common mistake is assuming you know why someone said or did something, and simply moving along from there.

It usually happens that after you have already run the same scenario with a number of users, and many of them have had a problem with a certain part, when another user is having a problem there, you might assume it’s for the same reason. Don’t do that. Approach every user fully objectively, without being influenced by previous tests.

4. Not understanding what the outcome of tests represents

Usability tests are good for finding problems, not solutions! I can’t stress that enough.

The results of the usability testing cannot tell you which solution would be optimal. It only tells you (if you have performed it correctly), where is the problem, and why is that a problem. In order to know which solution is optimal (out of few of them), you have to test them and see which one performs the best.

Usability tests are good for finding problems, not solutions!

Sometimes users themselves have ideas how they would solve a problem they just encountered. Of course, it does not mean you should do what they said, but don’t dismiss their idea. If anything, it shows you how they are thinking. It is just one person, it’s not a representative sample, but it does give you a valuable insight into their way of reasoning.

5. Testing for nothing

I’ve seen this too many times to not mention it here. Clients do everything they need in order to test with the users, they get the recommendation what should be changed and how, and then simply put that aside for now, because of some other development priorities. Usually this ends up with about the third of the problems being solved eventually, and the rest of them never.

True, the explanation of why to test with users does only say to identify usability problems, which is what the tests do, but that’s not where your job stops. The tests are just the first part of the whole initiative, and the end goal is to have a better product, not just be aware of the problems in your current one.

When planning for the tests, plan for the development effort after the tests. Have that in mind when scheduling and defining budget. Sure, some of the problems that the tests will identify, might be so fundamental that they are not that easy to fix. That’s understandable. You really don’t have to fix all of them. Those that are show stoppers sure, but not all of them.

However, what I’m talking about here is the mindset that I’ve picked up form some clients after the tests, where they seem to be happy with finally understanding what the problems are, but not really keen on solving them in some conceivable future.

Share this article