UX CALENDAR – DECEMBER 22 – 5 questions to avoid in user testing and what to ask instead

The construction of questions in user testing is very important to collect reliable and useful insights. Let's take some common mistakes as an example: we'll see how to avoid them and improve our test protocols.

1 – How would you use this product?

Pay attention to any questions that ask the user to project themselves into future behavior. We must always take the "declarative" with a grain of salt, even when the user reports his memory of a real situation, and it is particularly difficult to rely on it when the user must imagine a fictional situation: what we would imagine doing is not often going to correspond with what we are actually going to do.

“How do you use/have you used this product?” 

Instead of asking him to imagine the future, we can instead ask the user questions about the present or the past: “how do you use similar products or functionalities today?”, “What are your past experiences with similar products or features?”

We can also see his spontaneous use of the product during the test scenario. To do this, we will try to guide the user as little as possible, even if it means going back to certain screens to ask questions and test elements that have not been seen previously.

2 – Should we add this feature?

When we ask a user if he would like us to add an additional element: information, a feature, a page, etc. we are still asking for a projection into the future. It will not be concrete enough for the user, who will practically always say “yes”: there are few reasons to be against an addition. It will be difficult during the analysis to calculate the share of convinced "yes" to really assess the appeal of the functionality. We cannot draw any conclusions.

Observing behavior 

Here, rather than a question, we prefer to observe the behavior during the test. The best way to assess the appeal of a feature will always be to observe its use directly or, if it does not yet exist, to observe the behavior around its absence: for example, we report an irritant that this feature could correct.

3 – Do you understand this element?

Asking the user to self-evaluate his understanding is risky: beyond the social desirability bias in the test context, the user is not always a good judge of his own understanding. They can perfectly understand a content or a functionality and feel complexity, or due to a lack of self-confidence, not believe that they understand, and at the same time they can have the impression of having understood and miss out on everything or part of the meaning.

“Can you explain this element to me?” 

Instead, the user will be asked to explain the item in their own words. When the answer is incorrect or imprecise, reminders will always be added to ensure that they constitute a misunderstanding on the part of the user rather than a difficulty in expressing themselves or explaining an element that is in fact understood. In a 2nd time, we can also ask if they found it clear or easy to understand, we will then have evaluated two things: understanding but also the complexity perceived by the user.

4 – How could this element be improved?

We generally avoid recruiting participants who work in digital design so as not to obtain an expert evaluation instead of a user test. Therefore, our users are not designers and asking them to put themselves in this role is complex and risky.

“In an ideal world, how would this element work?”

To know how to improve an element, we will rather observe the difficulties of the user and ask him to explain them to us, to detail what does not work and why. It's up to the designer to find the solution to these difficulties: it's a job!

We can actually ask this question, not to incorporate the desired design as is, but to relaunch it to understand even better what is blocking it. Rather, it is phrased as follows: “In an ideal world, what could you do with this element? How would it work? ".

5 – Is this product aesthetic?

Tastes and colors are very subjective and when we ask users to evaluate the aesthetics of a product, a site, a page, we end up with contradictory data: the color will please half respondents, and displease the other half. The example of color is relevant because it is often this element that strikes a non-designer participant first, and it is also the element that can rarely be modified.

“What does the design of the site mean to you?”

Aesthetics remains an important axis to evaluate, because it participates in the perception of usability of the interface. Users can tell us a lot about their perception of the product by their opinion on the aesthetics. We would rather ask “what does the design of the site evoke for you? », « what words come to mind to describe the aesthetics of the site? » and relaunch spontaneous remarks « You say that there are too many colours, can you explain to me why? What are the consequences of this “too many colors”? What do you feel ? ".

 

 

Take away

During a user test, comparing behaviors to participants' statements is the best way to collect relevant data.

When we challenge a shaky question in our protocol, we must always return to our research question and ask ourselves:

  • Why do we want to ask this question to the user?
  • What are we trying to measure?
  • How can we measure it?

 

Marie EUZEN, UX Designer @UX-Republic

 

Our next trainings