User testing & ROI: the secret to a profitable project

You want to maximize the ROI of your product But are you short on time? During our last UX Live session, Julie Petetin (UX designer in Luxembourg) and Florine Auffrait (UX Researcher in France) shared the keys to success. Did you miss it, or would you like to delve deeper? This interview expands on their discussion.

Florine explains in detail why user testing is the most powerful lever for ensuring a product's profitability. Through key figures and real-world experience, she demonstrates how observing actual usage patterns can eliminate "experience debt." This invisible financial drain arises from the disconnect between an interface and its users. Discover a pragmatic methodology for transforming every design uncertainty into a business certainty.

https://www.stlouisfed.org/open-vault/2018/june/fascinating-facts-cellphone-smartphone-usage

Julie: Florine, for those who didn't have the opportunity to attend or watch the replay of our webinar, could you remind us in a few words what a user test is?

Florine: User testing is, above all, about confronting a product with real-world conditions. Unlike an interview, we're not looking for feedback or opinions, but rather to observe usage in a real-world context and to understand the users' perspectives by having them verbalize their thoughts.

What's important for us is our approach: we observe where it works and where it doesn't, we listen to the user's logic and we follow up to uncover usability problems before they become costly.

The calculation is simple: postponing corrections costs 100 times more

Julie: Let's come back to the concept of testing costs. What do you say to those companies that spend too little, or even no, time on this step? User TestsBecause they feel that a test is slowing down the project?

Florine: If we look at a specific point in time, yes, of course we add a step by running tests. But this step is essential and actually prevents additional costs later on. And the best argument is always the numbers. I really like talking about the Boehm's lawThis states that the later a design flaw or misunderstanding of requirements is discovered, the higher the cost of correcting it. Let's assume that correcting a problem on a prototype costs €1. Waiting until the development phase will increase the cost to €10 for the same modification, and if we wait until the product is launched to correct it, it will cost €100. In other words, by not testing, we accept the risk of paying 100 times more an error that could have been spotted in an hour.

Julie: That's a good argument, but I imagine many companies think they know enough about their users and their usage patterns and believe they are not concerned by these figures?

Florine Yes, we can always imagine that we're better than others. Except that, once again, when we look at the studies on the subject, we realize that 50 % developers' time has been spent on rework considered avoidable, because it was spent redoing features that were poorly thought out and designed from the start. And besides, we also know that approximately One out of every two features of a complex software program is never or rarely used.It's really money down the drain: both during development and afterwards in terms of maintenance. So yes, we can gamble that we don't need to dig deeper because we already know the answer. It's a risky and costly gamble.

Ignored UX recommendations: a textbook case with 72 points of friction

Julie: In the webinar, you gave us an example of a situation where testing prevented the release of a product that wouldn't meet user needs. Conversely, have you ever encountered a situation with a client who didn't conduct testing or didn't follow the recommendations resulting from the tests?

Florine: Yes, I've encountered both of those situations. Often, when we're called in as UX researchers, the company has already seen the cost of a poor user experience and calls us in to avoid repeating those mistakes. So it's quite rare to find ourselves in a situation where they refuse to do any testing at all. However, ignoring recommendations is a classic scenario! I think the most striking example was on a project for a bank. At the beginning of this project, I was asked to "validate" a new, highly structured user journey. This journey was already in production; in fact, it was due to be released two weeks later: we arrived It was really at the last minute. In that case, I don't think we can really call it a "test": they were just looking for reassurance; it was a step that had to be completed very quickly. But during the user sessions, I identified 72 points of irritation, including several blocking ones. The main point was that none of the 10 users found the entry point of the functionality. Naturally, we couldn't "validate" this process, and that was the beginning of a battle to demonstrate to the client that it was absolutely necessary to correct this before the production release.

Julie: So, what happened?

Florine: I managed to get the entry point fixed, which was my big victory. But then, for the remaining 71 points, the client preferred to focus on training, thinking that simply showing users how to do things and how to work around usability issues would be sufficient.

https://powell-software.com/resources/blog/bad-communication-at-work/

Why training won't save a bad interface

Julie: It's true that many companies think, "If it's complex, we'll do a training session for the users." Do you think that's a viable solution?

Florine: It's a very costly strategy. Training should focus on developing skills for the job, not on compensating for a lack of usability.According to the Ebbinghaus curve, users forget 70 % of what they learned in training within 24 hours.

Julie: Regarding the feedback you mentioned, did the customer regret their choice?

Florine: I sincerely believe so. Because subsequently, support tickets raised most of the points of friction I had identified. For two years, we spent time on the product roadmap integrating improvements to this particular user journey every quarter, when we could have fixed them in a few hours on a prototype.

Julie: And as for the users, do you think they also pay the price for the dead end of a UX approach?

Florine: Yes, of course! A poorly designed interface requires extra effort and generates frustration. This inevitably leads to cognitive fatigue, and we know that This fatigue in the face of a cumbersome tool increases the risk of human error. 10 to 30%.

A return on investment (ROI) of 1 to 100

Julie: Conversely, what gains can be expected from a UX approach ?

Florine: There's a study conducted by Forrester in 2016, subsequently confirmed by other studies, which has become a benchmark for discussing UX ROI. It states that, on average, $1 invested in UX can yield up to $100. And in fact, it makes sense, because an easy-to-use product is one that people don't abandon mid-experience, thus increasing conversion rates. It's a product that people enjoyed using or that helped them achieve their goal, so they come back to it. Furthermore, since the vast majority of potential user problems have been anticipated and resolved, customer support isn't overwhelmed.

Julie: These figures are really interesting; do they relate to the UX approach as a whole, or just user testing?

Florine: It's difficult to isolate a specific method within a UX approach. User testing is an evaluative method: at the very least, we've already started brainstorming and developing a concept. UX needs to be considered earlier in the process. So, if we want to invent a product, we'll start with exploration through interviews, for example, but testing will come as soon as the first solution is sketched out. And for existing products, it's the best possible way to test them against reality. In any case, testing is clearly a fundamental step to guarantee a product's success.

Customer experience is even a driver of stock market growth, since the 10 most mature companies on this subject outperform the S&P 500 with returns three times higher than the average.

Julie: So, to sum up, conducting user tests costs money and time. But it's an investment that pays off as soon as the first test is analyzed?

Florine: Exactly! There's an example that few people know but that I really like to tell; it concerns the application YukaFor those who don't know, it's a mobile app which allows you to scan the food products you buy and get a score based on criteria such as sugar, additives, fat… Initially, they wanted to sell a connected device to hang on the fridge to scan products when putting them away. By listening to consumers, they realized that this device they had envisioned came too late in the consumer journey. They made the choice to pivot to a different productwhich is enjoying immense success today because it meets a need and fits perfectly into the routine of consumers.

This is the very essence of UX: you don't cling to your initial idea, you cling to solving the user's problem, even if it means having to go back.

https://xenakeo.com/yuka-app-redesign

Let's dispel some myths: no, you don't need to interview hundreds of people.

Julie: But I imagine many people believe that to achieve this success and for these figures to be true, one must interview... hundreds of people ?

Florine: In fact, we're not looking for statistical representativeness in UX; we're looking for behavioral patterns. A study by Jakob Nielsen, which many have probably already heard of, demonstrated that with just 5 users, we can already identify patterns. 85% of usability problemsBeyond 5, we reach a saturation curve and we begin to observe the same thing.

Of course, it's 5 people per profile/persona. So the important thing is to clearly identify the target audience at the beginning, and then it's a matter of prioritization.

Julie: So a product that targets 4 personas will need to undergo 20 user tests?

Florine: There are two different things. First, of these four personas, there are probably only one or two primary target groups that represent 70-80% of users. Perhaps we should, at least initially, make sure that we are indeed responding to them.

And the second thing is that at UX Republic, we often advise doing less but more often: rather than testing once with 20 people at the end of the project, we prefer to do four sessions of 5 tests throughout the design process. This allows us to make adjustments as we go, without ever straying from the user's needs. And whatever happens, I often say that a test is better than nothing at all. Each test will provide us with valuable insights that will allow us to improve the product.

Julie: What would be your final word to a client who is still hesitating?

Florine: I would tell them: don't be afraid to discover you're on the wrong track. What's costly isn't changing your mind after an hour's test; it's stubbornly persisting in developing a product that no one knows or wants to use. Failing to test means accepting the risk of a "knowledge debt" that will consequently be very costly later on.

sources:

Robert N. Charette, “Why Software Fails”IEEE Spectrum, 2005.
Standish Group, "Exceeding Value", 2014.
Dr. Susan Weinschenk, “The ROI of User Experience”, Human Factors International, 2011.
Barry Boehm, “Software Engineering Economics”, Prentice-Hall, 1981.
Forrester Research, “The Six Steps For Justifying Better UX”, 2016.
Jakob nielsen, “Why You Only Need to Test with 5 Users”, Nielsen Norman Group, 2000.
Jon Picoult, "The Customer Experience ROI Study”, Watermark Consulting, 2019.
Hermann Ebbinghaus, “Über das Gedächtnis”, 1885.