User Testing: How a couple of cups of coffee saved me thousands

If you found out right now that there was just one thing you could do that would save time and money and would improve your digital products all at once, you’d do it right?

Actually, you probably wouldn’t and here’s why…

I know this because I still see agencies and clients big and small still making the same mistake. The same one I made for years, and still make it with reluctance, when I can’t convince teams to do it!

The answer is simple: You need to involve users, real users, in your product design and development.

Guerilla Testing

Contrary to what you might think, testing doesn’t need to be expensive. A few years ago, me and a friend sat in a bustling coffee shop with an idea on the table. I’d crafted some wireframes and that’s all we had. We offered to buy people’s coffee in return for 2 minutes of feedback. 

In the politest way, they all told us the idea wasn’t very good or they didn’t understand it. They were in a rush and they were time poor. Perfect because that was the idea the product was trying to solve. It cost us about £20 to conduct the test and without this, we’d have designed and built a prototype. Probably at a cost of £2000, maybe even more and the outcome would have been the same.

Instead, we got our time back. And we saved money.

The rule of 5

As a rule you need to speak to 15 people to identify all of the usability issues with your product, but the majority of those issues will be found by running tests with just 5 people.

It’s still important to speak to 15 people, but it makes more sense to speak to 5 people, at 3 different times. Based on the graph above, by the time you speak to people 12, 13,14 and 15 you’ll largely be hearing the same things.

By breaking it into 3 groups of 5 people, this allows you to conduct a larger number of small experiments, spreading the learnings out over 3 groups. If you look at the curve, each user testing session you’ll find the vast majority of those insights from the first 5 people.

Though the curve is interesting, there is one thing you need to pay attention to. It’s not the % of problems found after 5 people or the % of problems found after 15 users. The most important thing is the % of usability issues you’ll find after 0 user tests.

Companies who put human-centered design methods at the core of their growth strategy see a 228% higher ROI

“Build it and they will come”. No, they won’t.

This is one of those adages that belongs in 2000 – right before the dot com bubble burst. Back then, you could build something and put it out there and see people seemed to start using it with limited marketing or promotion. This had something of a resurgence in 2008 when Apple launched the App Store. 

With limited Android competition and not a huge number of apps, you could build something fairly basic and generate a load of downloads mainly because if the app wasn’t entirely useless, you’d show your mates in the pub who would then download it. (Remember that beer app that you could “drink” from your phone? Ah, the good old days.)

But those times are over and have been for a long time. Big brands now throw entire departments behind the PR and promotion of their apps and still struggle for downloads. The proliferation of WordPress, Squarespace and the discipline of SEO means you can no longer rely on people stumbling across your website. Influencer marketing now means even ‘word-of-mouth’ is a paid for discipline.

So don’t build it and expect people to come. Build something you know has fundamental value to users. It will make your marketing budget go a lot further, it will mean influencers will want to promote your app. User testing gives you an unfair advantage over your competitors, so why not take advantage?

Beware of “Confirmation Bias” and embrace criticism.

There have been a number of times that I’ve spoken to clients who have said “we tested it at our user conference and people loved it!”. Of course they did – they were there to advocate the product!

Furthermore, you see questions like “can we show you our new website?” Such phrases create confirmation bias – the idea that you’re asking the user to confirm your thoughts as opposed to seeing if they actually test the effectiveness of the site.

I have confidence that We Are Star create visually impressive sites. I don’t doubt we create well organised easy to use sites.  But my opinion doesn’t really matter when stats tell me users are not doing the thing we want them to do; whether that’s downloading something, signing up to something or buying something.

User testing needs to validate that users can do the thing you want them to do. A pretty site with a clever navigation is useless if it’s not converting. An ugly site that seems confusing is far more beneficial for a business.

Don’t enter into using testing with an ego. Consider it a success if you find out where your site fails. You want to see where users struggle or where they do things you could never have predicted. Walking out of user testing with loads of positive comments might be great for your ego, but walking out with a list of things to fix to improve your product is the whole point.

Save Your Money!

Obviously, saving money by not building something, like in my example earlier, is an obvious way of keeping costs down. There is no point building something that no one will use, but there’s more cost benefits to user testing, even once you’ve committed to building something.

It is estimated it can cost up to 100x more to fix something when it’s built than getting it right in the first place. An estimated 50% of engineering time is spent fixing things, doing rework.

Yes, some rework might be needed, the QA process will pick up some issues and bugs will pop up, but QA testing is unlikely to pick up confusing navigation, incorrect assumptions or building features no one really cares about.

Conclusion

The value of user testing is clear: better ROI, more usable products and cost savings. The benefits seem endless, yet clients still seem reluctant to take it seriously. Why?

Well one thing seems to be vanity. When we want a new product, we want the product we want and users telling us it’s not what they want feels counterintuitive, almost insulting. 

There is also the argument that users don’t always know what they want. Users know what they know, but a new site can offer something different.

Countless times we hear that a brief has been pulled together ‘with the user in mind.’  But no amount of user understanding can replace actual human behaviour, actual human confusion, building features that were great in the marketing brainstorm but no user actually wants.

So next time you’re building a product, before you spend a penny on development, knock together a few wireframes and spend an hour in a coffee shop. It could be the most valuable Americano you’ve ever had.

So stop building things no one will ever use. Test with users at every stage. Listen to potential users. They are your superpower.