Justifying usability research
Clients often ask me to explain to their boards why we’re running user tests. And why we spend our time user testing mock-ups, rather than the finished website. I thought I’d share my answers, below.
Why user centred design is the solution
User centred design delivers return on investment for two reasons: because it provides a realistic picture of how a website will perform, and because it catches problems early.
IT projects fail when they don’t listen to users
The reason so many IT projects fail to deliver the expected benefits is because they do not take into account whether or not people can use the system that has been developed.
There is a tendency for people inside a project team to assume that their designs are easy to use. This ignores the fact that project teams are generally comprised of experts who have a lot of background knowledge about the products and industry in which they work.
User centred design ensures that right from the start, real users are brought into the design process and that design assumptions are challenged. This leads to better designs that fit users’ needs.
Trust users, not experts
Even ‘expert’ usability reviews are less effective than involving real users. Usability expert Jacob Nielsen estimates that a usability review by an outside expert only identifies 35% of usability problems. Worse, expert reviews sometimes identify issues as being problems when they are not. So time may be wasted ‘fixing’ things which are not broken.
By comparison, Nielsen states that testing with a relatively small group of 8-9 users can identify 95% of problems.
In other words, it’s more effective to run a user test than to trust the judgment of an expert – even if she or he comes from outside the project team.
Catching problems early
The cost of fixing problems rises throughout a project. At the start, often an handful of people are involved in specifying the project. By the end, the number of people has shot up. This means that just getting agreement to fix a problem can be hard. Unpicking work and making a fix is often costly.
Roger Pressman calculates that if the cost of fixing a problem early (before coding) is $1, then the cost of fixing it during coding is $6.50 – a huge leap. Waiting until the code is tested sees costs rise to $15.00. And waiting to fix the problem in version 2 sees the cost rise to $60.00-$100.00.
The cost of failing to identify errors early is punishing. Often, errors just get brushed aside by project teams who are judged on meeting a delivery deadline. This means that, once launched, software often under-performs and costs even more to fix.
It is good sense to spend time at the start of a project in fixing problems.
What’s more, the things you find out about what users do and don’t do will help with all your design decisions further down the track.
That’s why we recommend testing mock up designs, existing websites and even competitor websites before the development phase of a project.
About the author
Giles has been making products more usable for over fifteen years. He was President of the UK Usability Professionals’ Association from 2003-2007 and speaks frequently on usability in the UK and overseas. He writes on usability for Revolution magazine and was one of the editors of the PAS 78 accessibility guide from the British Standards Institute. Email Giles, or call +44 (0)117 946 3930
Subscribe to the RSS feed
Having observed several user tests first-hand in recent months, the sort of insight these lead to can completely change the way you think about what you’re trying to achieve. And the end products were tangibly better for it – both from a specification and development point of view, as well as the end user experience.
Great overview, Giles.
Another really nice argument well packaged for easy client digestion, when it comes to defending the right to create high quality work!
This is a great quick reference summary of why its useful to speak to users and value of upfront research – thanks G!
Using the Agile development methodology, where features are developed in iterations users can test a feature after every stage of development, using the real software in it’s environment rather than a mockup.
Enhancements to the usability of a feature can then be identified and fed back into the list of tasks or stories, prioritised and actioned should the return on investment be sufficient to warrant the additional cost.
This process negates the need for an extensive and expensive up-font specification and mockup generation phase, which identifies all the features and scopes them to optimum usability, and may then turn out to be outside of budget, no longer required further down the line or lack the return on the investment required to achieve them.
Don’t get me wrong, I think user centred design is of massive importance, you just do not need to do it upfront. The Agile development process welcomes and actively encourages change during the development life cycle.
The Standish Group has researched 40,000 IT projects in the United States over 10 years between 1994 and 2004 and compiled the findings in the “CHAOS Chronicles” report. It found that still, two thirds of all projects either fail totally, or are over time, over budget and/or lacking critical features and requirements.
However, success rates have improved more than 100% over the period, the primary reason for which, according to the Group Chairman, Jim Johnson, is “Doing projects with iterative processing as opposed to the waterfall method, which called for all project requirements to be defined up front, is a major step forward.”
What are your thoughts on this?
I like Agile – it’s focussed on communication, rather than documentation, on iteration rather than dictation and on outcomes rather than specifications. In my experience those things make for better projects.
You still need to involve the users up front, though. You shouldn’t start building something unless you know it’s the right thing for the users.
If you’re running an agile project where the work is done in short iterative chunks of a couple of weeks then, yes, you can get user feedback throughout the iterative build process.
One common way of doing this is to have the user centred design people working one iteration ahead of the developers. So when the developers come to their iteration, the knowledge about what will work is ready for them – just in time.
But the topic of how best to integrate user centred design into Agile is far from closed. There’s loads of interest in it at the moment and user experience conferences are full of success stories and cautionary tales.