- 02 Jun 2010
- 1 Comment
After attending many sessions during a wonderful UPA conference in Munich, one sticks firmly in my mind.
As a consultant one often gets asked the same questions by different clients. Certain subjects come up time and time again and we have well worn answers to alleviate concerns.
It’s like being a professional golfer (I wish). Tiger knows what club he needs from 235 yards and I have answers to these common questions. But sometimes it’s good to challenge your preconceptions and a session by Rolf Molich, Whitney Quesenbury, Jakob Biesterfeldt and Karen Bachmann did just that.
Their session “5 users will find 85% of the usability problems – and other myths about usability testing” challenged us to consider nine ‘myths’ to see if we agreed or disagreed with them.
Have a scan through them and make a mental note of where you stand.
- Five users are enough to catch 85% of the usability problems in practically any product.
- The main goal of a usability test is to discover usability problems.
- Usability testing can be conducted by anyone.
- Expert reviews provide results that are as reliable as those from usability tests.
- Participant selection should represent market segments, and demographics are important.
- It’s better to use an outside consultant to moderate usability tests because they are more objective.
- Eye-tracking shows us what users see, so it reveals important issues that usability testing without it can’t.
- Observers should be kept in a separate room to prevent interference with the test.
- A recommendation for improvement should be provided for each problem finding.
After being asked each question in turn, over one hundred usability professionals raised a piece of coloured card to convey their opinion.
What was interesting was just how varied the opinions in the room were on key points such as the ‘Five users myth’. This ‘myth’ stems from an article by Jacob Nielsen entitled ‘Why you only need to test with 5 users” in which he uses some confusing looking maths (which must be true) to comfort us that 5 users will find 85% of problems.
Molich disputed this by referring to his own work on ‘Comparative Usability Evaluations’ which involved lots of usability professionals testing the same thing. Molich proposed a revision to this myth by stating that five users are enough to use within an iterative cycle of testing.
Another contentious point was around expert reviews vs usability testing. Whitney Quesenbury dealt with this one beautifully by saying “An expert review is not a usability test, if you do a usability test you get an expert review too”. We all nodded sagely when the point was made regarding the superior political impact of a user test.
The problem with an expert review is that it’s just another opinion. The beauty of a user test is that you can say “what we saw was” as opposed to “I think that” and it’s statements like that that people use to make informed decisions within web projects.
Much as Tiger Woods has probably challenged his preconceptions recently, why not take a moment to to jot down your position on each of these statements and then discuss them with your colleagues. Did you change your mind or can you justify the opinion you hold?
I changed my mind on a few of these and I’m sure you will too.
Thanks for a nice article. If anyone would like to see the views of the audience in Munich on these issues, I took a photograph for each of the questions showing the colored papers being held up in the air.
http://www.flickr.com/photos/whitneyq/sets/72157624169191424/
As one of the panelists, I found the discussion very interesting, with good points made about each of the myths from both the stage and floor.
What was most interesting was the last question. Rolf asked how many people had changed their minds based on the discussion. We expected that most had come with firm beliefs and that they would not change much, so we were surprised at the large number of hands that went up.
Leave a comment