Thanks for the comment, in answer to your question, ‘How much construct validity is there…? This is a big subject and this will only be a brief look at some of the major concerns.

Truthfully, we can’t know how much construct validity there is in a typical usability test but we do recognise that the setting is unnatural and can therefore respond to this by presenting a comfortable friendly environment and behaving in ways that make the participants experience as neutral and as comfortable as possible.

We understand that an audience of impartial observers would effect results, therefore we provide separate viewing facilities for observers where they can talk and discuss the research as it happens. Our test suite and observation room are physically separated (by another room), rather than just a one-way mirror as is typical in market research facilities. We also understand that observation with the facilitator would effect the outcome of research, so our facilitators are trained to keep their spoken language and body language neutral during the research so as to eliminate as much influence as possible.

Incentivising participants could be a problem whilst researching usability if we asking participants if they liked something or not. However we ask neutral open questions to encourage participants to talk about how they use a system or perform a task. Therefore payments are less of an issue. Because we understand that payments my effect participants we try to diminish the effect by paying a reasonable amount of money so people feel valued and by paying prior to the research so participants can set aside concerns about being paid.

I agree that participants are aware that they are being video-ed, we tell them that they’ll be recorded and ask them if they are happy with the arrangement prior to a test. Anecdotally I’d argue that participants concerns about being recorded quickly diminish as they become more engaged with the scenarios they are asked to imagine themselves in. Additionally we understand that much of a participants’ reactivity toward the environment and experience will be around modifying personal characteristics. We are not researching a participant’s personal characteristics we’re researching usability, so their reactivity matters less.

Regarding the physical environment the participant finds themselves in, we try to create as neutral and unimposing environment as possible, we like to give participants time to make themselves comfortable, we offer them a drink and we talk to them about the research and what they are likely to expect during the session to eliminate surprises and concerns about being tested.

Well that begins to answer your question, but we still don’t know why we don’t know how much construct validity there is. Because cxpartners are user experience practitioners rather than academic researchers commercial constraints of time and budget rarely allow us to accurately measure construct validity. Accurate measurement of construct validity would require academic research methods where time was less of a constraint and the research could be more fully explored. Practitioners can not and do not claim to find all the problems with a system but can find the problems that if rectified make a measurable difference to the success of a commercial website, system or piece of software.

I’ll point you to this paper by Gitte Lindgaard – Notions of thoroughness, efficiency, and validity: Are they valid in HCI practice? – http://bit.ly/4xqOUr

I’d be very interested to know what you think?

Walt