- 02 Jun 2011
- 7 Comments
Following on from James’s post iPad app user testing : Observations from the field, this post reveals the equipment we’ve refined to collect iPad test data and how to deliver an engaging experience for the project owners.
Testing apps and websites on the iPad
From our perspective the killer feature that the iPad 2 introduced was screen mirroring. The question was how to take full advantage of following the user’s journey in real time. And how to do this without Jail Breaking (hacking) the iPad.
Our goals for iPad user testing:
- The participants experience of using the iPad has to be as natural as possible. That meant no cameras fixed to the iPad that stop people turning or tilting the iPad, or cameras mounted on glasses or other intrusive off putting equipment.
- The clients experience of observing the tests has to be rich and engaging. This meant: screen mirroring, video of the user (just like website user testing) and an overview camera so we can see how the participant is using the iPad e.g. gestures, tilting, turning, getting closer to read small text etc.
- We had to gather rich material to analyse. An overhead or document camera just wasn’t good enough.
- We had to be able to easily analyse the data. That meant getting all the video and audio in one easily managed file, no nasty synching of separate files for us thanks.
- The gear had to be easy to use. Simple software and simple equipment.
Our iPad user testing solution
Our iPad test equipment meets all of these requirements and worked out of the box, really simple. Anyone who’s seen me cursing MS Windows will understand how much I like simple.
The equipment / software list:
- Apple iPad 2
- Apple iPad – VGA cable
- Epiphan VGA2USB frame grabber
- HD webcam
- Quicktime Player
- Silverback
- Floor standing boom mic stand
Our iPad user testing kit
The simple setup:
- First, get your HD webcam mounted on a boom mic stand, it lets you position the webcam over the action and isolates it from vibration.
- Now open up Quicktime Player and select the webcam as your source. You won’t use Quicktime to record, just to show the webcam picture on your laptop screen.
- Next plug the VGA cable into your iPad2, plug that into a VGA splitter. One side will feed the screen in your observation room. The other will get plugged into the Epiphan VGA2USB frame grabber.
- Plug your VGA2USB frame grabber into your laptop and open up the frame grabber software that comes from Epiphan. It works a treat and you can see the iPad screen mirrored on your laptop.
- So, now you’ve got a webcam feed and the mirrored screen of the iPad on your laptop screen.
- Finally open up Silverback and set up a new project. Your going to use Silverback to record the screen and provide a picture-in-picture of the participant using the iPad.
- Position the laptop in front of the participant. Set up a Silverback session and dim the laptop screen so the participant can’t see anything on the screen. Don’t worry it’ll still record all the action.
- That’s it, hit record on Silverback and your away.
What you get
Silverback will record the iPads mirrored screen, the webcam feed and the picture-in-picture and the audio of the test in one file. Nice and tidy.
The resulting iPad usability test video contains rich compelling evidence for your analysis. and provides great contextual information about what the participant did, how they did it and why they did it.
The output from iPad user testing sessions
Experience
cxpartners have heaps of experience testing iPad prototypes, apps, websites, mobile sites and have conducted exploratory research with people about what, when and why they use iPads and smart phones instead of laptops and desktops. Why not find out more about what we know.
Note:
The photo below illustrates my comment made on March 2 2012. It shows an example of the simple handheld test equipment we use to allow as natural environment as possible.
Nik, testing iPad apps
hi Walt.
This sounds very interesting. We are thinking using the same technique as you described for a project at our university.
Did you use the standard VG2USB frame grabber from Epiphan? How was your experience in terms of video quality and framerate? Or would you recommend to go for a frame grabber with a higer resolutions/fps such as the LR version from Epiphan (which costs more than the double)?
Hello Ivo,
Good question. We used the standard VG2USB frame grabber in our setup.
Recently, we’ve questioned the value of capturing the device screen. It does after all end up being a video of the the screen changing but with no sense of the participant interacting with it. Instead, we’ve concentrated on improving the quality of the video of the participant’s hands/fingers using the device.
The outcome is that the device is untethered from wires, allowing the participant full freedom of movement. We feel the way the devices are being used is more natural and the gestures we see are less restricted.
Thanks for the comment, what do you think?
Walt
Hi Walt
Good point. I had a similar thoughts while thinking about the use of having the screen captured. I also was unsure about if Silverback is capturing every gesture on the device and marking every single touch with a highlight as Silverback would do when using it directly on the Mac with the mouse pointer? It seems this (important) piece of information is missing?
For us it is more important to get answers about which gesture the user is trying to use than having his facial impression captured. This gives us more valid information than a girn. :)
The only downside I can think of, is that the happening on the screen won’t be fully captured and people in the observatory room won’t be able to see what’s going on, when the user is changing the angle of the device. I also could imagine that having a nice screen capture is always handy when you need to present your results to a audience. What is your experience?
Best,
Ivo
Hi Ivo,
You’re right to think Silverback does not capture the interactions with the mobile device, only those interactions with the Mac itself.
The videos of a participant with an untethered device are not the same as a perfectly still screen capture video.
My feelings are, backed up by feedback from clients, that the greater naturalism of the device being used untethered is of greater value than a picture perfect video.
It pays to pilot the tests thoroughly and practice getting the camera into a position where it gets a great shot but is at it’s least intrusive. Oh, and put anti-glare film on the device that helps get a better quality picture too.
Thanks,
Walt
Very informative post.
We’re about to embark on some Android testing any suggestions as to what set up we should use would be really helpful?
Our goals are very similar to the ones you outline above.
1: For the participants experience to be as natural as possible. We don’t want cameras and arms attached to the device. People will be using their own phones.
2: We have an in-house usability lab with a small obs room. It’s important the team can see what is happening on the phone, the gestures people are using,
3: Ideally easy to analyse data. With audio and vid in one place.
4: Simple easy to use set up. I only have 30 mins between each session to gather my thoughts, write up a top 5 of the last session and reset everything.
Thanks
// Jo
Hi Jo,
Thanks for your question. Happy to share what we’ve doing, in truth our setup has got simpler still since I posted this last summer.
Rather than focus on the position of the camera e.g. sleds, rigs, document cameras – we use an HD webcam on the end of a microphone boom positioned at about head height of the participant (who’s sitting down). As the participant tips their head forward to focus on the phone, the webcam falls outside their peripheral vision. Webcams are good at capturing a reasonably wide angle shot so the participant has total freedom to move the phone naturally. Their webcam ability to focus at this distance is perfectly adequate to capture the dynamic use of the phone.
We use the VLC2.0 app on our Macs to show the webcam video, we use Silverback to capture the Mac screen and the audio of the conversation in one file. VLC is neat as it has some nice controls for tweaking the quality of the webcam video.
We simply mirror the screen of the Mac in the test room to the observation room, everyone can see what’s happening.
We’ve taken to using PearNote to record notes and a separate audio track. Great because it timestamps the notes so you can skip to interesting bits in the Silverback video later.
Oh, and I use MPEG Streamclip to edit and export video clips, once it’s exported out of Silverback.
I’ll try and post a photograph of a test session to illustrate what I’ve rambled on about here.
Be great to know how you use this and develop it further.
Thanks,
Walt
Leave a comment