On Wednesday 20th May, I attended the UK UPA event at the Bunnyfoot Offices in Westbourne Studios. I must say how much I loved their offices, especially the open plan lower area and the fact dogs were actually permitted. Slightly dubious location but really nice once inside.

The debate was about the value of Eye-Tracking. Robert Stevens, the co-founder of Bunnyfoot seems to think it’s something we absolutely must do as part of any usability testing. Kara Pernice, managing director of Nielson Norman Group believes it’s just not necessary.

After a quick coin toss, Kara presents her viewpoints first. She starts by saying that she likes eyetracking for a number of reasons, including the fact that eye tracking gets people involved through it’s geeky propaganda. There are also a number of things you can learn like obstacles, magnetism and reading patterns. However, there are no required usability methods!

From research Nielson Norman Group covered, people tend to use server log analysis the most when conducting usability research. Companies tend to do this upwards of ten times a year. After that, there tends to be a big drop-off with card sorting, user surveys and usability testing the next popular methods. These tend to be done around three to four times a year on average.

In terms of cost, user testing costs on average around US$5600 and eyetracking costs about US$156000. Paper prototyping was about US$12000 and Field Studies coming in at $28000. Given that a round of user testing with five participants will pick up the majority, around 80%, of problems, it’s often a case of economics when choosing not to do eye-tracking. 10% is currently the recommended amount for usability budget spend on testing.

Participants do a lot of activities which can skew the results of eyetracking. Kara also recommended testing full processes rather than just single pages. Eyetracking is hard to set up and requires significant time investment for analysis. It’s also hard to ask people to articulate why they looked or didn’t look at something.

Heatmaps are easy to create but easy to misinterpret. Kara recommend 30 participants for eyetracking to create accurate heatmaps or 6 participants to just study gaze replay.

Next up was Robert Stevens. Bunnyfoot uses the Post Experience Eyetrack Protocol or PEEP for short. Jared Spool once said “PEEP is a voodoo technique IMHO”. Other people have said that eyetracking is a presentation layer that adds no value.

Robert insists the technology is NOT the methodology. Why does he recommend eyetracking? Better user tests, optimisation and engaging stakeholders. He continues that a lot of our behaviour exists in the subconscious. The Talk Aloud Protocol or TAP can intefere with primary task by increasing attentional demands (Russo, J., Johnson, E., Stephens, D., (1989) The Validity of Verbal Protocols, Memory & Cognition,
Vol. 17) and users find thinking aloud difficult and that it makes them feel uncomfortable (Neilson).

PEEP supposedly discovered more issues than normal testing according to these results here. This caused quite a storm in the audience about the validity of this claim.

Robert also showed a session of young child, who was linked to an eyetracking machine whilst looking at the Teletubbies website. The child looked at something and then decided where they wanted to go. Is this an example of cognitive process occurring perhaps? They are currently doing research around this with Reading University.

Robert finished that it’s not just an eyetracker, when you can actually use it properly, that it’s a neuro tool.

There was lots of discussion and a little controversy that followed the Robert and Kara’s talks. Robert tended to be on the receiving end on some of the harshest questions and comments. He handled them pretty well though. At the end of the day, my position remains unchanged. Eye-tracking can be useful but so can other cheaper methodologies. If money isn’t an issue, eye-tracking could certainly benefit the end result but I’d hardly consider it essential usability method.

Thanks to the UK UPA, Bunnyfoot, Kara and Robert for what was an entertaining and insightful session.