The use of eye mark recordings to support verbal retrospection in software testing

The work of eight computer novices on a number of closely defined problems with a PC text editor was recorded with an ordinary video camera as well as with mobile equipment for eye mark recording. Immediately afterwards, each recording was used to support the subjects in two separate retrospections on their task performance. The verbal protocols form this session were scored for the occurrence of manipulative, cognitive, and visual operational comments. A comparison showed the eye mark retrospectations to contain 50% more visual operational comments overall and to be slightly more problem-focused than the video retrospections. The video retrospections did, however, show the highest number of manipulative comments, whereas the number of cognitive comments was almost identical for the two kinds of retrospection. Eye mark recordings may thus satisfy the need for elaborative information on visual strategies without a concomitant loss of comments on the cognitive processes. Examples from the protocols illustrate how retrospection based on eye mark recordings can yield information which is of practical value for iterative software development and knowledge elicitation. The validity of the comments on eye mark recordings is discussed on the basis of experience with two recordings, which, unknown to the subject, had actually originated form another subject.