What We See and What We Get from Visualization: Eye Tracking Beyond Gaze Distributions and Scanpaths
Technical progress in hardware and software enables us to record gaze data in everyday situations and over long time spans. Among a multitude of research opportunities, this technology enables visualization researchers to catch a glimpse behind performance measures and into the perceptual and cognitive processes of people using visualization techniques. The majority of eye tracking studies performed for visualization research is limited to the analysis of gaze distributions and aggregated statistics, thus only covering a small portion of insights that can be derived from gaze data. We argue that incorporating theories and methodology from psychology and cognitive science will benefit the design and evaluation of eye tracking experiments for visualization. This position paper outlines our experiences with eye tracking in visualization and states the benefits that an interdisciplinary research field on visualization psychology might bring for better understanding how people interpret visualizations.
9740 \vgtccategorySubject Development \vgtcinsertpkg \teaser \CCScatlist \CCScatTwelveHuman-centered computingVisualizationVisualization design and evaluation methods
Eye tracking experiments in visualization research have provided insights into how people interpret and interact with visualizations. In contrast to classic performance analysis, the analysis of gaze behavior provides information about the distribution of visual attention over time and visual strategies employed in interpreting a visualization or in working with a complex visual analytics system. Typical measures derived from gaze data are fixation duration, fixation count, saccade length, and numerous other aggregated values . All measures can be indicators for specific perceptual or cognitive aspects (e.g., cognitive load , working memory ) that are potentially interesting for the assessment of a visualization.
Previously, we surveyed the types of eye tracking studies performed in visualization research  and noticed that mainly gaze distributions, fixation sequences, and comparisons between user groups were analyzed. Back then, we argued that eye tracking can serve “as a method and data source that can
be interpreted from both psychological and visualization perspectives, acting as a bridge between cognitive and computing science” with the long-term goal to build an interdisciplinary scientific community. Since then, collaborations between communities have increased, for example, reflected in several activities at IEEE VIS
Another example is the collaborative research center SFB-TRR 161
We summarize our experience from the aforementioned activities in which we were involved and from our observations of the community. Hypothesis building in eye tracking studies for visualization is often data-driven, based on observations from pilot studies or previous experiments. Similarly, reported results of eye tracking studies are typically summaries of observations. In both cases, theories of perceptual and cognitive psychology are less prominent, although they might help explain specific observations. We see three main reasons for this situation concerning the research background and interests, and the complexity of visualization problems at hand.
Although many design principles are based on perceptual and cognitive theories, in-depth psychological background knowledge is often not part of the education for visualization. Researchers starting with eye tracking studies are confronted with learning eye tracking methodology, which is, starting with proper calibration to a comprehensive analysis of the data, a complex field on its own. As a consequence, deeper knowledge of a whole new research field, i.e., psychology, is hard to achieve within the short time span of an average PhD student’s career.
Psychologists’ core topics are often disconnected from topics relevant for visualization research. Yet, there are some successful examples of combining communities, for example, at the Symposium on Eye Tracking Research and Applications (ETRA). Such events provide great opportunities for interdisciplinary discourse and establishing collaborations. However, publication strategies and research topics might significantly differ between communities. Hence, a fusion of expertise just by project collaborations might cover some research questions, but from a long-term perspective, other solutions are necessary.
The increasing complexity of visual analytics approaches exacerbates an interpretation of how different perceptual and cognitive processes work together when participants interact with a system. Furthermore, study tasks in visualization are often a performance measure to assess the quality of a technique, while in psychological experiments, the task is often just a vehicle to induce cognitive processes. Consequently, more directed research on visualization-specific aspects is necessary, which conflicts with research on generally applicable theories.
2 Visualization Psychology for Eye Tracking
For the aforementioned reasons, we see the need of visualization psychology as a new field in visualization research. Close collaborations with psychologists and cognitive scientists in the education program for visualization researchers promises multiple improvements:
Improving Study Quality
Based on cognitive and psychological models, an eye tracking study might be designed in a way that it focuses more on the research questions and the hypotheses in mind. Given the tasks to be solved and the participant group with lots of involved participant-related properties can make it easier to design such a study since it restricts the number of independent variables to be tested. Hence, knowing more about the psychological aspects can improve the study quality and consequently, the reliability and generalizability of the results.
Theory-based Study Design
A stronger integration of theory in eye tracking study designs has the advantage that the generalizability of the results can be better argued. If one or multiple theories can be linked to specific behavior from interpreting a visualization and interacting with it, the significance of the results is strengthened. Hypotheses derived from observations and supported by theory need to be emphasized in visualization research and should be an essential part of visualization psychology.
Explainability of Observations
Vice-versa, study results can be related to theories to explain the observations. For many observations, we can explain what happens and how something happens. However, another important question is why it happens. Because many user studies already focus on the first two aspects, visualization psychology for eye tracking should focus on the aspects that can potentially answer why people interpret or interact with a visualization in a specific way.
For many visualization problems, a general understanding of how people process the visual stimulus would be desirable. Consequently, cognitive architectures (e.g., ACT-R ) that simulate user behavior, based on theoretical concepts and data-driven methods, would help to evaluate new visualization methods, reducing the effort and cost to perform user studies.
3 Example Scenario: Metro Maps
As an example, we discuss one of our eye tracking studies and how it might benefit from visualization psychology. In a recent eye tracking study, Netzel et al.  describe the results of comparing color-coded and gray-scale public transport maps with the major outcome that color is an important ingredient to reduce the cognitive burden to follow lines. The analysis showed that in the colored map version the participants had much larger saccades and we hypothesize that the colored lines made them feel safer and hence, the route finding tasks could be answered faster and more reliable. On the other hand, in gray-scale maps, the participants’ eyes moved with significantly smaller saccades to trace a line reliably.
The analysis of the eye tracking data focused on gaze distributions, in particular, which regions were investigated how long and in which order. Due to the lack of a cognitive model that explains the interpretation of such maps, we are not able to argue what someone is thinking at the moment of interpreting the map. Finally, we did not include any cognitive models in the eye tracking study to simulate user behavior. This is difficult because the study was one of the first in this domain and a domain expert served as a means to simulate a person who has a high experience level. However, we did not have a variety of different experience levels and the theoretical background to model participants’ behavior in a generalizable way.
4 Future Directions
Visualization psychology could bridge gaps and strengthen and explain observations from eye tracking experiments in visualization. It could be a relevant contribution to visualization and psychology research in general, but eye tracking could benefit in particular due to its complexity and the relationship to understanding perceptual and cognitive processes that are involved in interpreting visualizations and working with interactive systems.
A key question, of course, is: How can we integrate the expertise from both research fields in a common research endeavor? We think that activities such as this workshop or our own experience with the ETVIS workshop and joint research centers (like SFB-TRR 161) are a good way to go, but are alone not sufficient and need further action. Building a research area of visualization psychology could be a viable means, for example, by establishing publication and other presentation opportunities that work for visualization researchers and psychologists alike, by setting up a canon of teaching new students, and by lobbying for funding possibilities for such interdisciplinary work. We also see the need that visualization researchers should reach out into psychology; so far, much effort has been put into bringing psychology into visualization research, but the long-term strategy should strive for a greater balance between the fields.
Acknowledgements.This work was funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 251654672 – TRR 161 (Project B01).
- (1997) ACT-R: a theory of higher level cognition and its relation to visual attention. Human–Computer Interaction 12 (4), pp. 439–462. Cited by: §2.
- (2017) Visualization of eye tracking data: a taxonomy and survey. In Computer Graphics Forum, Vol. 36, pp. 260–284. Cited by: What We See and What We Get from Visualization: Eye Tracking Beyond Gaze Distributions and Scanpaths.
- (2017) Eye tracking methodology: theory and practice. Springer-Verlag, Berlin, Heidelberg. Cited by: What We See and What We Get from Visualization: Eye Tracking Beyond Gaze Distributions and Scanpaths.
- (2011) Eye tracking: a comprehensive guide to methods and measures. Oxford University Press, Oxford UK. Cited by: What We See and What We Get from Visualization: Eye Tracking Beyond Gaze Distributions and Scanpaths.
- (2016) Eye tracking evaluation of visual analytics. Information Visualization 15 (4), pp. 340–358. Cited by: What We See and What We Get from Visualization: Eye Tracking Beyond Gaze Distributions and Scanpaths.
- (2017) User performance and reading strategies for metro maps: an eye tracking study. Spatial Cognition & Computation 17 (1-2), pp. 39–64. Cited by: §3.
- (2020) Toward objective evaluation of working memory in visualizations: a case study using pupillometry and a dual-task paradigm. IEEE Transactions on Visualization and Computer Graphics 26 (1), pp. 332–342. Cited by: What We See and What We Get from Visualization: Eye Tracking Beyond Gaze Distributions and Scanpaths.
- (2009) Uncovering cognitive processes: different techniques that can contribute to cognitive load research and instruction. Computers in Human Behavior 25 (2), pp. 325–331. Cited by: What We See and What We Get from Visualization: Eye Tracking Beyond Gaze Distributions and Scanpaths.