Abstract

With an increased creation of and access to data, and a growing demand on humans in analytical and decision-making processes, there is a need to further facilitate perceptual and cognitive abilities to support these processes. An approach to support these tasks is to leverage more sensory systems, such as the auditory, to increase information retention to get a more comprehensive understanding of a dataset or situation, while also involving more senses to further engage the user. Audiovisual data interfaces enable the distribution of data variables to any of the two senses to reduce the risk of cognitive overload, or highlight specific data variables by mapping them to both sensory modalities. However, the success of an audiovisual data interface is dependent on the integration of the two senses and how this is utilized in the resulting interface.

This thesis contributes to sonification research by taking a human-centered approach to integrating sonification and visualization. The human-centered approach involved working with domain experts and users during the design process of the sonification, and to create perceptually motivated designs by utilizing how the auditory and visual systems complement each other and how they are integrated in cognition for sense-making.

The thesis contains six studies that explored the integration of sonification and visualization, ranging from literature surveys to design-oriented studies. A state-of-the-art report provided a survey on the integration of sonification and visualization to offer an introduction to the field to new and current practitioners. In another study, a conceptual framework for scene analysis was developed to support the design and analysis of audiovisual data representations. The rest of the studies were design-oriented, where each focused on a specific aspect of how sonification can complement visualization, depending on the domain and tasks of the study. Other than providing concrete examples of audiovisual integrations, the main contributions of the design-oriented studies are provided in their evaluation results in the form of design recommendations. These include the use of redundant mappings for multi-dimensional data analysis, and considering subjective differences of domain experts for situational awareness support in air traffic control. The design-oriented studies also compared sonification designs, where one study showed a potential trade-off of using informative or pleasant designs for astronomy science communication. Another comparison showed that concrete sonification designs can complement abstract visualizations. Lastly, the thesis provides design considerations by comparing the level of redundancy, indexicality, and complexity of each sonification design through the developed conceptual framework. Overall, this thesis offers motivated recommendations for the integration of sonification and visualization.

Citation

Acknowledgements

Resources

•  Publisher
•  Publication (PDF)