Understanding Modal Synergy for Exploration

Abstract

Photone is an interactive installation that explores the sonification of visual images, enabling users to experience a fusion of color, sound, and interaction. By mapping hue and brightness values of image pixels to musical parameters, Photone creates a sensory experience where visual features are translated into auditory expressions. This work introduces the concept of modal synergy, where visual and auditory modalities integrate into a cohesive and emergent experience. Observational studies reveal that users engaging with Photone for extended periods focus on subtle, syntactic visual features rather than semantic content. To deepen understanding, we plan qualitative interviews with expert users, analyzing their interactions and reflections to uncover the nuanced experiential qualities of modal synergy.

Citation

Resources