A Perceptual Approach to Audio-Visual Instrument
Design, Composition and Performance

This research consists of a perceptual approach to audio-visual instrument design, composition and performance. The approach informed practical work as well as a parametric visualisation model, which can be used to analyse sonic expression, sensory dominance and spatial presence in any audio-visual performance language. The intent of the practical work was for the image to function as a stage scene that reacts to the sound, without distracting attention from the relation between the sounds themselves. This is challenging, because usually vision dominates over audition. 

To clarify the problem, I bridged audio-visual theory, psychology, neuroscience, interaction design and musicology. The investigation led to three creative principles: a) to threshold control and unpredictability in order to convey sonic expression; b) to dispense with disruptive visual changes and apply Gestaltist principles to visual dynamics, so as to facilitate perceptual simplification; and c) to create an audio-visual relationship that produces a sense of causation, and simultaneously confounds the cause and effect relationships. The latter principle was also demonstrated with a study on audio-visual mapping and perception, whose conclusions are equally applicable to the audio-visual relationship in space.

My three creative principles informed the development of an instrument that combines a zither and audio-visual 3D software, exploring disparities between the acoustic and digital outputs. The parametric visualisation model can represent how any audio-visual performance work might converge or diverge from these principles. It combines parameters for interaction, sonic & visual dynamics, audio-visual relationship, physical performance setup and semantics.