Bridging art music, philosophy and perception science, this article proposes a spherical way of thinking; the term evokes an inclination to sense connections between all things, implying a transdisciplinary approach to the world. The tensions between different viewpoints can drive us to think beyond any dichotomy, and uncertainty can inspire creativity. By exposing how this reflects in her work, the author hopes to activate new questions and new ways of addressing those questions.

Among the artists and researchers who have explored music with 3D environments, many have made explicit references to gaming while exploring participatory interaction, video-game player paradigms, and allusive iconography. I have also been using 3D technologies since 2007, but in a quite different way. My 3D software operates based on the sound from an acoustic instrument, and the image is intended to work like a reactive stage scene.

Whatever motivates a practitioner, it is useful to think of how interaction designs and audio-visual relationships drive experience. I will begin this paper with a short description of how 3D engines render sound, and will then proceed with an overview of the diversity of approaches to music in 3D environments. Following this, I will analyse convergences and divergences between the notion of flow in music and gaming. Then, I will draw from perception science and film theory to analyse how audio-visual relationships and the dynamics of sound and image drive attention. Armed with these tools, it is possible to glean how any work exploring music with 3D environments conducts perception.

This chapter proposes a parametric model that is useful in audio-visual instrument design, composition and performance. We can draw a separation between those activities, but in practice that separation might not be so obvious: ultimately, the iterative creation process must always consider the final, global experience. Derived from a perceptual approach, the model is applicable to the broad diversity of aesthetical options and technical platforms. One can equally discard part of the parameters to analyse recorded audio pieces and films. On the one hand, the model enables the separate analysis of performer-instrument interaction, sound, image, audio-visual relationship and physical setup. On the other, it enables the analysis of how the combination conducts the audience’s experience.

The chapter begins by presenting each parameter independently, while illustrating their use with a range of artistic examples. It then explains how their combination facilitates the analysis of expression, of the relative strength of sound and image and of the audience’s feeling of presence. Finally, it demonstrates how the model can be used in creative practice, showing its usefulness as a compositional tool.

When we enter the words interface, music or expression into a search engine, we are presented with many links for "NIME" - New Interfaces for Musical Expression. This term is well known among those dedicated to the invention of new musical systems and instruments. It emerged in 2001 to designate a conference that has taken place annually ever since, bringing together a wide range of creative perspectives. Research into new sonic interfaces is flourishing, but interestingly, the meaning of 'expression' is rarely discussed. This is an important topic, not only because the spectrum of sensibilities and creative motivations is vast, but also because interaction with digital technologies is governed by implicit theories, hidden beneath various layers of code.

In this chapter, we bring together creative practice and perceptual science to set out a notion of sonic expression and outline the ways in which this understanding may converge with or diverge from others. We begin by proposing a method that facilitates the analysis of any sonic performance, regardless of its aesthetic approach or the technology used — whether analogue, digital, or hybrid. This method consists of a parametric model that helps us understand how a performance shapes the audience’s experience.

The term NIME, acronym of New Interface for Musical Expression, applies to a great diversity of creative practices. Nevertheless, the meaning of expression is rarely discussed. In this article we formulate an understanding of expression where the reciprocal interaction between performer and instrument is important, as well as the relation between audition, vision and space. Articulating artistic practice and the science of perception, we describe three creative principles and a parametric visualisation model. The model includes parameters for interaction, sonic and visual dynamics, audio-visual relationship, physical setup and semantics. Those parameters are applicable to any technical platform and aesthetic approach. Our proposed visualisation method facilitates the analysis of how their inter-relationship drives the audience’s experience.

The term ‘spatial presence’ refers to the feeling of presence in a mediated space. This subjective experience has been discussed in media theory, sound art, film and performance. It depends on multiple variables, or parameters. This paper presents a parametric model that can be used to analyze those variables and their relationships. It exposes methods to assess interaction, characteristics of sound and image, audio-visual relationship and physical setup. It also exposes methods to assess how these variables intertwine in perceptual experience. The model draws from perception science, interaction design, music and audio-visual theory. It is applicable to the broad diversity of aesthetical options and technical platforms, facilitating the analysis of spatial presence in any performance. One can also discard part of the parameters so as to analyze installations, sound art and film.

The term New Interface for Musical Expression (NIME) has been applied to a great variety of instruments and systems, since the first NIME conference in 2001. But what is musical expression, and how does an interface intended for idiosyncratic expression differ from ubiquitous interfaces? This paper formulates an understanding where the reciprocal interaction between performer and instrument is important. Drawing from research in perception science , interface design and music, the paper specifies methods that can be used to analyse interaction, attention dynamics and semantics. The methods are applicable to any technical platform and aesthetic approach, facilitating the discussion of creative strategies and the analysis of music experience. The paper uses these methods to describe a NIME that combines an acoustic string instrument and software that operates based on the acoustic sound. The software applies the difference between the detected pitch and the closest tone / half tone to the processing of pre-recorded sounds. The proposed methods help to explain how this NIME enables versatile musical forms, and prevents undesired outcomes.

Mapping digital sound to an acoustic input enables the performer and the software to ‘talk’ simultaneously. Whilst the performer has direct control over the acoustic outcome, the digital can become a means of destabilization - as it is mediated through code. Musical expression substantiates as the performer addresses the unexpected resourcefully. This text describes the performative dynamics in terms of perceptual mechanics.

This article draws a perceptual approach to audio-visual mapping. Clearly perceivable cause and effect relationships can be problematic if one desires the audience to experience the music. Indeed perception would bias those sonic qualities that fit previous concepts of causation, subordinating other sonic qualities, which may form the relations between the sounds themselves. The question is, how can an audio-visual mapping produce a sense of causation, and simultaneously confound the actual cause-effect relationships. We call this a fungible audio-visual mapping. Our aim here is to glean its constitution and aspect. We will report a study, which draws upon methods from experimental psychology to inform audio-visual instrument design and composition. The participants are shown several audio-visual mapping prototypes, after which we pose quantitative and qualitative questions regarding their sense of causation, and their sense of understanding the cause-effect relationships. The study shows that a fungible mapping requires both synchronized and seemingly non-related components – sufficient complexity to be confusing. As the specific cause-effect concepts remain inconclusive, the sense of causation embraces the whole.

This article exposes a perceptual approach to instrument design and composition, and introduces an instrument that combines acoustic sound, digital sound, and digital image. We explore disparities between human perception and digital analysis as creative material. Because the instrument repurposes software intended to create video games, we establish a distinction between the notion of “flow” in music and gaming, questioning how it may substantiate in interaction design. We further describe how the projected image creates a reactive stage scene without deviating attention from the music.

The text reports a study, which draws upon methods from experimental psychology to inform audio-visual instrument design. The study aims at gleaning how an audio-visual mapping can produce a sense of causation, and simultaneously confound the actual cause and effect relationships. We call this a fungible audio-visual mapping. The participants in this study are shown a few audio-visual mapping prototypes. We collect quantitative and qualitative data on their sense of causation and their sense of understanding the cause-effect relationships. The study shows that a fungible mapping requires both synchronized and seemingly non-related components - sufficient complexity to be confusing. The sense of causation embraces the whole when the specific cause-effect concepts are inconclusive.

The chapter formulates an understanding of how an audio-visual instrument can be composed in such a way that the experience is driven through sound organization – modulated, but not obfuscated, by a moving image. This is particularly challenging, as normally the audio-visual relationship is skewed in favor of the visual. The investigation is motivated by insights derived from artistic practice. It outlines psychophysical boundaries with the aid of existing cognition/ attention research, and it describes three principles for the creation audio-visual instruments. As an example, the article describes how they are explored in a specific audio-visual instrument, combining an acoustic zither and modified software from audio processing and video-game technologies. This instrument addresses the three principles while exploring the disparities between an acoustic and a digital output.The chapter formulates an understanding of how an audio-visual instrument can be composed in such a way that the experience is driven through sound organization – modulated, but not obfuscated, by a moving image. This is particularly challenging, as normally the audio-visual relationship is skewed in favor of the visual. The investigation is motivated by insights derived from artistic practice. It outlines psychophysical boundaries with the aid of existing cognition/ attention research, and it describes three principles for the creation audio-visual instruments. As an example, the article describes how they are explored in a specific audio-visual instrument, combining an acoustic zither and modified software from audio processing and video-game technologies. This instrument addresses the three principles while exploring the disparities between an acoustic and a digital output.