Doctoral thesis
A Perceptual Approach to Audio-Visual Instrument Design, Composition and Performance. Goldsmiths, University of London, 2016
Editorial work
Director / editor/ designer of the LIVE INTERFACES journal , ISSN 2975-9943
See the first edition
See the second edition
INTER-FACE: International Conference on Live Interfaces 2014 (ICLI 2014), ed. Adriana Sá, Miguel Carvalhais, Alex McLean, pub. Porto University, CECL & CESEM (NOVA University), MITPL (University of Sussex), 2015. ISBN 978-989-746-060-9
Preface discussion “Live Interfaces: Seeds of Debate” by Adriana Sá, Joel Ryan, Edwin van der Heide, Atau Tanaka, Andrew McPherson, Thor Magnusson, Alex McLean, Miguel Carvalhais and Mick Gierson.
Book chapters and articles
Transdisciplinarity, Composition, Expression: Reflections of a Spherical Way of Thinking.
In Leonardo MIT Press, 2023. DOI: https://doi.org/10.1162/leon_a_02472 .PDF
Abstract
Bridging art music, philosophy and perception science, this article proposes a spherical way of thinking; the term evokes an inclination to sense connections between all things, implying a transdisciplinary approach to the world. The tensions between different viewpoints can drive us to think beyond any dichotomy, and uncertainty can inspire creativity. By exposing how this reflects in her work, the author hopes to activate new questions and new ways of addressing those questions.
|
A Parametric Model for Audio-Visual instrument Design, Composition and Performance.
Adriana Sá & Atau Tanaka. In Live Visuals:
History, Theory, Practice. Routledge, 2022 .
Abstract
This chapter proposes a parametric model that is useful in audio-visual instrument design, composition and performance. We can draw a separation between those activities, but in practice that separation might not be so obvious: ultimately, the iterative creation process must always consider the final, global experience. Derived from a perceptual approach, the model is applicable to the broad diversity of aesthetical options and technical platforms. One can equally discard part of the parameters to analyse recorded audio pieces and films. On the one hand, the model enables the separate analysis of performer-instrument interaction, sound, image, audio-visual relationship and physical setup. On the other, it enables the analysis of how the combination conducts the audience’s experience.
The chapter begins by presenting each parameter independently, while illustrating their use with a range of artistic examples. It then explains how their combination facilitates the analysis of expression, of the relative strength of sound and image and of the audience’s feeling of presence. Finally, it demonstrates how the model can be used in creative practice, showing its usefulness as a compositional tool.
|
Sonic Expression, Mediation and Perception (in Portuguese) .
In The Sonic Experience: from linearity to circularity. Documenta, 2023 .
A Method for the Analysis of Sound Art and Audio-Visual Performance.
In Audiovisual e Industrias Criativas: presente e futuro vol. 1, McGraw Hill Education (2021): 575-589..
Abstract
The term NIME, acronym of New Interface for Musical Expression, applies to a great diversity of creative practices. Nevertheless, the meaning of expression is rarely discussed. In this article we formulate an understanding of expression where the reciprocal interaction between performer and instrument is important, as well as the relation between audition, vision and space. Articulating artistic practice and the science of perception, we describe three creative principles and a parametric visualisation model. The model includes parameters for interaction, sonic and visual dynamics, audio-visual relationship, physical setup and semantics. Those parameters are applicable to any technical platform and aesthetic approach. Our proposed visualisation method facilitates the analysis of how their inter-relationship drives the audience’s experience. |
The Variables of Spatial Presence: a Parametric Model.
Adriana Sá & Atau Tanaka. In xCoAx 2019 Proceedings.
Abstract
The term ‘spatial presence’ refers to the feeling of presence in a mediated space. This subjective experience has been discussed in media theory, sound art, film and performance. It depends on multiple variables, or parameters. This paper presents a parametric model that can be used to analyze those variables and their relationships. It exposes methods to assess interaction, characteristics of sound and image, audio-visual relationship and physical setup. It also exposes methods to assess how these variables intertwine in perceptual experience. The model draws from perception science, interaction design, music and audio-visual theory. It is applicable to the broad diversity of aesthetical options and technical platforms, facilitating the analysis of spatial presence in any performance. One can also discard part of the parameters so as to analyze installations, sound art and film. |
Designing Musical Expression.
In xCoAx 2017 Proceedings
Abstract
The term New Interface for Musical Expression (NIME) has been applied to a great variety of instruments and systems, since the first NIME conference in 2001. But what is musical expression, and how does an interface intended for idiosyncratic expression differ from ubiquitous interfaces? This paper formulates an understanding where the reciprocal interaction between performer and instrument is important. Drawing from research in perception science , interface design and music, the paper specifies methods that can be used to analyse interaction, attention dynamics and semantics. The methods are applicable to any technical platform and aesthetic approach, facilitating the discussion of creative strategies and the analysis of music experience. The paper uses these methods to describe a NIME that combines an acoustic string instrument and software that operates based on the acoustic sound. The software applies the difference between the detected pitch and the closest tone / half tone to the processing of pre-recorded sounds. The proposed methods help to explain how this NIME enables versatile musical forms, and prevents undesired outcomes. |
Exploring Disparities
Between Acoustic and Digital Sound.
In Leonardo Transactions, vol. 48 No. 3 , MIT Press (2015): 280-281.PDF
Abstract
Mapping digital sound to an acoustic input enables the performer and the software to ‘talk’ simultaneously. Whilst the performer has direct control over the acoustic outcome, the digital can become a means of destabilization - as it is mediated through code. Musical expression substantiates as the performer addresses the unexpected resourcefully. This text describes the performative dynamics in terms of perceptual mechanics. |
The Fungible Audio-Visual Mapping and its Experience.
Adriana Sá, Baptiste Caramieux and Atau Tanaka. In Journal Of Science And Technology Of The Arts vol. 6 No. 1 (2014): 85-96. p-ISSN: 1646-9798 | e-ISSN: 2183-0088
Abstract
This article draws a perceptual approach to audio-visual mapping. Clearly perceivable cause and effect relationships can be problematic if one desires the audience to experience the music. Indeed perception would bias those sonic qualities that fit previous concepts of causation, subordinating other sonic qualities, which may form the relations between the sounds themselves. The question is, how can an audio-visual mapping produce a sense of causation, and simultaneously confound the actual cause-effect relationships. We call this a fungible audio-visual mapping. Our aim here is to glean its constitution and aspect. We will report a study, which draws upon methods from experimental psychology to inform audio-visual instrument design and composition. The participants are shown several audio-visual mapping prototypes, after which we pose quantitative and qualitative questions regarding their sense of causation, and their sense of understanding the cause-effect relationships. The study shows that a fungible mapping requires both synchronized and seemingly non-related components – sufficient complexity to be confusing. As the specific cause-effect concepts remain inconclusive, the sense of causation embraces the whole. |
Repurposing Video Game Software for Musical Expression: a perceptual approach.
In Proceedings of New Interfaces for Musical Expression 2014 (London, 2014): 331-334.
Abstract
The article exposes a perceptual approach to instrument design and composition, and introduces an instrument that combines acoustic sound, digital sound, and digital image. We explore disparities between human perception and digital analysis as creative material. Because the instrument repurposes software intended to create video games, we establish a distinction between the notion of “flow” in music and gaming, questioning how it may substantiate in interaction design. Furthermore, we describe how the projected image creates a reactive stage scene without deviating attention from the music. |
A Study About Confounding Causation in Audio-Visual Mapping.
Adriana Sá, Baptiste Caramieux, Atau Tanaka. In xCoAx 2014 Proceedings (Porto, 2014): 274-288
Abstract
The text reports a study, which draws upon methods from experimental psychology to inform audio-visual instrument design. The study aims at gleaning how an audio-visual mapping can produce a sense of causation, and simultaneously confound the actual cause and effect relationships. We call this a fungible audio-visual mapping. The participants in this study are shown a few audio-visual mapping prototypes. We collect quantitative and qualitative data on their sense of causation and their sense of understanding the cause-effect relationships. The study shows that a fungible mapping requires both synchronized and seemingly non-related components - sufficient complexity to be confusing. The sense of causation embraces the whole when the specific cause-effect concepts are inconclusive. |
How an Audio-Visual
Instrument Can Foster the Sonic Experience.
In Live Visuals, eds. L. Aceti,
S. Gibson, S. M. Arisona, O. Sahin, Leonardo Almanac vol. 19 No. 3, MIT Press (January 2013):
284-305. ISBN: 978-1-906897-22-2. | ISSN: 1071-4391
Abstract
The chapter formulates an understanding of how an audio-visual instrument can be composed in such a way that the experience is driven through sound organization – modulated, but not obfuscated, by a moving image. This is particularly challenging, as normally the audio-visual relationship is skewed in favor of the visual. The investigation is motivated by insights derived from artistic practice. It outlines psychophysical boundaries with the aid of existing cognition/ attention research, and it describes three principles for the creation audio-visual instruments. As an example, the article describes how they are explored in a specific audio-visual instrument, combining an acoustic zither and modified software from audio processing and video-game technologies. This instrument addresses the three principles while exploring the disparities between an acoustic and a digital output. |
|