adriana  sá
trans-disciplinary   music

HOME PROJECTS INSTRUMENTS AUDIO SCORES VIDEO PUBLICATIONS BIO
AUDIO-VISUAL RESEARCH
Research summary

 

AG#1: repurposing video game technologies
in combination with a zither

Screen shot 2013-04-23 at 4.10.39 PM.png

Screen shot 2013-04-23 at 4.15.19 PM.png

AG#1 is the first version of the software from my audio-visual instrument, which started being developed in 2008 - prior to any scientific research. The instrument combines acoustic sound, digital sound, and digital image. The software operates based on amplitude and pitch detection from a zither with custom tuning and strings. This is also the case with the subsequent software versions, Arpeggio-Detuning and AG#2

AG#1 was a modification of software originally written by John Klima (2007), ironically titled Portuguese Guitar Hero after the popular Guitar Hero video game series. AG#1 uses a 3D engine called Torque and an audio library for video game programing, FMOD.

3D software can create a sense of depth, and extend one's sense of spatial presence into a digital world beyond the screen. I wanted to use a video game platform, and simultaneously make it evident that my creative concerns and aesthetics are very distant from gaming. I modified the software without changing the code structure, yet these modifications circumvented design theories embedded at low level in the digital architecture of the game engine.

When AG#1 detects audio, it changes the 3D camera view and places virtual sound emitting objects in the 3D world. Their position in the 3D world affects the spatialisation of related sounds. I parameterised these digital functions, created a new 3D world, new sounds and new mapping strategies.

Bellow is a description of how the intrument explores the creative principles of sonic complexity, visual continuity  and audio-visual fungibility, as well as a demonstrative screen capture and performance documentation.

Related article:

Adriana Sa. How an Audio-Visual Instrument Can Foster the Sonic Experience. In Live Visuals, eds. L. Aceti, S. Gibson, S. M. Arisona, O. Sahin, Leonardo Almanac vol. 19 No. 3, MIT Press (January 2013): 284-305.

Theoretical work:
Creative principles
Parametric model
Parametric representation
Practical work:
Practice summary
AG#1
Arpeggio-Detuning
AG#2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Technical diagram from the first version of my audio-visual instrument:

 

Demonstration of how AG#1 operated based on the zither input:

 

 

To see this full-screen click the button on the bottom-right of the video frame.
Or click HERE to see on Youtube

Creative strategies for sonic complexity:

AG#1 operates through diatonic pitch tracking. Each of twelve pitch detection groups is mapped to four digital audio samples – the samples are organised in four banks assigned to game pad buttons.

AG#1 operates according to mappings, but in performance the outcome is not fully predictable. Whilst the physical interaction with the zither is immediate, with software the interaction is necessarily mediated by code. My ways of combining these fundamentally different modes of interaction convey a musical language formed of surreptitious chromaticisms and timings, where expression comes from avoiding easy developments.

Several technical aspects convey performative control. The zither playing is audible regardless of software detection. The digital audio sample banks (not the samples themselves) are activated or muted through the game pad. And clear notes generate predictable response.

Other technical aspects convey performative instability. The zither tuning is somewhat ambivalent, and a few of its strings are purposefully aged - I do not fully predict which audio sample will be activated, and a single zither sound might activate more than a single sample. Also, the digital sound may overlap with its acoustic cause, or appear immediately after, or overlap with subsequent sounds. Moreover, when the acoustic sound has no particular tonality – e.g. when zither strings are plucked, picked or strummed - the software response is unpredictable. And finally, whereas playing mezzo forte or forte activates digital audio samples, playing piano does not – while fluidly playing the zither, I do not fully predict where that threshold is.

Creative strategies for visual continuity:

The image aims to create a sense of space without distracting the audience's attention from the relations between the sounds. The visual dynamics are not demanding in terms of cognitive processing; there are no radical visual discontinuities, which would automatically attract attention. Yet, the image is not really continuous. The camera view over the 3D world changes, light particles appear and faint, either isolated or drawing paths through the world. Such variations subordinate to an overall continuity, as a few Gestalt principles facilitate visual simplification.

There are no changes of scenario in the 3D world, and the light particles placed in the 3D world are monochromatic and similarly shaped. This prompts the Gestalts of Invariance and Similarity. Any accumulating particles create visual paths, which may draw subtle shifts in the foreseeable logic of events, producing ambivalent discontinuities - at low resolution such discontinuities become continuous. Furthermore, the virtual camera moves creating an undulating view over the 3D world. In-between two audio input detections, successive visual events display similar interval of motion, forming a progression towards a point of limit (progressive discontinuity). This point of limit corresponds to the moment when a new is detected. It initiates smooth, rather than sudden, camera view shifts in another direction.

Creative strategies for audio-visual fungibility:

Perception binds sonic and visual shapes that change or move simultaneously (Principle of Common Fate), as well as those that change adequately proximal in time (Principle of Proximity/ sequential integration). Yet, the sense of sensory unison may not always correspond to the digital mappings, which range from transparent to opaque. Whilst moments of mechanical transparency indicate the instrument, moments of opaque mapping counteract its understanding - an invitation to explore perception itself.

A few connections between sound and image are clearly perceived. Bigger visual changes – i.e. the emergence of visual particles and the changes of 3D camera direction - occur immediately upon sound detection. In addition, when digital audio is played-back its spatialisation aligns with the moving image, reproducing a seemingly natural relation.

Other factors blur the mechanical causation. The acoustic input does not always produce visual changes - the zither is always audible, but the software is adjusted to respond above certain amplitude only. Using digital recordings of the zither makes acoustic sound and digital sound at times undistinguishable, since they mix in the output. Furthermore, the visual changes are not always synchronised with the digital audio – not only because the digital audio can be muted, also because a number of digital audio samples start with different lengths of silence.

 

 

 

Performance work

AG#1 was developed along with a performance series: WINDOWMATTER. This series includes solo performances and collaborative performances. Solo performances were in Fundão, Moagem do Fundão; in Madrid, at Off-Limits; in NYC, at Share; in London, at St. James Church/ Goldsmiths College in London. Duos with John Klima involved two different, networked versions of AG#1. These duos were performed in Lisbon, at Calouste Gulbenkian Foundation, ZDB, at Festival Bang, at Espaço Avenida ("15 anos Bolsa Ernesto de Sousa"), at Bomba Suicida and Largo do Século. Another duo, with Fala Mariam, was presented at Teatro S. Luis, in Lisbon. The audio architecture and the digital sounds from AG#1 were also modified to operate based on the audio input from other artists. This was explored during a collective residency with John Klima, Toshio Kajiwara, David Maranha, Sei Miguel, Fala Mariam, Cesar Burago, Andre Gonçalves and O.blaat. The performance, which involved several networked versions of AG#1, was shown at Regueirão dos Anjos and Largo do Século, in Lisbon.


Adriana Sá solo audio recording funding & supports
duo with John Klima  
duo with Fala Mariam audio recording
orchestral version

SOLO VERSION


AUDIO RECORDING

In the WINDOMATTER performances where I played solo, AG#1 operated based on the input from my custom zither, which also inputted a sampler / delay modulator. In this way, the sonic constructions intertwined acoustic sound, digital sound, and electronic sound. I wanted the audience's perceptual motion to be driven by the music, and the image to create a sense of total environment . The image should extend their sense of presence to a digital world beyond the screen. It should become an organic stage scene that reacts upon the sound, modulating perceptual experience with an undulating view over an unchangeable landscape.

Performances:

  • St. James Church/ Goldsmiths College, London, UK 2011
  • Share, NYC, USA, 2010
  • Off-Limits, In-Sonora Festival, Madrid, Spain, 201
  • ZDB, Lisboa, Portugal, 2010
  • Moagem, Festival Y, Fundão, Portugal, 2009
   
GRAPHIC SCORE
for a solo performance:
 

 

DUOS WITH JOHN KLIMA




These WINDOWMATTER duos were performed with two different versions of the AG#1 software, which operated based on my zither input and the input of John's Portuguese guitar, or his 'noise-makers'. In each version, the interaction design, audio architecture, digital sounds and audio-visual mappings were very different. Yet the image was similar: the two versions were networked, so that each provided a different visual perspective over the same digital 3D world. I made the 3D world look like a moving painting. In performance, two moving paintings were projected side-to-side, upon us.

Performances:



DUO WITH FALA MARIAM

AUDIO RECORDING

 

In this WINDOWMATTER duo I played my zither with aged strings and ambiguous tuning, inputting the AG#1 software. Fala Mariam played her trombone, dry. Each of us made a graphic score, and the sonic construction resulted from their combination. The image was projected upon us both.

Performance: Teatro S. Luiz, Lisboa, Portugal, 2010


ORCHESTRAL VERSION

studies

toshio's instruments




In this WINDOWMATTER project several musicians extended their instrumentation with AG#1. Each created a set of digital sounds, to be activated upon the sound of their instruments. John wrote different versions of the C++ code. Each version of the software had its particular sound architecture and configuration, and all versions were networked. I made a graphic score for the collaborative sonic construction. We performed the piece in different site-specific situations. In each occasion, several perspectives of a same 3D world were projected over semi-transparent surfaces, creating an architectural space of light.
      

Adriana Sa, zither, AG#1
John Klima, portuguese guitar, AG1
O.blaat, hacked wii controller
Sei Miguel,  pocket trompet
Manuel Mota,  electric guitar
David Maranha,  violin & dobro
Fala Mariam
,  trombone
Cesar Burago
,  percussion
Andre Gonçalves
,  analogue modular synth
Toshio Kajiwara, turntables, modular synth, toys

Performances | see text in PT

  • WINDOWMATTER _ inside the time (photos above)
    esidency at Regueirão dos Anjos, Lisboa, with a final, site-specific performance-installation (July 1st - 21th 2008)
  • WINDOWMATTER _ outside the shpere (photos bellow)
    Site-specific performance-installation at Largo do Seculo/ outdoors and Século Bar/ indoors (July 25st 2008)
 
     The in-door performance:
john klima and david maranha
manuel mota
adriana sa, fala mariam and and andre gonçalves
toshio kajiwara
0
toshio kajiwara and sei miguel
         The outdoor rehearsal:
john klima
adriana sa
margarida mendes
cesar buraga
sei miguel, john klima and toshio kajiwara
adriana sa and fala mariam
toshio kajiwara and margarida mendes
fala mariam, cesar buraga, manuel mota and david maranha
john klima, david maranha, toshi kajiwara and sei miguel
fala mariam and manuel mota
photos by O-blaat (that's why she's not in the pictures!...)