adriana  sá
trans-disciplinary   music

 

AG#1: repurposing video game technologies
in combination with a zither

To see this full-screen click the button on the bottom-right of the video frame.
Or click HERE to see on Youtube

AG#1 operates through diatonic pitch tracking. Each of twelve pitch detection groups is mapped to four digital audio samples – the samples are organised in four banks assigned to game pad buttons.

AG#1 operates according to mappings, but in performance the outcome is not fully predictable. Whilst the physical interaction with the zither is immediate, with software the interaction is necessarily mediated by code. My ways of combining these fundamentally different modes of interaction convey a musical language formed of surreptitious chromaticisms and timings, where expression comes from avoiding easy developments.

Several technical aspects convey performative control. The zither playing is audible regardless of software detection. The digital audio sample banks (not the samples themselves) are activated or muted through the game pad. And clear notes generate predictable response.

Other technical aspects convey performative instability. The zither tuning is somewhat ambivalent, and a few of its strings are purposefully aged - I do not fully predict which audio sample will be activated, and a single zither sound might activate more than a single sample. Also, the digital sound may overlap with its acoustic cause, or appear immediately after, or overlap with subsequent sounds. Moreover, when the acoustic sound has no particular tonality – e.g. when zither strings are plucked, picked or strummed - the software response is unpredictable. And finally, whereas playing mezzo forte or forte activates digital audio samples, playing piano does not – while fluidly playing the zither, I do not fully predict where that threshold is.

The image aims to create a sense of space without distracting the audience's attention from the relations between the sounds. The visual dynamics are not demanding in terms of cognitive processing; there are no radical visual discontinuities, which would automatically attract attention. Yet, the image is not really continuous. The camera view over the 3D world changes, light particles appear and faint, either isolated or drawing paths through the world. Such variations subordinate to an overall continuity, as a few Gestalt principles facilitate visual simplification.

There are no changes of scenario in the 3D world, and the light particles placed in the 3D world are monochromatic and similarly shaped. This prompts the Gestalts of Invariance and Similarity. Any accumulating particles create visual paths, which may draw subtle shifts in the foreseeable logic of events, producing ambivalent discontinuities - at low resolution such discontinuities become continuous. Furthermore, the virtual camera moves creating an undulating view over the 3D world. In-between two audio input detections, successive visual events display similar interval of motion, forming a progression towards a point of limit (progressive discontinuity). This point of limit corresponds to the moment when a new is detected. It initiates smooth, rather than sudden, camera view shifts in another direction.

Perception binds sonic and visual shapes that change or move simultaneously (Principle of Common Fate), as well as those that change adequately proximal in time (Principle of Proximity/ sequential integration). Yet, the sense of sensory unison may not always correspond to the digital mappings, which range from transparent to opaque. Whilst moments of mechanical transparency indicate the instrument, moments of opaque mapping counteract its understanding - an invitation to explore perception itself.

A few connections between sound and image are clearly perceived. Bigger visual changes – i.e. the emergence of visual particles and the changes of 3D camera direction - occur immediately upon sound detection. In addition, when digital audio is played-back its spatialisation aligns with the moving image, reproducing a seemingly natural relation.

Other factors blur the mechanical causation. The acoustic input does not always produce visual changes - the zither is always audible, but the software is adjusted to respond above certain amplitude only. Using digital recordings of the zither makes acoustic sound and digital sound at times undistinguishable, since they mix in the output. Furthermore, the visual changes are not always synchronised with the digital audio – not only because the digital audio can be muted, also because a number of digital audio samples start with different lengths of silence.

AG#1 was developed along with a performance series: WINDOWMATTER. This series includes solo performances and collaborative performances. Solo performances were in Fundão, Moagem do Fundão; in Madrid, at Off-Limits; in NYC, at Share; in London, at St. James Church/ Goldsmiths College in London. Duos with John Klima involved two different, networked versions of AG#1. These duos were performed in Lisbon, at Calouste Gulbenkian Foundation, ZDB, at Festival Bang, at Espaço Avenida ("15 anos Bolsa Ernesto de Sousa"), at Bomba Suicida and Largo do Século. Another duo, with Fala Mariam, was presented at Teatro S. Luis, in Lisbon. The audio architecture and the digital sounds from AG#1 were also modified to operate based on the audio input from other artists. This was explored during a collective residency with John Klima, Toshio Kajiwara, David Maranha, Sei Miguel, Fala Mariam, Cesar Burago, Andre Gonçalves and O.blaat. The performance, which involved several networked versions of AG#1, was shown at Regueirão dos Anjos and Largo do Século, in Lisbon.

AUDIO RECORDING

In the WINDOMATTER performances where I played solo, AG#1 operated based on the input from my custom zither, which also inputted a sampler / delay modulator. In this way, the sonic constructions intertwined acoustic sound, digital sound, and electronic sound. I wanted the audience's perceptual motion to be driven by the music, and the image to create a sense of total environment . The image should extend their sense of presence to a digital world beyond the screen. It should become an organic stage scene that reacts upon the sound, modulating perceptual experience with an undulating view over an unchangeable landscape.

Performances:

• St. James Church/ Goldsmiths College, London, UK 2011
  
• Share, NYC, USA, 2010
  
• Off-Limits, In-Sonora Festival, Madrid, Spain, 201
  
• ZDB, Lisboa, Portugal, 2010
  
• Moagem, Festival Y, Fundão, Portugal, 2009

These WINDOWMATTER duos were performed with two different versions of the AG#1 software, which operated based on my zither input and the input of John's Portuguese guitar, or his 'noise-makers'. In each version, the interaction design, audio architecture, digital sounds and audio-visual mappings were very different. Yet the image was similar: the two versions were networked, so that each provided a different visual perspective over the same digital 3D world. I made the 3D world look like a moving painting. In performance, two moving paintings were projected side-to-side, upon us.

Performances:

• Numero Festival, CAM / Calouste Gulbenkian Foundation, Lisbon, 2008
  
• Bang Festival, FBA-UL, Lisbon, 2008
  
• 15 years Ernesto de Sousa fellowship awards, Espaço Avenida, Lisbon, 2008
  
• Musicbox, Lisbon, 2008
  
• Creative Fest #1 (view flyer), Bomba Suicida, Lisbon, 2008
  
• Knock on Woods (view flyer), Largo do Século, Lisbon, 2008

AUDIO RECORDING

In this WINDOWMATTER duo I played my zither with aged strings and ambiguous tuning, inputting the AG#1 software. Fala Mariam played her trombone, dry. Each of us made a graphic score, and the sonic construction resulted from their combination. The image was projected upon us both.

Performance: Teatro S. Luiz, Lisboa, Portugal, 2010

In this WINDOWMATTER project several musicians extended their instrumentation with AG#1. Each created a set of digital sounds, to be activated upon the sound of their instruments. John wrote different versions of the C++ code. Each version of the software had its particular sound architecture and configuration, and all versions were networked. I made a graphic score for the collaborative sonic construction. We performed the piece in different site-specific situations. In each occasion, several perspectives of a same 3D world were projected over semi-transparent surfaces, creating an architectural space of light.       

Adriana Sa, zither, AG#1
John Klima, portuguese guitar, AG1
O.blaat, hacked wii controller
Sei Miguel,  pocket trompet
Manuel Mota,  electric guitar
David Maranha,  violin & dobro
Fala Mariam,  trombone
Cesar Burago,  percussion
Andre Gonçalves,  analogue modular synth
Toshio Kajiwara, turntables, modular synth, toys

Performances | see text in PT

• WINDOWMATTER _ inside the time (photos above)
  esidency at Regueirão dos Anjos, Lisboa, with a final, site-specific performance-installation (July 1st - 21th 2008)
  
• WINDOWMATTER _ outside the shpere (photos bellow)
  Site-specific performance-installation at Largo do Seculo/ outdoors and Século Bar/ indoors (July 25st 2008)