Session 6

Presenters: Sergio Santiago Renteria Aguilar, Sze Tsang

01:00pm: Sergio Santiago Renteria Aguilar Multimodal Soundscape Synthesis

In this work in progress, we study various generative probabilistic models as a critical media for producing artificial soundscape elements from multimodal input, mainly natural language. This is motivated by the lack of generative environmental audio models in the deep learning literature and their potential in sound synthesis frameworks. On a technical level, we use off the shelf models such as multimodal autoencoders to find semantically adequate sound vectors in the latent space of generative adversarial networks. By controlling raw audio adversarial synthesis engines with multimodal interfaces, we flesh out the connections between abstract semantic manifolds and latent sound design spaces. At this point our results lack the quality and resolution of natural soundscapes, but we propose technical improvements. Ultimately, the models will be evaluated in terms of the degree of conceptual resemblance between generated sounds and semantic contents of the conditioning inputs. As such, this work is not concerned with reconstructing causal or physical processes underlying soundscape generation but seeks to leverage crossmodal correlates in humanly annotated audio distributions for creative purposes. More broadly, by interweaving creative practices in soundscape composition and multimodal learning techniques, we contribute to the discussion on the effects of the automation of creative labour.

01:30pm: Sze Tsang Exploring aspects of place through sound mapping

Combining aural and visual elements of a place can be a powerful way of exploring the intersections of time, history and geographical features that exist within a location. This paper details the author’s processes in incorporating place into compositional practice through a combination of field recordings and sonification, in relation to the author’s work, The Lost—an audio-visual contemplation of the sensation of loss and the subsequent feelings of dislocation—and how these feelings related to the artist’s own life experiences at the time. The Lost is a work partly based on a map of Perth from 1838, detailing many of Perth’s now-lost wetlands. This map was then sonified using Iannix (a graphical sequencer), and the sounds were processed and combined in Ableton Live (a Digital Audio Workstation) with a field recording from the still-existing Herdman’s Lake and sonified longitudinal and latitude values of these lost wetlands.

The Lost is an exploration of connections between artist, history, and place, and how these aspects can inform the creation of a work. Through this practice, the author aims to explore how sound and visual elements can combine and resonate with the other, and how such a practice can highlight the connections between artist and place.