IMAGO DIXIT: Painting, sound & realtime gesture capture

This article describes Imago Dixit, an interactive installation incorporating sound design, realtime gesture capture and painting. The text places the work in context considering some aspects of composition, interaction and perception. Any poetic analysis is primarily conducted from the auditory rather than the visual perspective.


Compositional Process
The creative process resembles a game, being a collaboration between two people from different disciplines (painting and sonic arts). Both artists started working privately on the material (color and sound respectively) the same day and with a fixed deadline, each freely creating a work with the sole constraint being that a relationship between image and sound should be clear. Hence, the first step constitutes the creation of a musical piece on the one hand and a painting on the other that have nothing in common. On the deadline the artworks are exchanged and each one creates a new piece based on the work he has received, arranging also a new deadline to reveal the final woks. The outcome is two paintings accompanied with their sound sculptures where in one case the sound drove the image and in the other the image drove the sound.

Background
The project could be thought to link to the concept of heterotopia as exemplified by Foucault (1967). The term can be used to denote the juxtaposition of many spaces (τόποι) in a single place. The artwork brings together spaces that are foreign to each other in reality. The aim is not to create an illusionary space consisting of the sum of real environments, but to create another (έτερος) space, which is real by structuring the different elements in an artistic fashion. In the current work the idea of many-spaces is linked to many-times (heterochronia), denoting the segregation of linear time, which by means of physical interaction is exposed to the viewer/listener.

Description
The title Imago Dixit translates from Latin as The Image Said. The interaction between image and sound in a physical way is the primary concern of the work. It comprises of two paintings (135 x 18 cm each) suspended on an empty wall. Each painting is equipped with an infrared sensor at the frame's edge which traces movement along the horizontal axis of the painting. The sensors are connected via an Arduino board to a computer running SuperCollider, the output of which is fed to a quadraphonic speaker setup surrounding the audience.
The participants are free to examine the paintings from any distance they think appropriate. Apart from the obvious visual aspect, the artwork has a hidden aural aspect as well. Every painting has a sonic counterpart corresponding to particular drawings. The viewer becomes also a listener by touching the artwork. Any movement made along the horizontal axis, which is closer than 2 cm from the surface of the painting, is traced by the infrared sensors revealing the appropriate sonic entities that match the exact position of the painting(s). There is always a one to one relationship between the visual and its sonic representation and vice versa. That is, for every texture/drawing of each painting exists a hidden soundworld waiting to be discovered by 'tactile' interaction. The artwork is considered unfinished without the contribution of the audience. The design of the system allows the participant to unfold the music at his/her own pace, perhaps discovering elements through sound that the eye overlooked.

Interactivity
There is plenty of literature on human-computer interaction in the field of music, both from the perspective of gestural interfaces, sound generation units and mapping strategies. In the current installation we decided that the system should be responsive enough so that an immediate cause and effect relationship between the performer's actions and the corresponding sonic result is clear, making use of a strategy that wouldn't be too complex so as to lose the audience, (Schloss, 2003). Therefore, on the one hand giving the impression that the participant is in command, and on the other, with a thorough sound design beforehand, avoiding the mapping being tedious.
I shall exclusively detail the physical interaction of the participant with the artwork, since the current configuration does not embody any behavior that flows from machine cognition. Technically speaking it is simply a matter of assigning the sensor data to sounds. However, the system should not be confused with a gestural interface and understood as such. In this sense it is not a matter of constructing an expressive instrument in order to make a convincing performance.
The work is to be viewed within the context of an interactive environment, where the concern is to provide an additional level of engagement with the paintings as opposed to the traditional 'look but not touch' behavior in a gallery. The actual process of viewing a painting in an exhibition space has a long tradition in the history of western art and inevitably carries a great deal of social conventions. The installation purposely makes use of the traditional medium of the canvas enhancing the visual clues by adding an audio-haptic layer.

Figure 2. Imago Dixit Image © Orestis Karamanlis
The system gives the impression of reacting with touch but in fact it does not. As stated earlier it is only the infrared sensors attached closely on the surface of the paintings that trace movement and trigger the sounds. Since the audience has usually little understanding of the mechanics, the user develops an idiosyncratic way to play with the artwork and react to the sounds. Experience showed that numerous visitors would experiment and try to invent a particular technique to bring about new sonic elements attempting to control the work in an unorthodox way. Many participants had the illusion that the degrees of freedom of the system were more than two, one for each infrared sensor being the true case, meaning that even though the design implements a simple human-computer interaction scheme and a straightforward mapping strategy, the work as a whole hopefully provides an engagement at a higher level than what the available degrees of freedom allow. I find the concept of 'higher-level engagement' by using simple architecture and easily accessible technology tempting, although in order to evaluate the effectiveness of such a design a thorough statistical analysis of the participants' behavior would be needed.

Designing sound objects: On the verge of acousmatic
Being a composer of electroacoustic music I cannot resist viewing the sonic material through the lens of the theory of the sound object in the acousmatic fashion. There are though two fundamental differences in the way the current installation contextualises the notion of the sound object: a.
The 1960s period at the Group de Recherches Musicales in France is marked by the concept that sounds can be used with no relation to meaning This attitude had an impact on the theory of listening as well, where the audience should listen to the sounds as sounds and try to enjoy their morpho-typological properties than assigning extra musical significance that could pollute the perception (Schaffer, 1966). In the case of Imago Dixit even if the true source of the sounds is hidden, 'cause and effect' relations develop between the participant's gesture and the sonic equivalent and extra musical references spring either from the images or the sounds themselves. While interacting with the artwork the listener relies heavily on vision and he/she is intentionally modifying the soundworld in realtime with his/her actions. It is the drawings on the painting that help the audience to relate the 'sounds to supposed sources and causes, and to relate sounds to each other' appearing to have 'shared or associated origins' (Smalley, 1997).

b.
Next, in Imago Dixit the listening mode of the audience shares some properties with the composer's mode. By this I mean that the process of reduced listening, the concentrated and repeated listening of a sound over and over again, which is a privilege of the electroacoustic composer when building a musical piece, is also available to the audience. A static touch on a particular point on the painting will only trigger the specific sound objects and will continue doing so until there is a change in the user's action. In the same way that the timbral qualities of a particular sound can be revealed to the composer through concentrated listening, the audience has equally the ability to dive into the process of microscopic listening/seeing, since a gradual disposition of the hand will slowly reveal new sonic material accompanied by their drawings. All the same, it should be made clear that this investigative process of playing with the artwork does not require from the participant to concentrate on the morphological qualities of the sounds but only enjoy the interplay.
With regard to the notion of musical narrative it has been recognised that 'hearing on the level of the object cannot be the same as hearing on the level of the work' (Natttiez, 1990). The sounds reflecting the drawings on the paintings in many cases resist segmentation and their perception depends on the energetic shape of the performer's action through time. But within the context of the current installation the sounds' temporal evolution dissolves; what is heard 'before' can also be heard 'after'. The manifestation of sounds within time is not a linear process. There is a significant difference between the interactive narrative of the installation and the supposedly linear narrative of a musical piece.

Drawing and sound
There are a few things taken into account prior to the compositional process that deal with the relation between sound and image specifically from the sonic perspective. These issues can be summed up as follows: i.
Classification of figures/shapes and sounds and their relationship. ii.
Developing a dialogue. Sound driving image versus image driving iii.
Notions of proximity, polyphony, transformation, growth and expectation. v.
Representation of 'unknown sounds' and vice versa representation of 'unknown drawings'. Influencing the audience's perception about the quality of the sound objects and drawings.
Here I am more interested in polyphony and transformation; i.e. how can multiple shapes on the canvas be represented at a given time and how can their evolution on the painting have a meaningful sonic counterpart. A painter has the ability to draw objects on a 2-dimensional terrain. A composer can regard the horizontal axis of a painting as time, which is the most obvious case, and the vertical as pitch. Still, there is simply not enough spectral space for everything if someone is indeed interested in representing multiple objects and developing simultaneous processes. Additionally, there is probably little value if the painting simply functions as a spectrogram of the music. In the current installation the sonic counterpart primarily comments on the most important elements of the painting, either being discrete objects or more abstract textures, while the temporal boundaries of the sounds are determined by the spatial boundaries of the drawings. The pitch boundaries of the sounds are arbitrary and depend on the amount of 'polyphony' on the vertical axis of the painting. By adopting the convention that the total length of the painting in centimeters is equivalent to the total duration of the sounds in seconds (135cm107.5sec), the sounds corresponding to an image should last for the 'duration' of the image. The overall form of the painting is broken down into groups of textures that blend together while the participant scrubs through the sound sculpture.
The viewer/listener needs to be able to find similarities on the actual drawings in order to establish a link between the visual and the auditory. Experience showed that no matter how strange and alien to reality the musical material was, the audience was still able to engage and enjoy the interplay between the music and the images, probably considering the whole process as a game. From this it springs that the often-encountered inability of the audience to follow an acousmatic piece (projected over loudspeakers) would not be associated to the nature of the sounds themselves, but probably to the way the piece is presented, or to the absence of visual queues or even to the structuring of the materials.

Exit
In Imago Dixit the paintings are not the listening score for the electroacoustic sounds. Furthermore, there is no intention to guide the audience to a particular mode of listening and/or interaction with the artefact. There are indeed 'performances' that work better but there is no expected type of behavior that relates to the design of the system. The interactive environment is simply open to interpretation and play. In this sense the process of discovering the artwork is part of the reason of its existence.