Intersensoriality , immersion and environment in digital art : Paroles trouvées , a spatialised musical , videochoreographic , optical installation

This paper proposes reflections of an ongoing nature about the relationship of sound, image, and movement in art. Following a range of productions from stage choreographies to dance films, CD-ROMs and interactive-generative installations to videoclips and multimedia scenographies for dance and music, practical experience has shown that what may be seen as standard and complimentary elements of much contemporary interdisciplinary creation, in fact prove to offer a gamut of very different creative potentialities with each new project. This said, the question of interdisciplinary practice has also been occulted in recent years by artistic fervour and fascination surrounding digital productions that propose “new” sensorial experiences. While artists and critics alike are keen to assert how it is the technologies themselves that are responsible for “augmenting”, “embodying” or “interfacing” sensorial experiences in art, the basic configuration of sound, image, and movement nevertheless remains a constitutive parameter of most productions, be they specifically digital, interactive, generative or not.

I'd like to begin with an excerpt from an article by Atom Egoyan on Canadian artist Janet Cardiff.Viewing Cardiff's interactive mixed media work Whispering Room (1991) for the first time challenges the filmmaker's prior perception and understanding of art.For the purpose of the analysis, I have superimposed a grid over the text based on three factors: 1) actions and subjective impressions of the spectator (in pink); 2) media elements employed to create those impressions (in yellow); and 3) artistic strategies (in green): I entered a room at the Art Gallery of Ontario where a series of audio speakers mounted on thin metal stands emitted a soft murmur of conversation.As I got closer to each speaker, I could make out individual texts.At certain moments, my movement would trigger a projected image of a young girl in a red dress dancing in the woods.
It's difficult to express my excitement in this room.I had the sensation of being in the middle of a film that was still being formulated; that was still in someone else's mind.I was completely overwhelmed by the collision of technological artifacts-speakers, projectors, lights, wiresand narrative abstraction.I found myself drifting through the emotional residue of a personal trauma that was both immediate and distant, visceral yet disembodied.
Whispering Room was an experience of installation art as a forum for dramatic storytelling.It made me feel inspired, and at the same time frustrated by the constrictions of traditional film practice (Egoyan, 2002: [online]).
From this grid, spectatorial actions/impressions (9), artistic strategies (7) and different media elements (5) can be seen as producing a singular, interactive form of artistic expression.Their configuration 'inspires' Egoyan because of the ways the work constitutes an alternative form of 'dramatic storytelling' or 'narrative abstraction'.In other words, heterogenous elements of sound, image and movement combine harmoniously in an artistic statement that is coherent and efficient AND the new media interfacing configures them in such a way as to provide a new 'experience' of art.
If media art purports to offer a spectrum of perceptual and kinesthetic experiences that forge heightened levels of perception for spectators, it does not follow that artists simply deploy new tools for conveying and exchanging information or a point of view.In 'The Question concerning Technology', Heidegger argues that 'the essence [Wesen] of technology' has been 'obscured by the commonly accepted definition of technology as instrumental, as a means to an end' whereas for the Greeks, technē was 'the name not only for the activities and skills of the craftsman but also for the arts of the mind and the fine arts ' (1993: 318).From a contemporary perspective however, the question is not so simple.As Australian artist Jeffrey Shaw notes, 'The trajectories of new media exemplify a complex set of negotiations between body and space, negotiations between the actual domain of the real body of the viewer and the real space we inhabit, and the virtual domain of the represented body and represented spaces' (2009).
Philosopher and social theorist Brian Massumi situates the debate on yet another level in an opposition of body/machine: The activity of the body externalizes its self in the mediating materiality and coded protocols of the adaptive interface.(...) The abstracted activity of the human body and the matter within which it acts become isomorphic-formal repetitions of each other.All the weight of their former materiality is taken on by their mutual transformer, the interface.The body disappears behind a technological shield, becoming a backstage director, an organizing desire, or will, in self-protective hiding -a defensive self informing inert matter in homogeneous space, forming it in its own likeness.Except that it loses its likeness.When it looks at its face in the mirror, it sees the interface, the display device ' (1995: [online]).
Massumi's argument would seem to contradict the sensual experience of Cardiff's work described by Egoyan.Against this unflattering view of digital interfaces, a third argument identifies digital interdisciplinary works as proposing sensorial experiences that are embodied.As far back as 1886 when scientific and philosophical developments were ushering in new visions of the world that notably paved the way for phenomenology, Ernst Mach, in The Analysis of Sensations and the Relation of the Physical to the Psychical (1914: 2), advanced the idea that colours, sounds, temperature, pressure, space and time are connected with one another and can be further associated with states of being, feeling and willing.In 1935, Paul Schilder proposed that all sensation is synaesthetic and that it is only through secondary processes that we come to isolate individual senses (1935).For Maurice Merleau-Ponty, cross-sensual activity is characteristic of all human experience, generating an embodied experience of external reality: 'natural perception [that] we achieve with our whole body all at once (...) opens on a world of inter-acting senses ' (1962: 225).And, if phenomenology has been widely recuperated by artists, critics and scholars to explain and justify sensoriality in digital art, critic RoseLee Goldberg notes that already with performance works of the 70s, a 'huge laboratory [providing] real experiences [had] turned audiences' senses inside out ' (2003: online).Rejecting conceptual art's dematerialisation of the art object, performance art set the precedent by giving 'materialisation or physicality to ideas ' (2003: [online]).
Yet another perspective is offered by Italian researcher Enrico Pitozzi with respect to 'augmented' stage performances.Pitozzi signals an '(...) epistemological break -a real shift of perspective in the performing arts -that invests the role of performer with other, pluralistic forms of presence ' (2008).These forms of presence do not concern the performer alone: 'on the contrary, [they] should be expanded to include (...) "objective presences" such as light and sound (...) device(s) which allow(s) these forms of presence to manifest' (Pitozzi, 2008).Pitozzi further argues, 'We must understand that technologies are not a means, but environments that convey a thought process ' (2008).I would add that digital interfaces both install, and act as, a metalanguage that assembles, unites and coordinates heterogenous media elements and that it is precisely through this common coding (metalanguage) that a new understanding of interdisciplinarity can be advanced today.
Traditionally, heterogenerous elements in theatre, dance and opera combine to create the vivid and engaging quality of performances.A new emphasis on works as creating 'environments', be they for the stage or installations, suggests that something about the nature and perception of time, space and the body in representation has changed.Pitozzi's observations take into account both live and mediated forms of presence in performance as articulated through such artistic strategies as simulation, displacement, mirroring, multiplication, complementarity, juxtaposition, polarity, amplification and trace.Qualities of presence can be described as being dynamic, kinetic or aesthetic and also in terms of texture, scale of intensity, tone, accumulation and/or combinatory effects.
In installation art, presence might be more adequately described as the staging of spectators' presence to/in works; firstly, through spectators' physical mobility and circulation within a work and secondly, by other types of immersion: 1) proprioceptive (through means of heightening receptivity or favouring a sense of interiority) 2) psychological (by engaging or highlighting feelings of attraction or repulsion or imposing constraints) 3) mental or intellectual (solicitation through ideas, concepts or aesthetic propositions) 4) sensual (stimulation via the senses) and 5) multidimensional (different degrees, combinations or intensities of the above).A sense of immersion can be produced physiologically for example through sound: using surround systems, multiple sources/types of sound and a consideration of factors such as volume, quantity, distance, clarity, rhythms, colours, frequencies and pitch, or silence.It will notably be shown how the 360° soundscore of Paroles trouvées and the work's visual interfaces combine to generate a particular form of immersion.But before, a brief historical detour to outline aspects of immersive devices may be instructive.

Historical precedents
The earliest precedent for immersive works in Western art can be traced to 1781 when Philip James de Loutherbourg created the Eidophusikon, a visual device that attempted to stage movement through the presentation of successive pictures painted on taffeta in translucent colours that were viewed through a 6 foot square opening in a wall.Lit from behind at different distances according to the desired effect and augmented by a play of reflecting mirrors, the works were also accompanied by live music (Burns, [online]).With Robin Barker's patenting of panoramas in 1787, a new viewing device was proposed in circular buildings that housed paintings of up to 50 feet high, 300 feet in diameter and weighing 8,000 pounds representing wideangle or 'all-view' 'picture(s) without boundaries' (Zoetmulder, 1981).Quickly becoming a form of mass entertainment, these grand-scale vistas of landscapes or historical scenes were viewed from central platforms, addressing both the central and peripheral vision of spectators while engulfing them by their size.According to Paul Zoetmulder, 'Quite simply, the secret of the panorama lies in the elimination of the possibility to compare the work of art with the reality outside, by taking away all boundaries which remind the spectator that he is observing a separate object within his total visual field' (1981).
In the 1840s, cycloramas, after dioramas, presented panoramas mounted on cylindrical walls with spectators standing in the center of a room.Turning around, they could take in all points of a horizon presented.Attempting to reconcile the static nature of visual representation with performance, these huge installations also included live entertainment and lighting effects that added a sense of drama to spectacle.One famous cyclorama, The Siege of Paris (1870), included the presence of illusionists, ventriloquists, performing mice, birds and cats and a character named "Herman the Magician".
In the 1925 Paris Exposition des Arts Décoratifs, futurist architect and scenographer Enrico Prampolini presented a model 'Magnetic Theatre' with dynamic architecture, moving lights and projection screens.His manifesto, Futurist Scenography (1915) also describes effects produced by electrical current and colored gases.Following his example, theatre directors such as Max Reinhardt, Berthold Brecht, and Armand Gatti began to incorporate film projections in their mises-en-scène, creating prototypes for future multimedia performance.However, the first electronic environment to combine architecture, film, light and music in a specifically immersive experience was Poème électronique by Le Corbusier, Xenakis and Varèse, commissioned for the Philips Pavilion at the Brussels World Fair in 1958.Researcher Marc Boucher remarks that works like this, 'coalesced with a rhetoric of empowerment (out with the stage, in with the participant) that had less to do with politics (the struggle of the labouring classes) than with sensory experience' (2010).
The first multimedia environments by today's standards included creations by Josef Svoboda and Alfred Rádok.Laterna Magika (1958) was a kinetic multimedia spectacle combining live and mediated action with projections, dance, music, lighting effects and pantomime.Their 'psychoplastic' stage or Polyekran system not only involved panoramic projections but also suspended screens on which synchronised and dynamic visual displays created rythmic effects.For the 1967 World Fair, the Diapolyekran, a multimedia wall deploying 224 slide projectors and 30,000 slides, involved 112 screens that were in fact cubes, each equipped with 2 slide projectors that could alternately display individual images or parts of a global image with a variety of transitional effects.Being kinetic, each cube was also capable of protruding or receding like a drawer thus producing a hypnotically pulsating screen.
Two other milestone works were presented at Expo 67: the first, Canada 67, a film by Robert Barkley deploying a CircleVision 360° projection system designed by Walt Disney built with 9 cameras and projectors and creating a new form of spectatorial immersion in the round.Kinoautomat: One Man and His House by Radúz Činčera was the world's first interactive movie, playfully engaging spectators to vote on the outcome of certain scenes by activating one of 2 possible films whose interactivity was regulated by a lens cap being switched between synchronised projectors.
Parallel to these inventions and others to follow including Expanded Cinema and IMAX theatres, various postmodern multimedia performances, installations, experimental films, video artworks and site-specific works, too long to describe here, were all to propose new immersive strategies.Jacques Polieri's multiple, mobile, and circular stages can be cited as an attempt to create a dynamic merging of audience and stage through complex scenography.Performances by Marina Abramovic and Ulay, installations by Dan Graham and Bruce Neumann, and films by Stan Brakhage and Michael Snow, although not pointedly immersive, challenged traditional perspective by allowing spectators to choose their own point of view or by introducing new ways of understanding and experiencing art.Pioneer works like CYSP 1 (1956), a Maurice Béjart-Nicholas Schöffer collaboration or Merce Cunningham and John Cage's Variations V (1965) introduced new interdisciplinary relationships in performance involving interactivity.
As of the 1990s, artists intrigued by interactive possibilities began developing original devices, programs and immersive digital scenographies.Bearing an evident kinship with the early panoramas and multimedia walls, some of the works enclose spectators inside large scale structures for image projection with sound in 360° and the possibility of circulating both physically and virtually within their spaces.Spectators are also given control of the works' functioning.Jeffrey Shaw's projects notably involve immersive multi-media data mining, 360° interactive data browsers, and portable stereo panoramic virtual reality systems with high-resolution interactive, 3D graphics and video.Masaki Fujihata's works deploy GPS tracking, with recorded video sequences marked as locational data and then presented in futuristic virtual settings into which spectators "enter".For Fujihata, moving images are 'freed' by digital technology: 'The screen is no longer to be considered as a physical object, it can be defined as a virtual screen in computer space, cyber space (...) a 3D screen floating in the air' ( 2009).
The multimedia performances of the Japanese collective DumbType or Catalonia's La Fura dels Baus can also be cited.While often, though not always, presented on stages, their works literally jolt spectators' perception with daring audiovisual effects: sophisticated multimedia projections as mediated intertextual annotations, blaring electronic music, innovative lighting effects etc. Mention should also be made of VJayed music performances and visual music that propose visual "artefacts" and generated forms in electronic environments integrating hybrid systems, analog video feedback, 3D images, real-time digital processing and object manipulation to create interaction between music, video and light.
Another tangent of multimedia production relevant for the analysis to follow concerns interactive installations proposing musical instruments or visual representations of music.Music Plays Images X Images Play Music (1997) by Toshio Iwai is an interactive musical instrument that can be activated by individuals via hand-held devices or by groups playing on a board.In dance, researchers from Ohio State University have recently explored new means of visualising choreography with the web project, Synchronous Objects for One Flat Thing, reproduced (2009) based on a work by William Forsythe.Graphic representations in the form of traces, lines and forms in space and time make explicit the work's movement.

The case of Paroles trouvées
The installation Paroles trouvées, as its title suggests, plays with the idea of memory as a secret "speech" within that is revealed in moments of interiority when the frontiers between what is external and internal, the past and the present, efface and the body becomes a permeable membrane through which multiple spaces open, circulate and free inner voices (Besson: 2007).The work's tripartite score of musical, videochoreographic and optical elements generates a sensory space as a polyphonic form and kind of aquatic environment, creating a new representation of sound as it interacts with projected visual and optical structures.Based on the notion that movement influences the imagination, intersensoriality, immersion and different forms of presence act to ciment this relationship.
The performance space resembles a circular installation with 8 speakers delineating an outer circumference, a basin of water reflecting light at its centre, reclining chairs placed in an inner ring around the basin, and an overhead screen 4 metres wide.On a table in a corner, a monitor displays the sound score in real-time, proposing a first visualisation of the music.Although the audience is free to circulate, the work's mode of reception is not one habitually associated with installations.Primarily conceived as an aural experience of spatialised sound in 360° with optimal listening conditions, the work rather proposes the equivalent of a combined film and concert that lasts 18".So although the work takes place in an installation space, here, the idea of circulation was intentionally limited as composer-conceptor Dominique Besson wanted spectators to listen to the entire musical composition, as in a concert, and without being disturbed by others entering or exiting the space.

The Ring
The lights dim, the body relaxes into the chair and the spectator sinks into a floating state produced as much by the hypnotic quality of the musical-sound score emerging from all directions in the darkness as visually, when the large overhead projection space begins to display images of water rippling.These are not prerecorded images however, but water actually being activated in real-time by a hydromechanical system.Conceived as a means of revealing the undulatory vibration of sound at its purest, most abstract level, the water and the optical effects are piloted by the same original digital program, entitled The Ring, that was created to compose, spatialise and project the music.It is thus the musical score that controls the functioning and periods of activity of ventilators and bars located in the basin that in turn, determine the size, direction and intensity of the optical effects.
As previously noted, the particular effect of immersive works depends on diverse and varying degrees of intersensoriality.Vision and kinesthesis notably combine to create a sense of motion; sound and vision, to what sound theorist Michel Chion calls synchresis (1994: 2); touch and sight, to what is haptic; and internal/external elements to what is experienced as being prorioceptive/exteroceptive.It will be seen how sound and images in Paroles trouvées interact to produce a sense of immersion, place and presence, but with the optical device, a literal correlation is established between the intensity and directions of the water's movement and the vibrations of the soundscore in space, acting together on the spectator's perception and inducing a state of heightened receptivity.After this prelude, video images, projected into the basin and reflected onto the screen above via a mirror under the basin's transparent bottom, replace the water's movement, proposing a second narrative space to that of the musical environment.

The basin-optical device
In most choreography, screendance, theatre productions and cinema, music and soundtracks are conceived to accompany and enhance visual/narrative elements.With Paroles trouvées, three important artistic parameters prevailed that were to change this relationship, the first being that it was the musical score and not the choreography that preceeded and determined the project.With the videochoreographic images being subordinated to the primacy of the music and a score that had already been composed, the nature of the work's texture and the collaborative process involved radically changed.My role was to create images to "represent" sound through a visual environment.On one level, this reversal of roles entailed approaching the work intuitively, relying principally on the score as a reference.On the other hand, it meant having to reevaluate the very nature of how images are presented and what space they occupy in a sound-dominated environment.
A first word about the effect of hearing a musical score in 360°.Obviously, this experience cannot be compared to the perception of sound in stereo.Here, sound takes on a physicality that occupies and generates a sense of space while its allencompassing vibrations produce a soothing effect on spectators.The amplification of spoken phrases creates intimacy and, along with the multiple sound sources and control over their individual volumes, the accumulation and movement of sound in space, issuing forth as music, electroacoustic sound objects and other effects meeting, interacting and blending together, unfolds a rich narrative tissue.Besson's score creates images, generates tones and atmospheres, and 'breathes' with its suspensions, tensions and moments of rest.
Paroles trouvées takes as its source letters discovered in a trash bin in the French Alps written by women nearing the end of their lives in which they recount their memories and solitude.Although it was the composer's intention to commission an abstract graphic visualisation of the music, I proposed filming an intimate portrait of a single figure interpreted by dancer Ana Ventura.Adopting an art video aesthetic and a more or less direct and probing presence of the camera as in documentaries, my idea was to first hone in on private spaces of the body, revealing them as a kind of landscape, and then slowly over the course of the musical score, zoom back to uncover the entire body and a full view onto danced movement per se.My challenge as choreographer-videographer lay in avoiding an overt representation of the body while the function of listening and indeed, of listening with the eyes, was the composer's major concern.Care also had to be taken in avoiding a literal rhythmic translation of sound to image.In the editing process, I found myself constantly having to resist a hypnotic pull of the score pushing me in the direction of wanting to follow its rhythms.
A first solution lay in creating a certain abstraction of the body through closeups on textures and motifs of elements such as water, tears, talc, clay, hair and skin and secondly, through the camera's eye parsing the body's lines and forms as in a voyage in which the full picture or destination is not immediately revealed but sensed and movement is constant.Abstraction thus extended to encompass the pace, duration and intensity of time in the images.By privileging long uninterrupted shots with slow camera movements, introducing photographic stills, and using slow motion in postproduction, the video medium, as with the spatialisation of sound, could serve an abstract function of creating space in a manner inherent to video: what video artist Peter Campus refers to as 'the duration of the image becoming a quality of space' (2002).Or, in the words of researcher Frédérique Mathieu speaking of La morsure, an interactive dance installation I created in 2001, 'The place where an action develops becomes a space' (2003).In the context of a portrait, I would also add that duration instills a sense of the subject's presence.
The imperative of working in an octophonic environment was the second important parameter determining the collaboration; significant not only because sound projection in 360° bears on the way in which images, dance or other, are conceived, created and presented, but also, for the way in which spectators perceive and assimilate them.Amongst other issues, a right balance had to be struck between what might be construed as an intrusion or overimposing presence of the video images in an environment expressly dedicated to a full sound experience when normally, their scale or content, as in cinema, would not pose the same problem.Additionally, given a reduced budget that did not allow us to consider a more sophisticated means of image display, the question was also of how to compose images, that unlike the musical score, could achieve an equivalent sense of immersion without a projection device in 360°.
A first impression of visual immersion derives from the sheer size and closeups of images on the woman's body and the temporal magnification of tiny variations in the depicted action.The large and fixed projection space becomes a focal point onto a world.The spectators' gaze is drawn upwards towards a luminous space in the dark and through the camera's eye and presence, slowly but relentlessly penetrates this world.The screen as the sole focal point of the gaze constitutes a mise-en-abime of the woman's inner rhythms and voices that are also heard in the surrounding musical score.As a result, a sense of immersion also emerges as a physical amplification of the body's natural disposition and orientation wherein the eyes and ears engage in perception according to the direction of the gaze.
On another level, fluctuating intensities in both sound and image create a sense of depth and movement.In La Logique de la Sensation, Deleuze remarks that rhythm, sensation and movement are linked in painting.Referring to Francis Bacon's works, he notes, 'Sensation is vibration (...) To paint sensation that is essentially rhythm (...) presents itself as vibration (...) and in the coupling with sensation, rhythm is already freed because it confronts and unites different levels of sensation: it is now resonance ' (1981: 33, 48).Deleuze later identifies a similar process in cinema whereby the impression of movement derives from the spaces-in-between and the articulation of a film's rhythms and intensities (1983).
In Paroles trouvées, the relationship of rhythm, sensation and movement is expressed in the soundscore in the multiple encounters of sound objects in space.Along with the intensities, colours and gestures emerging in the visual interface, a synaesthetic effect is generated.Sound, sensed physically in terms of its vibrations, direction, distance, intensity, type and meaning, coalesces and harmonises with phenomena of movement, vibration, pulsation and light emanating from the choreovideographic score.However, to speak of intertextuality or mutually reflecting equivalences is not quite adequate: the terms environment and immersion seem more appropriate.Somewhat mysterious and defying explanation, physical experience of the work and observation of public reactions confirm the work's particular impact.

Optical effects on image
The optical score is the last parameter of the project to be addressed.The water's movement, being of both a physical and optical nature, was conceived as a kind of connective membrane linking the work's two qualities of space, aural and visual, while also inscribing a notion of the passage of time.The optical device was to serve the work's formal objective of generating a specific experience in which the eye listens, while also producing physical sensations reminiscent of our aquatic origins.Calling into play the spectator's imagination and body in a moment of interiority, the device creates 'an environment as a permeable sensory membrane' (Besson) and, challenging the primacy of sight as a sense organ, its synaesthetic nature both addresses and envelops the spectator's body in the act of perception.
However, the optical effects were to drastically alter the nature of the visual field and thereby, the aesthetic, functioning, and editing procedures of the original video shot.Completion of the hydromechanical system occurred shortly before the work's premiere and long after the original edit I had made.Tests with the device thus came late in the process and were reft with surprises.The effect of water washing over video images, themselves in movement, produced an overly charged visual field and, as a consequence, a partial revision of the videochoreographic score became necessary to handle the problem, showing the unpredictable and often tenuous nature of interdisciplinary research.A creative solution arose by introducing blocks of photographic stills in the scenario when the water was active.The photos' fixity allowed for the movement of the water to be visible while respecting a sense of narrative flow.In hindsight however, had it been possible to compose the visual score with the optical device from the outset, other and potentially more interesting effects might have been achieved.
In conclusion, it can be favourably argued that the metalanguage of digital interfacing has served to further the creation of new sensorial experiences in art.Digital works nevertheless continue to entail a delicate balancing of interdisciplinary elements and the manipulation of these elements neither conforms to uniform or self-evident procedures nor guarantees easy artistic collaborations or quality works.Additionally, the choice and nature of an artwork's particular medium will also condition this balance.In the final analysis, content, design and aesthetic remain fundamental to a work's coherence and reception.Though the bastion of contemporary art may have been irrevocably shaken by an ever-increasing plethora of digital creations, the integrity of artistic creation today, to paraphrase critic Annika Blunck referring to the advent of cinema and the 'trap of the cultural industry', lies beyond a transformation of the spectator into 'a perceptual apparatus (...) under the spell of (..) racing images ' (2002: 55) or, dare I add, into an interface between virtual worlds and reality at the mercy of commercial empires ... and artists.

February 7th, 2010
Dr. Andrea Davidson Senior Lecturer in Dance and New Media University of Chichester