Start Submission

Reading: Affectual Dramaturgy for Augmented Reality Immersive Heritage Performance

Download

A- A+
Alt. Display

Research

Affectual Dramaturgy for Augmented Reality Immersive Heritage Performance

Authors:

Mariza Dima ,

Brunel University London, GB
About Mariza
Dr.
X close

Holly Maples

University of Essex (basically Mariza Dima is at Brunel University and H Maples is affiliated with University of Essex), GB
About Holly
Dr.
X close

Abstract

This paper introduces the concept of Affectual Dramaturgy as a lens for creating digitally mediated immersive performances in the context of cultural heritage. In doing so we bring together two disciplines, smartglass Augmented Reality design (AR) and immersive heritage performance, with the aim to innovate experiences built for heritage sites.

For both disciplines, immersive experience building uses interactive methods to engage the public with the tangible and intangible heritage of a site. Immersive heritage performance incorporates narrative-led, affective, and ludic techniques found in virtual and live immersive and participatory theatre practice. Smartglass AR experiences communicate to the viewer how a place was in the past by means of superimposing on the physical space virtual material accompanied with audio.

However, often, the orchestration of the virtual material does not take the context into account and the superimposition of digital information misses an opportunity to connect into the existing narrative of the site tapping into its rich dramatic potential. This article explores how AR design and live performance can blend together through embodied storytelling techniques designed to draw the public’s attention to their sensorial experience of the site, and offer them a deeper understanding of the place’s history, which we refer to as affectual dramaturgy. By fusing embodied, affectual, and sensorial experience into the public’s engagement with the site, this article explores how interdisciplinary collaborations between theatre and AR design may create innovative tools and methods to tell stories of the past for the 21st century museums.

How to Cite: Dima, M. and Maples, H., 2021. Affectual Dramaturgy for Augmented Reality Immersive Heritage Performance. Body, Space & Technology, 20(1), pp.25–36. DOI: http://doi.org/10.16995/bst.368
63
Views
10
Downloads
2
Twitter
  Published on 05 Mar 2021
 Accepted on 07 Jan 2021            Submitted on 01 Nov 2020

Introduction

In a typical visit of a heritage site, walking through the place, looking at and sensing the surroundings immerse the visitor in the historical experience. Audio guides, labels, leaflets, and tour guides provide additional information that connects the visitor to the place’s history, especially for sites that are quite empty of physical objects. Yet, the supporting material offered to visitors is often dissociated from the experiential demands of 21st century audiences (Lang, C., & Reeve, J., 2016). Furthermore, such methods of engagement have become increasingly unsatisfactory as tools for the educational goals of many heritage sites which have developed ‘more sophisticated understanding[s] of the complex relationships between culture, communication, learning and identity’ (E. Hooper-Greenhill, 2007: 1). In order to lessen the dissociation, increase experiential and situated knowledge building, and invite younger audiences to museums and heritage sites, several performative tools have been introduced over the last few years, such as interactive scavenger hunts, theatrical performances, dramatized walking tours, immersive storytelling, AR and VR technologies, and multimedia (Gheorghilas, et als, 2017). These tools allow visitors to not only witness historical narratives, but physically and socially interact with a localized past (Auslander, 2013).

Through collaborations with professional theatre makers, performers, digital technicians and designers, the heritage industry is increasingly using sensorial storytelling devices to enhance participatory experiences between audiences and heritage sites (Kidd, 2018). In immersive heritage performance, the public become a part of, rather than separate from, the story being told. At the same time, heritage organisations are exploring the use of immersive technologies such as Augmented and Virtual Reality (AR/VR) to create more detailed reconstructions of the past and attract younger audiences. What is common in these experiences is the use of storytelling to guide the visitor, a powerful tool museums and heritage sites use to construct nuanced historical details and create experiential frameworks that encourage internal dialogue, connections with the past, and provide complex and satisfying engagements for visitors (Bedford, 2001; Wong, 2015).

However, in heritage performance, immersive storytelling is often framed in a linear, didactic way, one that critics find stripped of its ability to evoke emotional and affective responses (Tivers, 2002). On the other hand, the use of digital technology in AR experiences can appear disjointed and may lack the logical journey found in more theatrical heritage experiences leaving users less connected to the historical subject matter. The aim of this paper is to introduce a performance methodology for the heritage industry that combines AR design with immersive theatre techniques focusing on sensorial and interactive storytelling interweaving digital technologies, the heritage audience, and the historical setting. By combining immersive theatrical storytelling devices with digital design technology, the sensorial engagement of the AR device is enhanced through an embodied storytelling design, a technique we describe as affectual dramaturgy. This technique was created to fuse the users’ interactions with and through the AR technology with the historical characters, the story, the journey, and the site of the experience. Affectual Dramaturgy has the power to enhance visitor experience by increasing historical knowledge through emotional engagement and contextual understanding, leading to a more personal and complex relationship to the heritage site (Perry et als, 2017; Tivers, 2002). The following introduces the concept of Affectual Dramaturgy as a lens for creating digitally mediated immersive performances for the heritage industry.

Immersive Performance

For both digital technologies and live performance genres, immersive experience building uses interactive methods to engage the public with the tangible and intangible heritage of a site. The term ‘immersive’ has been both broadly defined and contested in both the live and virtual performance genres, and yet continues to be used as a term for a type of audience experience found in the industry. The authors are not arguing that immersive theatre is unique in providing a sensorial experience by its audience; it is merely a tool to draw audiences’ awareness to the sensorial aspects of what occurs in every viewing experience within the dramaturgical process. The participatory nature of immersive experience, whether live or virtual, has proved popular to both attract and educate audiences through their emphasis on embodied and affectual engagements with heritage sites, historic artifacts, or archival material on display. Immersive experience encourages multi-sensorial modes of audience engagement while also created thematic relationships enhancing spectators’ imaginative response to the space, the subject, and the performers (Machon, 2013). Moreover, it is the sensorial experience of the event that particularly defines the immersive experience’s relationship between spectator and performance.

Immersive experience design has drawn interest from the heritage industry for its potential as a pedagogical tool because of its focus on participatory audience interaction to ‘animate’ the historical content, objects, and characters of the site (Maples, 2021). In Maples heritage performance at Oxnead Hall in Norfolk, audience members described how much easier it was to absorb information through the performance. They reflected that heritage sites ‘tend to have limited entertainment or engagement value, so what you learn is not so well retained,’ while the Paston heritage performance did not feel ‘touristy’ but rather allowed them to ‘indulge in a genuine sense of being transported through time to another era’ (SurveyMonkey, 2018). There is a reciprocity in sensorial engagement, allowing for an intimate relationship between the environment and the individual or group. In the context of the heritage industry, Auslander’s (2013) definition of the ‘intimate trafficking’ found in immersive heritage performance creates a connection between the ‘mind’ and the environment allowing for an imaginative, affectual relationship between audiences’ present and the heritage site’s past. By combining historical objects and archival documents with a lived sense of historical storytelling, immersive heritage performance can engage in audiences’ affectual experiences of history. Such techniques turn heritage sites into what Lucas describes as places of ‘symbolic action’, theatricalizing the space to give it a narrative relationship with its audience. Theatre makers are increasingly employed at museums and other heritage sites to ‘better immerse guests and effectively play out the theme or context of the spaces at hand’ (Lucas, 2016: p4). Interest in this phenomenon is seen in the growing industry of immersive technologies and participatory theatrical techniques found in the heritage industry to reach new audiences and create an animated museum of the 21st century (Cotter, 2015).

Maples’s recent immersive creations for the heritage industry have included site specific performances, immersive storytelling, dramatized tours, and scavenger hunts at museums and heritage sites across Ireland and the UK, combining historical dramatized storylines from archival sources with immersive theatre designed to enhance audiences’ affectual, imaginative, and embodied response to history. Dima has experimented with dramaturgy as a design methodology to create an AR-enabled experience through which audiences engage with the history of a heritage site and the many stories the building and its objects tell through that site. Through our collaboration, we aim to create hybrid performances in heritage spaces designed to let the audience members walk into history, and doing so, imagine the challenges of living through a time period, rather than looking back at it.

Immersive techniques are used to create ‘not simply a meta-theatrical work used in the heritage industry, but one which is meta-historical’ (Maples, 2021). By drawing the audience’s attention to the characters and their story, the authors desire to build an imagined, ‘lived’ historic environment for the audience and one that complicates our understanding of that environment and its relationship to the past. Affective dramaturgy’s focus on experiential audience response from within the storytelling structure, is designed to encourage a sensorial experience of history, provoking an empathic relationship to historic characters and events by members of the public through physical sensation.

Smart Glass AR Experiences

AR technology projects computer generated digital material such as 3D artefacts, video and images onto the real environment. In its inception experimental work used a pair of smart glasses to superimpose the digital material on the real setting, however, with the rapid advancement of mobile technology, mobile AR became an easy, affordable, and well adopted technology, especially in cultural heritage.

In the last years, the development of AR has seen the return to developing a new type of headsets that look like a pair of glasses. These use advanced optics technology to project 3D computer generated imagery on the lenses. The earliest example was the, now discontinued, Google Glass project, while at the moment there are a few headsets on the, rapidly growing, market (e.g. Microsoft’s Hololens 2 (2020) and Magic Leap One (2020)). AR headsets superimpose 3D computer generated imagery onto the real world, producing a similar result to mobile AR. As opposed to VR headsets, AR glasses do not create a totally virtual environment, allowing the user to have a clear view of the physical space. The difference of using an AR headset to using the mobile is the opportunity and potential for full body interaction, since the hands are not engaged (Vainstein, Tsvi & Joel, 2016). Looking through a headset while not losing the view of the physical environment and being able to have the hands free is a more immersive experience than using the mobile (Mason, 2016). In addition, the experience design can make use of a variety of sensorial input, such as eye tracking and voice, to enable further interaction mechanisms that may be more natural to visitors of all ages. The majority of research on AR headsets have so far focused on areas of science and ergonomics (Mason 2016; Mokatren 2018, Vainstein 2016; Kerr 2011) such as navigation to a target location, situational awareness, and augmented imagery visibility. At the same time, an increasing number of applications are being designed for different contexts, and indoor and outdoor cultural heritage sites have been a fruitful ground for research and experimentation with AR. The hardware interface is visibly lowering the barrier to accessing and communicating cultural heritage information (Litvak 2020) and scientific research helps with improving the ergonomics with each new model. This is particularly important for heritage site visits as they require the interface to be as transparent as possible so that the focus of the visit is put on the navigation through the site for learning and social interaction.

Although there is more than 20 years of research on AR in cultural heritage, the literature focuses mainly on technological factors and very little on the context in which it is used. A few projects have addressed the necessity to utilise the power of stories (e.g. Liestoel 2019; Spierling 2014), however, these too have approached storytelling as a linear, didactic experience with stories being object-centric or applying simple game mechanics in a way that the experience is primarily a game that includes a learning aspect. In the area of AR headsets, with its immense potential for engaging visitors, the few heritage applications rely on the novelty of the technology and do not employ any narrative. A few indicative apps that have been developed over the last four years include the TouristicAR system, an AR application that provided context-aware content for tourists at UNESCO World Heritage sites in Malaysia (Obeidy, Arshad & Huang 2018), (MR Museum in Kyoto 2018)’s Kennin-ji, a 10-minute AR experience that combines The Folding Screen of Fujin and Raijin with 3D graphics, and ‘HoloMuse’ that allows user interaction with archaeological artefacts from the Anonymous Museum’s collection (Pollalis et als. 2017). With the lack of attention to the dramatic potential of the headsets these works miss the opportunity to connect into the stories of the site, and touch on the intangible heritage and the physical surroundings and bring them together in a unifying narrative that creates an affectual experience. Aspects of the experience such as the use of multisensorial modalities, points that provide opportunities for interaction, playful encounters, and emotional and moral engagement have proven to motivate audiences and offer a meaningful dialogue that helps them learn and be entertained (Vagnone 2015). AR headsets provide a different experience and, as such, open up a plethora of different opportunities for connecting to the past.

As with every new interaction technology, along with scientific research it is necessary to research the design processes under which the interface and the context are tied harmoniously together to produce a meaningful experience (Dima forthcoming). Mason (2016) and Tom Dieck (2014; 2016) used early prototype headsets to study user interaction through AR headsets in a cultural context, mapping requirements and offering recommendations based on the use of prototype experiences in cultural places. Hammady (2019) explored consideration for the design of the visual interface with Microsoft Hololens 1, and Pollalis et al. (2017) studied gesture-based interactions with holographic artefacts. In the Sutton House Stories project Dima researched the application of dramaturgy as a design methodology to create an affective AR experience using Microsoft’s Hololens 1. Dima (forthcoming) suggested that looking at the design of these AR experiences through the lenses of dramaturgy offers a fresh perspective in creating experiences that are immersive not only because of the technology used but because of their design. The methodology focuses on orchestrating elements within and across three intertwined design pillars, that of narrative design, technological considerations, and learning objectives for the heritage site. Design pillars, a term used in games development, are a list of no more than three or four design goals that act as a compass for the development of the interactions and help keep a clear vision of the final work. Dima’s design approach offers a holistic view of the experience without focusing only on simple superimposition of the virtual material but considering how to work with this material in ways that make up an engaging encounter with the past. The desired outcome is that when visitors leave, they have not only looked at fragments of the place’s history but also walked through the memories of the place. (Figure 1).

Figure 1 

Visitor looking at superimposed content through the Hololens AR glasses, Sutton House Stories, 2019.

This methodology was used to create Sutton House Stories, a 4 minute AR experience designed and developed in 2019 for Sutton House, a Tudor house in the London Borough of Hackney (UK). The narrative was developed based on the curatorial requirements of the site and a lot of the content was based in the house’s archives. The viewer walks around the Great Chamber, guided by three voices representing three key figures that lived in the house across four centuries. Professional voice actors were employed to act out the script in the form of a monologue as if they are talking to the viewer. While viewers listen to the voices, they see and hear superimposed content which may be related to what the narrator tells them or may not be. At certain points they also have the opportunity to interact although this is not made explicit to them. For example, there is a sequence where a pair of dance shoes perform a minuet (Figure 2).

Figure 2 

Dancing the Minuet, Sutton House Stories, 2019.

Viewers are free to walk around the dancing shoes and watch their dance and if they feel like it try to mimic the steps. A few participants did so or thought of doing so during the evaluation. When asked how the experience felt, the majority of the participants used the words ‘the house came to life’. This sentence reveals that the viewers were not just engaged or immersed in what they were experiencing but that they viscerally felt closer to the house’s history and its past inhabitants. Dima found that this feeling is something that can be enhanced by carefully orchestrating the materials of the experience, which include but are not limited to the 3D computer generated superimposed material. Grabbing and manipulating a holographic object is impressive on its own but if this act is placed within a wider story that unfolds from the moment the visitors wear the AR headset it acquires a more meaningful stance. The third pillar, narrative, can be viewed as a source of stories attached to the site but also as a means to bring the visitor experience under a unifying narrative. In Sutton House Stories this narrative was revealed at the end with one of the characters asking the visitors to share their story so that it is not forgotten. In this way the experience concludes with a moment of reflection by means of direct dialogue with one of the characters that accompanies the visitor through their journey.

Affectual Dramaturgy

We have seen so far how immersive heritage performance and smart glass AR design have separately the potential to create an embodied and affective experience of a heritage site. Deriving from our individual practices and in light of the need for new perspectives in digital heritage design is the exploration of a new form of audience interaction with, and participation in, cultural heritage. How best can the two practices be combined and what opportunities does this create for research, practice, and audience development strategies? By making a few steps towards a new perspective for creating such experiences in heritage sites, we evolve Dima’s dramaturgical approach and look at what affective dramaturgy entails, and how narrative design and technological considerations change to accommodate and enhance the affective qualities.

Combining live immersive performance and enhanced digital experiences, such as the AR design, allows audience members to encounter digitized historical performers, sounds, and objects in and around them, just as they encounter the heritage site, fusing the real with the virtual. Furthermore, interactions with live performers may also enhance an ‘intimate trafficking’ between an audience member and the historical characters and events (Auslander, 2013). Staging intimate encounters between live actors and audience, (such as performers whispering in the ear of an audience member, delivering a letter to them, or seeing them move in the distance), may enhance sensorial and embodied sensations and reactions from members of the public. By combining the virtual with the live performance, the richness of detail available through the digital design may be further enforced through encounters with live performers.

Our hybrid methodology combines AR design with immersive theatre by designing the visitor experience through an embodied sense, or indeed, affectual notion, of dramaturgical storytelling. Ann Pais distinguishes affect from the ‘emotion’ which is a common terminology for theatrical performance. For Pais (2016), ‘affect’ is an embodied experience of ‘social atmospheres’, one that enhances ‘the sensitive charges or felt intensities carried by words, sensations, thoughts and emotions that circulate in social spaces (3)’. Our definition of immersive storytelling as ‘affectual dramaturgy’ embraces the idea that immersive experience creates an embodied and sensorial relationship to the function of storytelling in the heritage site. The story is not solely material, fact-based, or text driven, but a lived, and embodied, experience which relies on affect as a physical, sensorial, and imaginative, act for the participant.

Cross-disciplinary methodology of sensorial research in this project allows for a multivalent understanding of human nature and society through public reception of heritage performance. The enhancement of sensorial experience for audiences in both the heritage and theatre industry have included touch, taste, smell, and sound as important ways to both engage and challenge audiences in their experience of a site. By creating immersive heritage performances through a dramaturgical focus on affectual, embodied experiences which allow the visitor to experience a heritage site as a sensorial, embodied event, the entire relationship between the museum goer or heritage audience and the site becomes a lived experience of the past.

Design Pillars For Affectual Dramaturgy

With Affectual Dramaturgy we widen the technology and narrative aspects of Dima’s methodology, to include interaction methods and processes from immersive heritage performance that enriches the dramaturgical process with affectual and embodied qualities. We integrate these methods with consideration of how immersive technologies can enable and enhance these interactions. The design pillars of Affectual Dramaturgy are focused on the experience of the user through their journey, the story being told of the heritage site, and the integration of the live and virtual experience.

Participatory Journey

One of the important aspects of the hybrid experience is the guiding and occasionally leading of the visitor throughout the experience and in the time shortly before it begins, what is usually called ‘user onboarding’. More often than not AR experiences start when the visitor is on site, and if there is a mobile AR application perhaps the visitor has been asked to download it on their phone prior to the visit. Immersive performance, by contrast, often incorporates the introduction of the visitor to the experience as a key part of the dramaturgical structure, with a character, or characters, greeting the audience and explaining the rules of ‘the game’ through world building devices which incorporate all aspects of the experience into the logic of the story. The AR headset can be integrated into the story through the use of a live performer explaining the magical or time travelling world of the AR experience and the visitor’s role in it.

Before or during the entrance to the site is when the AR headset becomes merged into the story an actor can be used to introduce the visitor to the experience while also offering key information the visitor needs to know about the device and rules of the journey they will undertake. The narrative should answer questions such as: What is this device for? Why does the visitor need it? What role does it play in their journey? This leads to questions about the role of the visitor who either constantly wears it or uses it at specific moments. For example, in workshops conducted using Dimas’s dramaturgical framework, the role of the ‘time traveller – historian’ was the most common suggestion. In this role the device can be a time travel device worn at all times, or a special magic lens through which one is able to see things of the past. In any case, the visitors need to be accustomed to wearing and using the device, and actors can help greatly with the smooth integration of this part into the story. Visitors, who are now viewers, need to be allowed some time to get used to seeing through the glasses, and practice some of the gestures that will be needed during the experience. Normally this part is done ‘before the experience begins’, but in this case the actors can weave this part into the story.

Maples’ immersive theatre techniques celebrate the sights, sounds, and smells of heritage location by drawing audience members’ awareness to them within the storytelling itself, and the performers’ looks, gestures, and choreography (Figure 3).

Figure 3 

Sarah Lawrie and Sarah Edwardson (actors), Oxnead Hall, 2018.

Maples uses not only performers, but objects and sounds, to guide visitors from site to site, allowing spectators to experience an environmental, as well as narrative-led, journey through time and space Storytelling may incorporate affectual dramaturgy devices by instructing the performers to enhance the environmental impact on the senses of the audience through the performers engaging in acts of listening and looking at the world around them. Through the performers’ attention to objects and the environment, the audience also begins to interact with the site in different ways. In Maples Paston heritage project in Norfolk audience members followed the performers across the vast grounds of a 17th century estate by following and interacting with actors and objects such as letters, funeral tokens, and flowers. Audience members reflected on how the participatory nature of the production enhanced their experience of ‘living history’ (SurveyMonkey, 2018). Members of the public mentioned how ‘[the play was] surreal as if we stepped back in time’, the past ‘came to life’, ‘history was brought to life’, or ‘I felt like Robert Paston was walking amongst us’, while other members of the audience spoke of the feeling of moving ‘in and out’ of the past as they followed, interacted with, and were guided by performers on their route through the heritage site (SurveyMonkey, 2018).

This technique creates a kind of performative mindfulness, bringing the site to the foreground of attention (Figure 4). Similarly, in Dima’s Sutton House Stories, environmental virtual materials were an important aspect of the experience. There was a virtual fire burning in the actual fireplace during the whole time with fire cracking sounds on the background of the monologues, and bird chirping when a virtual window opens on top of a real one to reveal the countryside as it used to be 400 years ago (Figure 5).

Figure 4 

Ben Porter (actor), Oxnead Hall, 2018.

Figure 5 

Virtual window opening to the view of the country side, Sutton House Stories, 2019.

According to Rickett, the use of affect in the heritage industry incorporates, ‘a wide range of social, sensory, and emotive registers’ into the meaning making constructed at the site’ (Fisher et als, 2015). Participatory performance practices invite the public to contribute to a multisensory experience of the site. They also allow for a level of improvisation and personalized experience which can enhance the public’s affectual, temporal, and embodied relationship to the heritage site.

Giving a Story to Heritage Sites

Heritage organizations often desire to have a variety of experiences, characters, and historical stories of a building which, like that found at Sutton House, span many centuries. This can be challenging for creating a cohesive experience for visitors to follow. There are ways to link the experience of a time-travelling visitor through the use of objects, characters, and themes. To draw connections between stories and time periods in a heritage experience, objects may be followed through the viewer’s journey, linking up a story from one location to the next. This can be done by leaving a trail of virtual instances (visual and/or oral) of an object that the viewers get in the end in its physical form connecting in this way the two worlds. Equally, viewers can see or hear a historical figure before they actually meet them (the actor/tress) later in person. In Maples’ Paston Footprints project, letters were used to travel from location to location, connecting four hundred years of stories through the receiving and reading of letters, central to the Paston family history (Maples, 2020). This technique can also be used to integrate the world of the AR experience with the live performance. A character first heard through the AR headsets may later appear in the flesh, while objects introduced through the AR experience may also reappear in later sites and possibly as a living object used by an actor or, indeed, given to an audience member. The interweaving of a live and virtual experience may further enhance the imaginative journey of the viewer through the heritage site.

Games design elements of the dramaturgical design also can enhance the experience of the viewer by making them active participants in the experience. Making, for example, the viewer a detective who has to discover and connect disparate clues incorporated into the dramaturgical design, allows them to follow the story through the experience. However, we argue that a fine balance needs to be maintained in this case so that the gameplay does not dominate and make finishing the game the experience’s main goal. There are audiences who may not be comfortable with such a level of activity and a slower pace allows for unfolding the affectual engagement better.

Meaningful Interactions in a Hybrid Space

It is important to cohesively link the live element of the performance with the virtual AR experience in a fluid and seamless manner. This aspect of the performance needs to be clearly choreographed with a well-designed logic that is integrated in the dramaturgical design. The live performers can play key roles in continuing the user’s engagement with the digital world. Theatrical techniques can be used to guide the viewer to look or move in the direction of where the next virtual material will be placed, while the performer’s movement can be choreographed to anticipate the next location of the AR experience. The focus of the viewer may also be guided by incorporating light and shadow in strategic ways in the room, by an actor’s gaze, focus, and/or movement, and even the location of a virtual object in the AR experience. Objects, sounds, and even characters which appeared in the virtual world through the AR experience may reappear in the live one, or vice versa, to further fuse the hybrid experience of the production. All of these techniques may be designed to direct the audience/viewer’s journey through the story and the heritage site, carefully integrating all aspects of the experience.

There is a challenge with having actors responding to virtual events in a natural manner that has to do with them knowing where the virtual material is displayed so that any bodily interaction with it is accurate. Actors should also know where to move so that they do not get in the way of the perceived distance between the viewer and the virtual material as this will break the experience. To this end, AR should be considered another actor in the company, to be rehearsed with and reacted to, and in this way considered a living part of the experience being constructed and created through the hybrid performance. The same may be said for the live performers, with the AR designer including the actors and the performance ‘script’ in their creation of the design. It is essential that the AR design and performance work in close collaboration to create the total experience for the audience.

By combining these devices between the live and virtual experience of AR and immersive theatre, the visitor’s engagement with site and story can be enhanced through the detail provided by the AR design and the ability of live performance to draw the viewer’s attention to distinct aspects of the experience while sparking an increasingly empathic and imagined understanding of the heritage site. During the journey from room to room and time to time, viewers, who are now active participants, engage with the actors, the space and the virtual material. This three-way engagement should be done as smoothly as possible to blur the physical/digital boundaries, bring the site to life, and maintain the ‘suspension of disbelief’ as much as possible. One way this can be achieved is by using methods that cognitively trick the brain into perceiving that the virtual material is really present. For example, actors can interact or respond to events that happen virtually e.g. an actor can warm their hands over a virtual fire that is burning in the fireplace accompanied with fire crackling sounds, or react with surprise when an ethereal virtual creature appears in the room. Actors can also call the viewer to eavesdrop on a conversation taking place in an adjacent room among virtual characters and react to what is being heard by the viewer as if they have also been listening to it at the same time.

When allied with other sensorial experience, such as existing sights and smells, sound can engender an experiential journey for users into different times and places. The Gloucestershire company PastPorte found, through their crafting of audio walking tours for the National Trust’s arts and crafts garden at Hidcote, the possibility to transport visitors into other time periods goes beyond present capabilities by integrating the recorded audio experience with the existing landscape, using geolocation technology to trigger sound as a user moves through space, using VR equipment to inscribe historic scenes on the current landscape, and by recording 3D audio for a sophisticated surround sound experience (Blackler, 2015). Using these methods enhances the feeling of being embodied in the hybrid world as all actions make sense cognitively to the viewers. In many ways, the marriage of technology, affectual dramaturgy, and live performance follows Wagnerian notions of the ‘total work of art,’ combining ‘meaning and message’ with design aspects to sculpt the viewer’s experience of the heritage site (Machon, 2013, pp 21–22).

Conclusion

We have outlined the first steps towards a methodology for creating immersive experiences in heritage sites that can bring their histories to life through affectual dramaturgy. We have introduced methods for engaging emotionally the viewer that derive from theatre practice and are enriched with opportunities offered by the technological devices. Under this new perspective, affective interactions are at the centre of the creation process while the acting and technology design support their creation.

This work is timely as AR hardware is rapidly developing so that AR smart glasses could be the next commodity immersive technology. With audiences growing more used to immersive technologies, a future where visitors will be entering a site with their own pair is not far away. In addition, AR allows a shared hybrid experience, which feels more natural than collaborative VR as parties are constantly in the real world and able to see each other there. The affectual dramaturgy perspective can help AR users through the change of their embodied sense when wearing the headset by providing tools to make the integration of the device into the storyworld seamless. Finally, our work emphasises the importance of cross and transdisciplinary collaboration. Digital interpretation is increasingly becoming the focus of heritage organisations and the heritage staff’s skill set is being updated extensively. Though there is not always the resources to combine live with the virtual AR experience in a work, we argue that the fusion of different disciplines, from technology-oriented experiences to theatrical embodied storytelling remains fundamental to creating digitally-mediated, affectual, interactive, historical narrative experiences designed to successfully immerse visitors in the stories and histories of the heritage site. Our proposed methodology is an example of different disciplines working together in a way that can open up opportunities for new skills and business models. For the heritage industry this can mean new methods of work to attract diverse publics and new roles that rise at the intersection of heritage education, immersive performance, and digital design. Equally, creative industries such as theatre and digital design can find a new outlet for their skillset that can offer resilience and innovation in current uncertain futures.

Competing Interests

The authors have no competing interests to declare.

Author Information

Dr Mariza Dima is an interaction designer and Lecturer in Games Design with a focus on Creative Technology for Games at Brunel University London. With a background in Human-Computer Interaction (HCI) and User Experience design, she has worked with physical, mobile, and haptic interfaces to serve interactions in socio-cultural contexts, prominently in performance art, cultural heritage and social innovation. She has led several projects in partnership with prestigious theatrical companies, museums, cultural organisations, SMEs, and visual arts institutions, and published widely on audience engagement, digital media and immersive technologies.

Dr Holly Maples is the Director of Impact and Post Graduate Research at East 15 School of Acting, University of Essex. She is a theatre practitioner, educator and scholar. Her performance practice particularly focuses on dramatized immersive and sensorial experience techniques in the heritage industry. From 2017–2020 Dr Maples acted as Drama lead on two distinct HLF and Arts Council funded heritage projects in Norfolk with the Paston Footprints project and the Norwich Castle Museum. In 2019–2021 she further created a heritage performance exploring the 18th century slave trade and abolitionist movement in collaboration with historian, Dr Inge Dornan.

References

  1. Auslander, M. 2013. Touching the Past: Materializing Time in Traumatic Living Reenactments. In: Signs and Society. Spring, 1(1): 161–183. DOI: https://doi.org/10.1086/670167 

  2. Bedford, L. 2001. Storytelling: the real work of museums. Curator, 44: 27–34. DOI: https://doi.org/10.1111/j.2151-6952.2001.tb00027.x 

  3. Cotter, H. 2015. Towards a Museum of the 21st Century. The New York Times, 1 November, Section F, 10. 

  4. Dima, M. forthcoming. Sutton House Stories: Augmented Reality Experience Design Through the Lenses of Dramaturgy. In Touching Past Lives: Immersive Heritage Performance, London: Routledge. 

  5. Fisher, J and Reckitt, H (eds.). 2015 Museums and Affect, Journal of Curatorial Studies, 4(3): 2045–5836. DOI: https://doi.org/10.1386/jcs.4.3.361_2 

  6. Hammady, R and Ma, M. 2019. Designing Spatial UI as a Solution of the Narrow FOV of Microsoft HoloLens: Prototype of Virtual Museum Guide. In: tom Dieck, M, Jung, T. (eds.), Augmented Reality and Virtual Reality. Progress in IS. Switzerland: Springer, Cham. pp. 217–231. DOI: https://doi.org/10.1007/978-3-030-06246-0_16 

  7. Kerr, S, Rice, M, Teo, Y, Wan, M, Cheong, Y, Ng, J, Ng-Thamrin, L, Thura-Myo, T and Wren, D. 2011. Wearable mobile augmented reality: evaluating outdoor user experience. ACM, New York, NY, USA, pp. 209–216. DOI: https://doi.org/10.1145/2087756.2087786 

  8. Kidd, J. 2018 Immersive’ Heritage Encounters. The Museum Review, 3(1), Website: http://articles.themuseumreview.org/tmr_vol3no1_kidd 

  9. Litvak, E and Kuflik, T. 2020. Enhancing cultural heritage outdoor experience with augmented-reality smart glasses. Personal and ubiquitous computing, 24: 873–886. DOI: https://doi.org/10.1007/s00779-020-01366-7 

  10. Machon, J. 2013. Immersive Theatres: Intimacy and Immediacy in Contemporary Performance. Basingstoke: Palgrave. DOI: https://doi.org/10.1007/978-1-137-01985-1 

  11. Magic Leap One. 2020 Available at: https://www.magicleap.com/en-us/magic-leap-1 

  12. Maples, H. 2021. Touching Past Lives: Sensorial Spectatorship in Heritage Performance. In Snyder-Young, D & Omasa, M. (eds.), Researching Audience Impact. London: Routledge. 

  13. Mason, M. 2016. The MIT Museum Glassware Prototype: Visitor Experience Exploration for Designing Smart Glasses, Journal on Computing and Cultural Heritage (JOCCH), 9(3): 1–28. DOI: https://doi.org/10.1145/2872278 

  14. Microsoft Hololens 2. 2020 Available at: https://www.microsoft.com/en-gb/hololens/hardware 

  15. Mokatren, M, Kuflik, T & Shimshoni, I. 2018. Exploring the potential of a mobile eye tracker as an intuitive indoor pointing device: A case study in cultural heritage, Future generation computer systems, 81, pp. 528–541. DOI: https://doi.org/10.1016/j.future.2017.07.007 

  16. MR Museum in Kyoto 2018 Available at: https://hakuhodo-vrar.jp/kyoto2018/en/ 

  17. Obeidy, WK, Arshad, H and Huang, J. 2018. TouristicAR: A Smart Glass Augmented Reality Application for UNESCO World Heritage Sites in Malaysia. Journal of Telecommunication, Electronic and Computer Engineering, 10: 101–108. 

  18. Pais, A. 2016. Re-Affecting the Stage: Affective Resonance as the Function of the Audience, Humanities, 20 September. DOI: https://doi.org/10.3390/h5030079 

  19. Perry, S, Roussou, M, Economou, M, Young, H and Pujol, L. 2017. Moving beyond the virtual museum: Engaging visitors emotionally. 23rd International Conference on Virtual System & Multimedia (VSMM), Dublin, 1–8. DOI: https://doi.org/10.1109/VSMM.2017.8346276 

  20. Pollalis, C, Fahnbulleh, W, Tynes, J and Shaer, O. 2017. HoloMuse: Enhancing Engagement with Archaeological Artifacts through Gesture-Based Interaction with Holograms, ACM, New York, NY, USA, pp. 565–570. DOI: https://doi.org/10.1145/3024969.3025094 

  21. Spierling, U and Kampa, A. 2014. Structuring Location-Aware Interactive Narratives for Mobile Augmented Reality. In: Mitchell, A, Fernández-Vara, C and Thue, D. (eds.), Interactive Storytelling. ICIDS 2014. Lecture Notes in Computer Science, 8832: 196–203. Cham: Springer. DOI: https://doi.org/10.1007/978-3-319-12337-0_20 

  22. Survey Monkey, Feedback forms, Robert Paston’s Whirlpool of Misadventures, Paston Footprints, Oxnead Hall, Norfolk, 18 June, 2018 

  23. Tivers, J. 2002. Performing heritage: The use of live ‘actors’ in heritage presentations. Leisure Studies, 21, 34, 187–200. DOI: https://doi.org/10.1080/0261436022000030678 

  24. Tom Dieck, MC, Jung, T and Han, D. 2016. Mapping Requirements for the Wearable Smart Glasses Augmented Reality Museum Application, Journal of Hospitality and Tourism Technology, 7(3): 230–253. DOI: https://doi.org/10.1108/JHTT-09-2015-0036 

  25. Vagnone, F, Ryan, D and Cothren, O. 2015. The Anarchist Guide to Historic House Museums: Evaluation Methodology for Historic House Museums. The Public Historian, 37(2): 97–111. DOI: https://doi.org/10.1525/tph.2015.37.2.97 

  26. Vainstein, N, Kuflik, T and Lanir, J. 2016. Towards Using Mobile, Head-Worn Displays in Cultural Heritage: User Requirements and a Research Agenda, ACM, New York, NY, USA, pp. 327–331. DOI: https://doi.org/10.1145/2856767.2856802 

  27. Wong, A. 2015. The whole story, and then some: ‘digital storytelling’ in evolving museum practice. In: Museums and the Web 2015. Retrieved October 1, 2019 from http://mw2015.museumsandtheweb.com/paper/the-whole-story-and-then-some-digital-storytelling-in-evolving-museum-practice/ 

comments powered by Disqus