Lost Origin is the title of an ‘Audience of the Future’ demonstrator project (2019-2022) which was funded as part of the Industrial Strategy Challenge programme delivered by UK Research and Innovation (UKRI). This report aims to provide an overview of the evolution of the project, focussing on the development of the story in relation to the immersive design process, showing how aims and objectives changed as a consequence of the onset of the COVID-19 pandemic, and present findings that could be useful for the design of future mixed reality experiences with theatrical components. Overall, the project aimed to better understand how cultural experiences could be reimagined for audiences of the future to boost understanding of the natural world and science, while at the same time facilitating museum learning, providing commercial touring and export opportunities. The proposed experiences intended to influence how the cultural heritage sector might expand in the 21st century, how it might reach new audiences, for example, by placing museum-content in non-museum settings, such as shopping centres, to reach audiences who might not otherwise visit or be exposed to museum collections. The format was meant to be scalable and hence applicable to different gallery and museum contexts. Modular storytelling was to be used to facilitate a shared collaborative experience in which the audience worked as team. The narrative was meant to be embedded in the design and in the technology to ensure that they all complemented each other.
The purpose of the Demonstrator programme was to support a series of consortia to create immersive experiences for large audience groups in the theatre, moving image, eSports and in the museum sector. The specific Lost Origin project, originally entitled Dinosaurs and Robots, led by the London-based immersive and television content production company Factory 42, specifically aimed to develop two mixed reality theatrical experiences in the Science Museum and Natural History Museum in London. The original vision had been for visitors to play detectives and interact with a cast of actors and digital characters, including dinosaurs and robots. At the Science Museum, visitors were meant to enter an advanced robotics centre and be presented with a security breach, and at the Natural History Museum audiences should have entered the museum to investigate strange happenings and uncover a mystery involving a Victorian palaeontologist and some ghost dinosaurs. Mood-boards were produced that even at an early stage in the project illustrated the intention to create a sci fi world for the Science Museum and an archaeological excavation, a library and forest for the Natural History Museum. The theatrical dimension, rendered through a set, narrative, and performers acting in character, was meant to bring together all other components for each of the experiences. Sampler versions were also meant to be made available to visitors at several Intu shopping centres.
The project comprised a wide number of partners with a range of over-lapping objectives. For Factory 42, whose wider mission was to change the way people experience entertainment through novel forms of storytelling and the creation of enhanced realities, part of the value of the proposed work laid in the building of a proof of concept and the creation of a sustainable business model for immersive experiences, as well as the identification of a pool of engaged visitors to target for future audiences. For the museums, the National History Museum and the Science Museum Group, the project aimed to attract new audiences, use new technologies to communicate science content, and generate a novel form of engagement with their respective collections that was entertaining and remained loyal to their educational briefs. For Sky and Magic Leap, the technology provider, it was crucial to learn more about how the technology could enable new forms of immersive storytelling and new audience experiences. The Almeida was interested in seeing how the use of storytelling would affect actors in bridging between physical and digital worlds. For the University of Exeter, the aim was to devise novel forms of documentation to understand the experience design and the creative process. For UKRI, the objective was to explore what forms of immersive content engages audiences of the future by experimenting with new forms using IP that would be already known to audiences. Overall, the main challenge of the project was to attract new audiences into museums by working in mixed reality with a wide number of briefs and objectives.
Museums have long been providing novel forms of technologically mediated experiences which have not only extended their visitor reach, generating new revenue nationally and internationally, but also provided original ways to engage audiences with their collections. While museums nowadays routinely host augmented and immersive experiences, there have, however, not been many examples to date of scalable and modular mixed reality theatrical experiences in the sector, by which I mean experiences including live actors, sets, and a certain suspension of disbelief.
A wide range of technologies are used in the design of cultural experiences for education, exhibition enhancement, reconstruction and the creation of virtual museums (Bekele et al 2018). These include augmented, virtual and mixed reality. The latter term, which is especially relevant for this project, indicates an overlap between physical and virtual worlds produced through the use of a range of more or less immersive technologies. The term was first discussed by Paul Milgram and Furnio Kishino, who proposed the idea of a ‘mixed reality continuum’, which connects what they called ‘real’ and ‘virtual’ environments, including intermediate points spanning from ‘augmented reality’ to ‘augmented virtuality’ (1994).
Mixed Reality has been successfully used in several contexts spanning from training and medicine to entertainment. While in performance mixed reality and immersive storytelling have been used for some time, for example, by Brighton-based collective Blast Theory (Benford and Giannachi 2011) and London-based company Punchdrunk (Machon 2018), in the museum context, the use of mixed reality is still relatively new, though examples developed by Microsoft have been piloted in recent years, and include the experience of a Ford car at the Petersen Locomotive Museum (2018); encounters with old Japanese artworks at Kennin-Jim, the oldest Zen temple in Japan (2018); a meeting with an astronaut in Defying Gravity exhibition at the Smithsonian (2018); and The Met Unframed (2021), an immersive virtual art experience which was developed during the pandemic. Interestingly, the Defying Gravity exhibition allowed four people to share the experience at the same time, while the Met Unframed, powered by Verizon 5G Wideband, made it possible to literally superimpose over 50 artworks from across the Met’s collection onto the walls of users’ own homes.
It has been shown that heritage providers hope that immersive experiences will ‘(a) increase visibility and contribute to a culture of innovation; (b) appeal to new audiences (c) allow for more meaningful participation, (d) facilitate better engagement and (e) provide additional revenue’ (Kidd and McAvoy 2019). Thus immersive experiences of art and heritage could enhance museum visiting, facilitate social interaction, emotional engagement, offer embodied and spatial interaction, and promote novel forms of learning, for example about sites which were destroyed or can no longer be accessed. Altogether the use of these technologies has been shown to generate deep spaces that can be penetrated both inside and outside the museum in which visitors continuously reposition their own presence across different temporalities and spatial configurations (Giannachi 2021). It is worth noting, of course, that theatre is in itself an immersive form and that it has, throughout its history, often used a wide range of technologies to deepen and widen its audience’s sense of immersion and presence.
The mixed reality technology selected for this project was Magic Leap which uses a head-mounted virtual retina display that superimposes 3D computer-generated images over real-world objects. The display works by projecting a field of light into users’ eyes. When using Magic Leap, it is literally possible to see the physical world and, blended within it, the digital, hologram-like world created through Magic Leap. The digital world in this case does not look like a film, but rather like sets of 3D entities, making it possible for users to establish a strong sense of presence in both the physical and digital world. Because of the innovative nature of adopted technology, it was decided to involve audiences who would be attracted into the museums by building novel kinds of experiences which were a hybrid of a game, film, theatre, and mixed reality. This would be immersive (referring to the objective level of sensory fidelity in virtual reality) and enable a sense of presence (referring to a user’s psychological response) (Slater 2003), producing a highly effective learning environment that could help audiences to imagine life at another time and in another space. In this respect, it was thought that the National History Museum experience could bring the past to life while the Science Museum experience could immerse them in a futuristic scenario featuring a world where hazard robots could be created to resolve several crises.
The target audience was identified by Factory 42 through a shared psychographic mindset rather than by a demographic segment. The name initially given to the target audience was The Open Gen, and their common characteristic was identified in their willingness to ‘give it a go, up for it’ mindset (Optimistic, Proactive, Engaging, Now). The closest museum segment to this group is formed by those defined as Entertainment Seekers, Engaged Community Drivers and Trend Awares, which are based on the Science Museum group audience segmentation and are based on the Morris Hargreaves Mcintyre (MHM) Culture Segments work and include a science engagement aspect (Factory 42 2019). As a broad set of characteristics, this audience likes to feel special, is engaged socially, culturally and possibly even politically, identifies value with shared and shareable moments, actively enjoys finding out about new products and experiences, especially in association to key influencers, and wants to define the personalised aspect of the experience.
To attract this audience, a franchise name was selected to comprise both experiences with the overarching aim being the creation of a new kind of entertainment that would be intriguing, fun and educational at the same time and address the fact that audiences nowadays tend to consume entertainment and even education in different ways. The name of the franchise, Dimension X, intended to invite the audience to step into ‘a new world’ where something ‘other worldly’ was happening. The consumer-facing materials intended to highlight the fact that audiences would become ‘dimensioneers’, finding their bearing within this new kind of experience by co-operating with family and friends as part of ‘an action’ that could be repeated across platforms and venues, either by progressing the narrative, like in a sequel, or by creating different versions of the experience, using distinctive content, for example by different museums. In this sense, the experience, aimed to be the pilot for a franchise that could then be expanded on and built on in subsequent years with other museums.
Early Research & Development
The original project packages included cross-team creative workshops, prototyping of concepts and testing with target audiences, followed by creative development, using themebooks whose different versions show the evolution of the project, to discuss ideas, software and physical design. The first objective was to develop, engineer and/procure the software, hardware and physical resources required to build the visitor experience needed for Beta testing with audiences. This in turn was meant to be followed by physical build and testing to integrate software, hardware and physical resources. Business models were meant to be developed throughout to maximise the commercial potential of the work.
The aim had been to conduct initial research in November 2019. This included obtaining ethics approval for audience research conducted by the Exeter research team (granted by the University of Exeter). Further tests with audiences were meant to be conducted in December and January 2020, with the plan to go live in the museums in the Spring of 2020, with an international rollout planned for 2021. Preliminary research conducted by Factory 42 before the start of the project showed that the new technology had inspired ‘wow’ movements; that at that point in time audiences had not quite experienced anything like Magic Leap before; that audiences were especially interested in seeing the overlay of physical and digital worlds; that the use of the technology would appeal to most ages, except very young children for whom Magic Leap is not suitable. Interestingly, films such a Jurassic Park and Minority Report were mentioned by audiences as possible points of reference.
The first project-wide workshops (see Figure 1) saw Almeida Theatre Director of Participation and project creative director Dani Parr involve the team in thinking about which kinds of robots or dinosaurs audiences might expect to see in Magic Leap; how groups would work together in mixed reality; how the chosen spaces within the museum would work in practice; what active learning could mean in this context; how the experiences could be accessible and facilitate a sense of exploration while also being semi-scaffolded and facilitate live orchestration on the day; how the story might be extended before and after the museum visit; and what the relationship between fact and fiction, imagination and museum learning might be. Discussions focussed on how the design of the experience would need to appeal to people from different age groups, cultures, and competences; work for families, friends, and individuals; have multi-sensory triggers; be accessible and scalable for touring; encourage repeat visits; have a wow factor and be scientifically accurate.
These workshops found that the team felt it was important to design collective experiences, and that audiences should be able to discover something, for example that different species of dinosaurs lived at different points in time in history. From a story point of view, roleplay was discussed as a strategy for discovery, for example of a new fossil, or time travel to explore different kinds of dinosaurs, raising ethical concerns as to how to communicate to audiences what was real and what wasn’t. Among the wow moments identified were the creation of very large dinosaurs, the possibility of seeing the world through the eyes of a dinosaur, the use of historic insects and plants and the possibility of creating a multi-sensory environment.
Subsequently to these workshops, the research focused on how to build the mixed reality experiences technically, creatively, and spatially, and understand what places they would occupy within the two museums both physically and curatorially. For the latter, the two museum audience research teams investigated how visitors would arrive at the two experiences, and what they would be encountering before and after each of the experiences. To carry out this element of the research, the Natural History Museum conducted several walk-throughs, testing how participants, who were observed by team members, moved within the museum, and what they engaged with and documented (see Figure 2).
At the same time, the creative team led by the Almeida ran a number of workshops looking into how audiences would behave in narrow spaces while wearing headsets (see Figure 3); how they might relate to and collaborate with each other; how easily they would understand and respond to instructions; how long they would be able to concentrate within the experience; and how they would cope with being asked to take on different roles, acting as onlookers, participants, gamers, learners, teams, or even researchers at different points in time. Concurrently, a series of intensive retreats were held, using storyboarding to workshop immersive theatrical approaches with an actor and a limited number of users in the studio. These tests, focussing on immersion, engagement, emotion, and learning, run in parallel with audience research into collection engagement led by the museums, and produced user feedback that informed the initial prototyping work at Factory 42 which, crucially for the museums, aimed to emphasise the delivery of scientific engagement and ensure the accuracy of the science drawn from.
Early prototype testing
A new iteration of the design of the pop up version of the experience was finalised in a theme book in late November-December 2019 and testing with around 100 people was completed at the Natural History Museum Jerwood Gallery providing crucial insights for subsequent versions, including the March 2020 launch at the Metrocentre, one of Europe’s largest malls based in Gateshead. The testing at the Jerwood Gallery, which was not for the main location-based entertainment (LBE) version but a pop up version that could tour while the LBE version remained located in the two museums, was done behind closed doors, but many members of the public became intrigued by the sign placed in front of the gallery and hovered outside trying to have a peak into the gallery and find out what the experience consisted of and when it would open to them (see Figure 4). The Jerwood Gallery experience took place within a black box type of set that was to be positioned inside the shopping centre experience. The first section of the set was built to host the on-boarding for each of the experiences and saw staff help audiences putting on the headsets and receive basic instructions about the experience. The set was then divided in two parts, one for each of the experiences. In the Dinosaur Experience, the core gameplay consisted of cleaning up a ghostly outbreak of dinosaurs by collecting their paleontological residue. In the Robots Experience, audiences built their custom robot which then appeared in front of them as a digital translucent blueprint model before being 3D printed and transformed into a solid photoreal machine. The robots were then used to navigate a city and either lift citizens in danger, repair collapsed buildings or spray water onto fires. In both cases scoreboards appeared at the end of the experience showing how the audience did in the experience.
Feedback from testing at the Jerwood Gallery indicated that the version had a lot of potential, but more attention needed to be paid to onboarding and the orchestration of the experience as audiences appeared to get confused about the overarching aim, context, purpose, and progression of what they had been asked to engage with. The testing also showed the complexity of the Magic Leap integration with the set built, especially in relation to lighting and materials used which could severely affect the Magic Leap ability to deliver the mixed reality. Feedback suggested some audiences struggled to fit the equipment on by themselves; were confused as to whether the experience was competitive or collaborative and whether they would receive a reward for achieving their mission and obtaining the highest scores. Feedback also indicated that troubleshooting had not been factored into the narrative, that people could not understand the relevance of the set in the experience and were not clear about what learning had occurred. Having said this, many enjoyed the prototype. Generally, audiences rated the experience highly, with 89.8% rating it at 4-5 (on a scale of 1 to 5) and 83% would do the experience again. Testing thus revealed that the experience needed to be rescoped and that more attention needed to be paid to onboarding (by including, for example, something to ‘entertain’ participants who were waiting to be onboarded with some initial snippets of the experience to heighten anticipation). Moreover, testing showed that the Magic Leap lost its calibration when moving from one room to another. Therefore, the decision was made to reduce each of the experiences to two rooms and rework the narrative in relation to the digital and software assets while also scoping a lighter mobile version to be used in festivals or shopping centres.
The shopping centre experience testing
Factory 42’s original proposal to the Audiences of the Future programme envisaged that a version of the mixed reality experiences designed for the two museums should also be experienced in shopping centres. This proposal raised interesting questions during the research and development phase of the project as to the design of the experience in a non-museum space, both in terms of its content and structure, and in terms of its layout and learning outcomes. Detailed attention was paid to Intu’s research into their own audiences, including as to how e.com, computer vision and AI are changing shopping experiences, and this, alongside recommendations by the two museums audience research teams, prompted decisions about the length of the experience, its content, structure, format, and pricing.
Intu is the largest shopping network in the UK with over 22 m2 ft of retail. The leisure space and the site put forward for the experience, the Intu Metrocentre near Newcastle, is one of the largest city centre shopping destinations in the UK with more than 370 shops, making it the second largest shopping centre in the UK. The shopping centre has an average dwell time of 1 hour and it is anticipated that the provision of an exciting mixed reality experience might prolong that while also offering a flavour of the longer and more complex London versions. Interestingly, while the setting up of museum experiences in a shopping centre, as well as in other non-museum spaces, such as airports, is a fairly common practice, especially in the USA and in Europe, they still rarely involve interactive and immersive technologies or performative events.
It was decided that for the Intu shopping centre, the two pop up versions of the Dinosaurs and Robots museum experiences, focussing on dinosaurs and robots respectively, would be built side by side, using the model tested at the Jerwood Gallery. This was largely to maximise marketing and visitor engagement within the confines brought on by this environment. Despite the fact this was designed separately from the other two museum experiences, and that it was meant to be a different version of it with not much narrative and no actors, it used some of the same digital assets and so it was considered crucial that it should align with both museums’ brands and that the content should be curated with a learning perspective in mind. This version, which could be described as a 10-minute game, showed that visitors craved a story. Moreover, while there had been some clear improvements and people really liked the T-Rex and the other dinosaurs, some people still struggled to find meaning in the overall experience, especially from a learning perspective, and it became clear that more work needed to go into the parallel development of the tech and the storyboarding so that the narrative could take better advantage of the technology. Hence it was found that if the shopping centre experience was redesigned, it should consist of a simpler version with a different narrative from the main LBE version.
Interestingly, as the Digital Director at the Science Museum Group, John Stack, pointed out, it had also become apparent that the management of the design of immersive experiences required a different approach from the one that had originally been envisioned. In fact, initially, this project, like other creative projects, had been planned in a linear way: narrative > concept design > technical design > build > commercial model > marketing > go live. But as all these components proved to be interrelated, they needed to be carried out in parallel. Not only, as different disciplines had been involved concurrently, more time than had originally planned needed to be devoted to considering differing practices and terminologies, but also the various creative dimensions of the project were moving at varying velocities and rhythms, in that not everything was simultaneously being progressed at the same speed. For example, at Factory 42 some aspects seemed to be developing rapidly for the immersive theatre team and slowly to the software development team. This finding is crucial in terms of the identification of an underpinning framework for the design of mixed reality theatrical experiences in that ways of prototyping and testing need to be adapted and so also scaled right across the development and production period.
Redesign due to COVID-19
The COVID-19 pandemic significantly affected this project, pausing the work for nearly a year during the lock-down periods. First, the enforced closures meant that the Intu experience was cut short, and the scheduled openings of the main experiences in the two museums had to be postponed. Revised scheduling in the museums during the pandemic meant that, upon reopening, the pre-planned exhibition spaces were no longer available. Due to a renewed exhibition programme and regulations regarding safe visiting, the museums were no longer able to commit to the new and still, at that stage, uncertain delivery dates of the project. Moreover, Magic Leap indicated that they would shift the development of the product to enterprise rather than the consumer, presenting additional challenges to the project and illustrating the problematics of developing content for immature technologies. However, new funding was made available which enabled Factory 42 to continue working on this project during the pandemic. This brought on two fundamental changes: the first led to Factory 42’s development of a mobile AR Route, resulting in the production of two augmented reality apps called ‘My Dino Mission AR’ and ‘My Robot Mission AR’ that launched in the Apple App Store and Google Play in 2020, and which this report will not cover. The second led to Factory 42’s production of a revised experience which opened for a short run in London at the Hoxton Docks in the Autumn of 2021.
It was also decided that the latter and final deliverable of the project was to be a single LBE called Lost Origin with a revised learning dimension that was not as closely tied to the Natural History Museum’s learning objectives. This single experience combined elements of Dinosaurs and Robots which were reimagined to match the chosen venue, Hoxton Docks, so that the story became about why dinosaur bones ended up in a warehouse. Assets for each of these experiences, including the volumetric audio and performance by Jodie Whittaker, dinosaur and fossil 3D models were built on, using aspects of the Robots Experience with an AI storyline as well as aspects of the Dinosaur experience, so that through a plot twist, audiences who thought they were involved in an experience resembling the former found to be immersed in a storyline more akin to the latter. When the venue was finally secured, the audience flow and size of sets in the rooms were refined, including several elements that were infused with environmental storytelling, so that props would respond to the audience and individual audience members would have a different experience according to how they engaged with the props on stage.
The final part of the development was dedicated to a new world building and consequent drafting of the world story bible and narrative treatment. These, building on previous iterations, were also devised and led by the Almeida Theatre Director of Participation and project creative director Dani Parr and co-written with Hannah Wood. The wider experience design team comprised Parr and Wood, as well as Michelle Feuerlicht, Ross Phillips, Mike Golembewski and other members of the set design and technology/interaction design teams, including also the Magic Leap experience design team. In their vision, the audience was meant to work actively and as a team by engaging with the story, its characters, puzzles, narrative mechanics, and, at the end, the mixed reality environment. The experience was to be narrative-driven, and include live performance, environmental storytelling, narrative mechanics, and elements of improvisation and audience interaction, where actors respond to audience personality but also around the use of the Magic Leap, especially in the Magic Leap onboarding area where audience members are fitted with the Magic Leap kit. Elements of the plot would be discernible through projections, 3-D miniatures, lighting and sound, and the Magic Leap itself, to create a fully immersive theatrical experience bringing together visual effects with credits spanning The Hobbit, Matrix, Harry Potter and Hunger Games. In other words, the story was embedded in all elements of production, including the set, so that the digital and physical dimensions worked together to create one world.
Bespoke multiuser VR tools, built in Unity for the Oculus and Photon platforms, made it possible for the team to develop spatial understandings of the proposed design concepts, and explore the impact of the physical environment on the narrative pacing and development even before the physical set was constructed through a 3D model that was built in Unity to enable all teams to see/experience how rooms would work. This was vital for the writers who needed to adapt the script to fit the interactions and timings and for the set designers for the placement of key features like the foliage and miniatures. This was also key to the iterative design process, and the emphasis on innovation and technology embedded in storytelling. Through developing and employing bespoke process-specific tools for the collaborative spatial pre-visualisation of in-progress work, the team were able to proactively address and mitigate some of the challenges of designing a highly site-specific project at Hoxton Docks in East London.
The story was described initially as a ‘supernatural thriller set in an authentic, realist world of dark web criminal networks, deep learning automation and clandestine operations infused with eerie paranormal mystery’ (Factory 42 2019). Subsequently described as a ‘supernatural mystery set in a dark web marketplace where a clandestine operation evolves into a magical adventure’, the story started when, the day before the experience, audiences received an in-world email and video briefing telling them about a suspected criminal activity linked to a deliveries business called origin. The email confirmed their callup to a secret task force called Wing 7 and the security clearance for Operation Origin, described as ‘an undercover mission to investigate an illegal dark web marketplace’. Through the briefing, the audience was made familiar with the task, to collect incriminating evidence on the person behind the criminal business. The live experience started when the audience, upon arriving at the Hoxton Docks venue, was snuck into a field base where Marsha Ingham, a computer engineer and cybercrime expert, played by a live actor, and Logan Flynn, a paramilitary logistics and surveillance specialist, also played by a live actor, introduced evidence that Origin was a fake front for an unregulated dark web marketplace called Emporium. This included a hacked call suggesting that Emporium’s founder, a mysterious figure called Haggledance, was about to go to ground to escape the authorities. The mission was set to break into Origin and collect evidence on Emporium and Haggledance’s real identity before he escaped (see Figure 5). An almost imperceptible interference in the video in which the Haggledance suspect appeared to be incriminating himself disclosed a ghostly apparition, suggesting that there may have been more at stake in Origin than met the eye.
After the briefing, the audience was guided to a reception where they were asked to solve a puzzle to access the Origin warehouse. In the security tunnel connecting the reception to the warehouse, they had to solve a new puzzle to bypass an AI-driven security system, part aided by an unexplained presence. By solving another puzzle, they proceeded to the next room where they uncovered several valuable artefacts and scientific objects being sold by Emporium and learnt that Josh Knox, the Origin warehouse CEO, is behind the username Haggledance. Here they could discern the supernatural presence of Gerel Orlowska, a Victorian palaeontologist whose ghost had been raised by the arrival of rare specimen at the warehouse, which she believed to be the one she had discovered in the 1880s. Gerel, interpreted by Jodie Whittaker, whose performance for this role had been captured volumetrically, though she used her own voice, then haunted the goods in the room so as to persuade the audience to proceed to the vault where they could hack into a safe and find her journal. This prompted the audience to move into a new space where they learnt about Gerel’s life and found out that she had taught herself palaeontology from childhood and over the course of multiple digs before finding what she believed to be a new dinosaur specimen. Because her discovery was ignored, she became a restless spirit until she found the bones up for sale in the warehouse. This room was literally formed by the giant pages of the journal (see Figure 6), which meant that the audience was inside the journal, surrounded by interactive fires, energy, and butterflies, who later became an older Gerel seen digging, immersed in the dust of history. Technically, the room was built on a foundation of cutting-edge depth sensing cameras (Intel RealSense), in combination with AI-based human segmentation and pose-detection software (NuiTrack AI) which were extended via the creation of a range of bespoke gesture-detection heuristics, grounded in an understanding of ergonomics and user-centred interactive design. At the end of this section of the experience, the audience was able to strengthen the supernatural energy in the building and bring to life Late Cretaceous era dinosaur ghosts which Gerel believed gave the audience the opportunity to identify her dinosaur bones.
Equipped with Magic Leap headsets, which were said to be para-forensic scanners, the audience then entered a new space called deep time at the Origin distribution room. Here, the cretaceous period had cracked through revealing a room covered in plants and a swamp. The room was populated with a range of dinosaurs including a T-Rex and the new dinosaur Gerel had discovered (see Figure 7). Technically, the set design integrated code within Magic Leap so that the headset could trigger associated physical actions, such as physical objects moving/reacting to the digital assets. By scanning the dinosaurs, audiences thus strengthened the dinosaurs and were able to confirm that the hatched dinosaur was a new species, proving Gerel right.
In the final room, the audience encountered Josh, played by a live actor. Here, the audience was caught in a clash of values between what Josh and Gerel wanted to do with the specimen, resulting in a conflict that raised the question of who has the right to own knowledge. The audience could then decide the ending, choosing whether Josh would be able to escape or not, witnessing the final moments of his freedom on CCTV and herewith also learning from the news that the dinosaur had been recognised as a new species and Gerel finally credited with its discovery.
Lost Origin proved to be a highly complex work, in which the interplay between the physical and digital in both the journal room and in deep time was very innovative and required co-ordination across all departments from set design, to show control, to technical, art development and live action. These where held together by Lost Origin’s executive producer Michelle Feuerlicht, a specialist in interactive narrative production and user experience. Lessons learnt throughout the development may be usedful for the design of future mixed reality theatrical experiences.
Towards the creation of a design framework for mixed reality theatrical experiences
One of the main findings was that lessons learnt throughout the development phase of the project could be useful for the design of future mixed reality theatrical experiences. The multi-disciplinary team who created Lost Origin came from different backgrounds and sectors, including theatre, films/gaming, and coding. The original intention of the project had been to adopt an iterative design practice that was based on the Almeida and Factory 42’s previous experience in developing theatrical and digital experiences and products. Traditionally the management of theatrical experiences follows a linear pattern (rehearsal to production). Subsequently, as theatre productions became more hybrid and used innovative technologies, more iterative models started to be introduced using steps that could be described as loosely based on design frameworks such as Bruce Archer’s 1963 development model which comprises observation (programming and data collection), evaluation (analysis, synthesis and development) and transmission (communication) (Archer 1965). Interestingly, Archer’s model was subsequently built on by Brian Lawson in 1990, who, still following a linear model, added analysis, synthesis, evaluation (The Design Council 2007 and Lawson 1997), recommending a process altogether closer to the iterative processes used nowadays in several production contexts including the original approach for Dinosaurs and Robots. However, Stuart Pugh put forward a design process of iteration, testing and evaluation which is both linear (market, specification, concept design, detail design, manufacture, sell) and circular (analysis, implementation, needs assessment, problem formulation, abstraction, and synthesis) (Pugh 1990). This process, largely due to the impact of COVID-19, was closer to the design approach finally used for Lost Origin, suggesting that for the creation of immersive experiences theatrical and design production models will tend to merge.
Lost Origin’s production process was led by Feuerlicht through a hybrid comprising aspects of agile development, which for her is key to success in any R&D project that aims for innovation. For her any work involving cutting edge technologies should approach the narrative and technical developments so that they allow the best possible use of the technology for the best possible audience-driven experience. To achieve this, she identified the ‘constraints’ pertaining to this project and then proceeded through an iterative development including prototyping and testing which, because of the Audiences of the Future brief, had the question of the audience experience and the identification of new types of audiences at heart. She also used a survey conducted by the team to identify the new audiences and remodelled concept, prototype and audience testing iterations on the basis of audience testing conducted throughout the concepting/development phases. The target audience spanned all age ranges, peaking between 35 and 65, including pioneers and early adopters. Testing at this stage revealed that the audience fund the concept multi-faceted, entertaining, and exciting.
The design approach finally used for Lost Origin builds on the Double Diamond model, which is formed by the four distinct phases Discover (market research, user research, managing information and design research), Define (project development, project management, project sign-off), Develop (multi-disciplinary working, visual management, development methods testing) and Deliver (final testing, approval, launch, evaluation and feedback) (The Design Council 2007), though it has a circular as well as linear development which allows for iterative testing and scaling, which, as Joris Weijdom has suggested, is crucial for the production of theatre in mixed reality (Weijdom 2017: 17). This approach enables designers to gain an understanding of their audience’s engagement throughout the experience. Thus, rather than designing and producing all content at the beginning, a selection of the content (and story) was produced as a rough-cut segment, which could then be experienced in a mock-up of the location to see if the interaction, subject matter and intended designs work together (Reid et al 2005: 7-8). Because of the constraints identified for Lost Origin, the Discover phase was limited while, in her words, the emphasis was on Define (test/iterate), Develop (test/iterate), and Deliver, though the Deliver phase did not in fact deliver a ‘traditional product’ but rather learnings, audience feedback and dissemination to the industry.
Several reports were written just before and during the Audience of the Future programme which influenced the writing of this report, and which had, like Lost Origin, the aim of providing toolkits and frameworks for the design of mixed reality experiences, whether in the theatre, museums, the games, or sport industries (Kidd and McAvoy 2019; Lessiter et al 2018; Ambasna-Jones et al 2018). For example, The UK Creative Immersive Landscape 2020: Business Models in Transition report (2020a) authored by Aki Järvinen published in 2020 included an Immersive Production Matrix dividing the field in distribution, technology and user experience. According to the author, the matrix can be used to look at any immersive production and break down its elements according to these categories. Interestingly, under distribution the matrix indicated several platforms which may be used concurrently or sequentially, also listing several technologies and related user experiences. Järvinen’s second report, The Immersive Audience Journey report, also published in 2020, divided the audience journey into five phases: awareness, consideration, decision, immersion, and loyalty (2020b). In analysing the audience’ journey through immersive experiences, Järvinen suggested that the audience should not be considered as a single cohort as traditional forms of marketing do not communicate the nature of the experience, pointing out also that creators need to invest in the customer journey to assess the audience’s willingness to engage and then re-engage with an experience, thus extending the impacts of the experience ‘beyond the immediate boundaries of the experience itself’ (Ibid.). In terms of design, the principal area of recommendation of his second report highlighted the importance of user testing with a wide range of target audiences not only from an audience experience point of view but also from a stakeholder responsibility point of view and not only at the end but, again, throughout the creation of the experience. The report also noted the value of collecting the long-term impact of a work, which could be useful in terms of building up an appetite for a subsequent version of the experience or new work by the same creators. The final recommendation had to do with the use of merchandising which plays a role in sustaining interest in an experience and creating a brand for its creators.
Both Järvinen’s reports found that audiences are increasingly active and often adopt a range of roles within an experience, something that confirms findings from previous studies such as Steve Benford and Gabriella Giannachi’s 2011 Performing Mixed Reality which showed that audiences can act as participants, onlookers, performers, re-enactors within the same piece and Catherine Allen and Dan Tucker’s 2018 Immersive Content Formats for Future Audiences, which noted that future audiences might be engaged as explorers (of another time or space), perspective shifters (entering another person’s body or experiencing a slice of someone else’s life even if only empathetically), gamers or seekers in a treasure hunt, interactors within data simulators, creators (through the use of immersive media toolkits such as Tilt Brush which lets you paint in 3D) etc. These studies are crucial in that they show that audiences are not only more active in these works, they also are asked to adopt different and even multiple roles, which makes testing even more important to ensure audiences are prepared to engage with all elements of the experience.
In Lost Origin, Feuerlicht introduced several kinds of testing aimed at identifying interest and understanding in the narrative and experience journey; identifying interest in the Magic Leap technology and the possibilities offered by immersion and presence; and testing the varying levels of interactivity and appetite to engage. As is often the case with designing in immersive contexts, for Feuerlicht the trick was to ensure ‘the audience wasn’t alienated by complex tasks, and that they retained full immersion’, maintaining a sense of presence as well as immersion throughout the work. This was especially complex in that audience members might have arrived with different expectations, with some, Feuerlicht noted, looking for an escape room and others wanting a more complex narrative-driven experience. Because it was decided to use the Magic Leap in one room only, so as to avoid problems with calibration, the interactivity was introduced gradually, building up throughout the experience.
The aesthetic forms (theatre and, broadly, intermedia, digital performance) that influenced the design of this project fall within the broad field that Richard Schechner famously defined as environmental theatre in his seminal 1968 text ‘6 Axioms for Environmental Theatre’ (Schechner 1968). Here, Schechner described environmental theatre in relation to a wide spectrum or continuum of theatrical events spanning from public events to traditional theatre, including intermedia and environmental theatre. For Schechner these were characterised by transactions, among performers, members of the audience and between performers and members of the audience, as well as transactions among production element and between these and the performers and the audience. For Schechner, in environmental theatre all space was used for both the audience and the performance. This space could be found or transformed with a focus that could be flexible and variable. He also noted that in environmental theatre all elements of production spoke their own language and pointed out that the text was not the fulcrum of the work, which disassociated it from drama, and other forms of narrative-based performance.
Building on Schechner’s axioms, as well as my previous research with Steve Benford on the design of mixed reality experiences, I propose a design framework using not six, but seven axioms for the design of mixed reality theatrical experiences which encompasses the forms described in Paul Milgram and Furnio Kishino’s continuum and so also includes elements of environmental theatre. By looking at Schechner’s axioms for environmental theatre and my axioms for mixed reality theatrical experiences next to each other it is possible to see how deeply theatre underpins the development of the immersive experience sector. But whereas Schechner’s axioms were in part dramaturgical and in part performance-oriented, the axioms I discuss here are oriented towards experience design, which in turn builds on the field of Human Computer Interaction (HCI) and can be viewed as an extension of interaction design, including experiences which, like Lost Origin, are grounded in a combination of a theatrical setting (however minimal), the physical world, and a digital world, and in which the hybridity of the environment becomes an intrinsic part of the experience. As the environment includes the people who are placed in it, be it performers, audiences, or passers-by, the interactions among those involved in the experience are a fundamental part of the experience design, which is why repeated testing with audiences is so important at all stages of the work.
Steve Benford and I have noted in Performing Mixed Reality how the audience’s experience arc, or, to use Alan Brown and Rebecca Ratzkin’s successful expression ‘arc of engagement’ (2011), is formed by the use of trajectories which often precede and follow the main experience and that may the audience’s presence (or co-presence) within the work. These define the difference between designed paths, experience paths and replay paths, each of which have their own undisputed value within an experience. In fact, documentation (and so replay) can play a significant role in this context not only to create a legacy of an experience, which could be used for preservation, but also to prompt people to revisit an experience to improve on their previous involvement in it or, in the case of very subjective experiences, simply re-encounter the story arc from a different perspective. It is therefore crucial for the design of mixed reality experience that this documentation or archival feature is built within the experience, so as to encourage repeat visiting or episode building and therefore augment the life of the work.
Below is a table with Schechner’s Environmental Theatre axioms (Schechner 1968) on the left and the Mixed Reality Design Framework for Theatrical Experiences on the right.
|1) ‘The theatrical event is a set of related transactions’.||1) Audiences are active, co-creative, taking on multiple roles through the experience which are based on transactions.|
|2) ‘All the space is used for performance; all the space is used for audience’.||2) Sets span digital and physical environments and can include viewing platforms and multisensory environments.|
|3) ‘The theatrical event can take place either in a totally transformed space or in “found space”’.||3) The event is convergent and pervasive, in that what happens before and after the event on multiple platforms may form part of the experience.|
|4) ‘Focus is flexible and variable’.||4) The production of the technology, the environment and the story must evolve concurrently, iteratively, preferably sustainably, and in relation to each other.|
|5) ‘All production elements speak in their own language’.||5) Digital dramaturgs and orchestrators provide guidance and continuity among different elements of production.|
|6) ‘The text need be neither the starting point not the goal of a production. There may be no text at all’.||6) Narratives are semi-scaffolded, combining linear (plot based), immersive (gameful and task based), documentation (replay) and improvisatory (free flow) elements using multiple platforms.|
|7) The analysis of presence and co-presence is a key design factor when it comes to building the audience experience and understanding audience behaviour (whether as spectators, participants, performers, gamers, etc.).|
To conclude, I suggest that these axioms might be understood as the stepping-stones of a framework for mixed reality experiences. The first axiom focuses on audiences, but not on audiences per se, rather on what activities, creativity, and roles, in other words, what transactions, they are prepared to take on during the experience. Audiences do not exist in isolation – they are defined in relation to what they do – they are the performers, spectators, participants, interactors, gamers. Complexity here is key: the storyline should neither be too simple nor too complex. The use of space in mixed reality performance is hybrid, spanning both physical (or to use Schechner’s term, environmental) and digital elements which make it possible for audiences to act in a wide range of more or less hybrid spaces. Too much of a set, and the experience becomes more theatrical, too little, and the theatrical dimension may be lost. The performance event can be convergent and pervasive, which means that the audience’s involvement may start and end before and after the actual experience. Because of the exemplarity of the design of environments involving audience participation, the production of technological, physical and narrative components must occur concurrently, iteratively but also, thinking of our future, sustainably and relationally. Rather than be developed in a linear way, these experiences ought to be developed through the use of a Double Diamond equivalent model considering circular iteration (re-use, re-play). Complex mixed reality theatrical experiences are likely to involve live actors as well as virtual actors and digital dramaturgs and orchestrators who would provide guidance and continuity among different elements of production during the design phase (by testing) and by interacting with audiences during the mixed reality experience. Audiences may need to be guided through these changes in character during the experience. To facilitate audience interaction, narratives will most probably be semi-scaffolded, combining linear (plot based), immersive (gameful and task based), including documentation (replay) and improvisatory (free flow) elements on multiple platforms. Iterative testing with a range of audiences around the scale of the scaffolding will be key to the success of the final experience. Finally, the analysis of the audience’s presence and co-presence will a key factor for the interpretation of the audience experience for it is when the audience feels present (and not only when it feels immersed) that the most rewarding and memorable experiences occur.
These key design findings, combined with the finding that the various creative dimensions of a project can move at varying velocities and rhythms, which means that any underpinning framework for the design of mixed reality theatrical experiences needs to identify from the outset ways of prototyping and testing that can be adapted and scaled right across the development and production period, are most probably just the initial step-stones of a new dramaturgy for mixed reality theatrical experiences.
This report has been informed by conversations with and feedback by John Stack, David Patten, Alex Burch, Dani Parr, Hannah Wood, John Cassy, Michelle Feuerlicht, and Joanne Evans. A shorter version of the report was presented in a keynote address at the Association for the Studies of Theatre and Performance (EASTAP), 23–4 September 2021, Vilnius University, Lithuania. We are grateful to Innovate UK which funded this project as part of the Audiences of the Future programme (TSB 18682).
The author was part of the research team that worked on the Audience of the Future project Dinosaurs and Robots, funded by UKRI.
Gabriella Giannachi is Professor in Performance and New Media at the University of Exeter, UK. She has published a several books including: Virtual Theatres (2004); The Politics of New Media Theatre (2007); Performing Presence, co-authored with Nick Kaye (2011); Performing Mixed Reality, co-authored with Steve Benford (2011); Archaeologies of Presence, co-edited with Michael Shanks and Nick Kaye (2012); Archive Everything (2016 and, in Italian translation, 2021); Histories of Performance Documentation, co-edited with Jonah Westerman (2017); Moving Spaces: Enacting Dance, Performance and the Digital in the Museum, co-edited with Susanne Franco (2021); Documentation as Art, co-edited with Annet Dekker (2022) and Technologies of the Self-Portrait (2022 and in Italian translation, 2023). Her research has been funded by the RCUK, the NLHF, Nesta, UKRI, and the EU.
Ambasna-Jones, Marc, Turnbull, Mitch, and Trewhella, Ben 2018 Creative Tools and Workflows for Immersive Content Creation, a report produced by Opposable Group and TechSpark for Digital Catapult, https://www.immerseuk.org/wp-content/uploads/2018/07/Creative_Tools_and_Workflows_for_Immersive_Content_Creation_.pdf.
Archer, Bruce 1965 Systematic Method for Designers, London: The Design Council.
Bekele, Mafkereseb Kassahun, Pierdicca, Roberto, Frontoni, Emanuele, Malinverni, Eva Savina, and Gain, James 2018 ‘A Survey of Augmented, Virtual and Mixed Reality for Cultural Heritage’, ACM Journal on Computing and Cultural Heritage, 11(2), 7.1–7.36. DOI: http://doi.org/10.1145/3145534
Benford, Steve, and Giannachi, Gabriella 2011 Performing Mixed Reality, Cambridge, Mass.: The MIT Press.
Brown, Alan, and Ratzkin, Rebecca 2011 Making Sense of Audience Engagement, https://galachoruses.org/sites/default/files/Barun-Making_Sense_of_Audience_Engagement.pdf
Factory 42 December 2019 Ghostbones* A Dimension X Experience, internal workbook, based on https://mhminsight.com/culture-segments.
Giannachi, Gabriella 2021 ‘Into the Space of the Digital Museum’, in Franco, S. and Giannachi, G. Moving Spaces: Enacting Dance, Performance and the Digital in the Museum, Venice: Edizioni Ca’ Foscari, pp. 118–131. DOI: http://doi.org/10.30687/978-88-6969-534-6/012
Järvinen, Aki 2020a The UK Creative Immersive Landscape 2020: Business Models in Transition, https://www.digicatapult.org.uk/news-and-insights/publication/the-uk-creative-immersive-landscape-2020-business-models-in-transition
Järvinen, Aki 2020b The Immersive Audience Journey, https://audienceofthefuture.live/wp-content/uploads/2020/07/Audience-of-the-Future_The-Immersive-Journey-Report_July-2020.pdf
Kidd, Jenny, and McAvoy, Nieto 2019 ‘Immersive Experiences in Museums, Galleries and Heritage Sites: A Review of Research Findings and Issues’, https://pec.ac.uk/assets/publications/PEC-Discussion-Paper-Immersive-experiences-Cardiff-University-November-2019.pdf, n.p.
Lawson, Bryan 1997 How Designers Think: the Design Process Demistified, London: The Design Council.
Lessiter, Jane, Mitchell, Sam, Ferrari, Eva, Borden, Polly, Bakhshi, Hasan, and Freeman, Jonathan 2018 Evaluating Immersive User Experience and Audience Impact, a report produced by Nesta and i2 Media Research for Digital Catapult, June 2018, https://www.digicatapult.org.uk/news-and-insights/publication/audience-immersive-report
Machon, J. (2018) The Punchdrunk Encyclopaedia, London: Routledge.
Milgram, Paul, and Kishino, Fumio 1994 ‘A Taxonomy of Mixed Reality Visual Displays’, IEICE Transactions on Information Systems, E77-D12, 449–455.
Pugh, Stuart 1990 Total Design: Integrated Methods for Successful Product Engineering, Wokingham: Addison-Wesley.
Reid, Josephine, Cater, Kirsten, Felurito, Constance, and Hull, Richard 2005 ‘Experience Design Guidelines for Creating Situated Mediascapes’, https://www.google.com/search?client=safari&rls=en&q=Experience+Design+Guidelines+for+Creating+Situated+Mediascapes&ie=UTF-8&oe=UTF-8, 10/7/2020, pp. 7–8.
Slater, Mel 2003 ‘A Note on Presence Terminology’, Presence Connect, 3, 1–5.
Schechner, Richard 1968 ‘6 Axioms for Environmental Theatre’, The Drama Review, Spring 1968, 12(3), 41–64. DOI: http://doi.org/10.2307/1144353
The Design Council 2007 Eleven Lessons: managing design in eleven global brands, https://www.designcouncil.org.uk/sites/default/files/asset/document/ElevenLessons_DeskResearchReport_0.pdf.
Weijdom, Joris 2017 Mixed Reality and the Theatre of the Future, Brussels: IETM.