Mixed Reality and the Theatre of the Future
This keynote address was delivered by Joris Weijdom, senior lecturer and researcher at HKU Utrecht University of the Arts / HKU Professorship Performative Processes at the IETM meeting in Amsterdam on April 16, 2016. The transcription has been edited for length. Watch the full lecture here.
Let me start by saying that I’m not a theatremaker. But I am honored to be here, and I’m very happy to be invited to do this lecture, but at the same time very anxious because I am not a professional in the sense that I make theatre myself. Actually, I am a mixed reality designer and researcher. I am also a lecturer at HKU University of the Arts Utrecht, where I teach master and bachelor courses embedded in education and research.
In the ‘90s I used to love the idea of virtual reality. I wanted to be somewhere else. I wanted to escape, to be in cyberspace, to do anything that I like. But I’m a bit older now and I sort of like the world—I like my body, I like nature. More and more I started to shift into this mixed idea, or augmented reality: to take reality as the substance and augment that into other kinds of experiences.
From 2012 to 2015, we were doing research on these kind of making processes at our Media and Performance Laboratory at the HKU University of the Arts Utrecht. What interests me the most is the process of making in an interdisciplinary team. I use technology in those processes, which can be a challenge, as technology can sometimes be counterintuitive and counter-creative—it can be linear and rigid. I will talk about mixed reality and how to use this technology in a creative process because I think it’s relevant and suitable for theatremakers and performers to try things out on the stage.
I want to talk about three developments: the Internet of things, transmedia, and mixed reality.
The Internet of Things
The Internet of Things is just an expression, but it stands for a development that has been going on already for a little while. You might know the idea of quantified self: having devices on our bodies that are measuring all sorts of things. This is very interesting for companies to understand our behavior, but in an open-source sense is also data that can be used by theatremakers in a more artistic context.
Around 2012, we shifted to a majority percentage of online traffic that is nonhuman. We’re seeing areas where machines take over.
Around 2012, we shifted to a majority percentage of online traffic that is nonhuman. We’re seeing areas where machines take over. You have to understand that the Internet of Things is not mostly about people, it’s about devices talking to each other. This will continue to skyrocket. Big data will explode. Of course everybody is very interested in what to do with this. Maybe it’s also interesting to realize that it’s not just about devices on our bodies; our whole world is being rigged and wired with sensors. They call these the “smart cities.” I put “smart” here between brackets because being smart means more than just collecting a lot of data, of course. But now that we have the Internet, all of this information is connected.
But the thing is, the Internet of Things and smart cities don’t seem to have a “human perspective.” Thankfully open-source development is trying to keep up and also create all sorts of networks that you, as artists, can make use of and start using that data for your own ideas and artistic and creative ideas—liberating this particular material into an open domain. And actually Amsterdam is very active in this with the Waag Society working at the Smart Citizens Lab. And then, of course, the smart city becomes about the “Smart Citizens.”
What does this have to do with theatre? In the ‘90s, this technology was being integrated into performance by one of the dinosaurs of performance art and technology, Stellarc. What you see here is Stellarc performing with a robot and robot arm. His heart rate is measured and other bodily functions are measured. This data is being used with all sorts of output. Things go in and things go out of the system. Interestingly enough, Stellarc also had electrodes on his muscles—mostly on his legs—and people on the Internet this is 1996, so early adopters—they could press buttons and give him electric current through his legs. So part of his physical movement is what he is trying to do, and part of it is involuntary because the audience, though the Internet, is interfering with his performance.
Another example is the work of a local researcher who was part of our lab, Marloeke van der Vlugt. She has done a lot of experiments with sensors. She puts sensors on her own body and participants come and touch her body, which is about privacy, intimacy, and how stories unfold from the body. Another example of her work is a room with all the chairs rigged with sensors and on the basis of how active or passive you’re sitting, the theatre light would change. So if everybody is lazy, we don’t see anything. This is about audience participation, and it’s about creating an ecology, a space that connects audience, performers, and space itself.
We know of many experiments that are using public space—Rimini Protokoll, and Blast Theory—and of course you probably know many others, but what is interesting is that the audience not only participates, but they also inherently become performers. There are people who are in public space who, by virtue of being there, are now part of the performance, simultaneously as audience and performer.
Because of the Internet, space has become connected. So as we’re in this nice theatre space with the doors closed, the Internet and the Internet of Things can in fact connect this space with everything else out there.
The interesting thing is the whole theatre space is rigged in a very hierarchical way. But the Internet of Things is modular, so the idea is that everybody in the room can potentially become the operator or performer of a given system.
So what does this mean for theatre? Well, like I just said, performances, audiences, and things can be at multiple locations, at the same time, sharing a connected experience. Performers and audience can get “connected” with the output systems of spaces and spaces become “sensitive.” So that would mean that you’re now controlling the theatre lights, or my slideshow, or you’re interacting with my body if I’m rigged in whatever way. Performers, audiences, and spaces can influence and manipulate each other, and the input-output system. In the classic theatre—the technician over there is sitting in the dark and is not supposed to be seen. But they’re there and they’re everywhere. The interesting thing is the whole theatre space is rigged in a very hierarchical way. But the Internet of Things is modular, so the idea is that everybody in the room can potentially become the operator or performer of a given system.
For that reason, if cities become sensitive at generating all this data, isn’t it interesting that you can use that data for your own performance purposes? So the traffic and trends that are right in front of this room could be part of what’s happening right here.
What it’s not: it’s not the same story in a different medium. It’s not that we have the book and we have the game, et cetera. You could call that cross-media or multimedia. It’s also not the recording of theatre plays on websites so you can see them again—that’s not transmedia. This is transmedia:
The series Freshers, about a fraternity, started on broadcast television, and there was a growing public following. Then they got on Facebook, and they realized that audience members wanted to be in this fraternity, to be in that experience of being a student and partying and everything it involved. So they started doing these Facebook parties, and they created a mobile app where you could play games and get invited to live events. People got so invested; they started to host their own parties. And then it gets interesting, because the makers of the actual series and the transmedia product were thinking, “Wait, if everyone is going to start conducting all types of parties, we’re out of control because we cannot direct everything anymore.” But it’s interesting because, of course, this audience becomes more and more involved with the whole story. And then the writers started to get it—they went to these parties, they went to network meetings, and then back to the parties. Some of the audience members became so active that they began to play a role in the story, and the writers incorporated that back into the television series. So here you see an interesting loop of the story world into public space, where media and different media are being used as platforms to give the public entrance into these story worlds.
Whether you’re doing it in the theatre, or you’re using many audiences, transmedia is actually inviting audiences to become co-makers, co-creatives. This has a lot of consequences.
People from funding organizations have a little problem with transmedia projects. Why? They want to see the actual numbers. How many people did actually participate? If it’s interactive, great—but how many then?
Here we get the 1 - 9 - 90 rule. Most people—90 percent—are passive: they’re just watching, and they like the stories and they like to look on Facebook and they like to maybe look at the games and who is winning but they don’t participate. Nine percent like to participate some of the time. They sort of throw in something to see if there is a response, to get a reaction, but more often they’re laid back. And only 1 percent of audiences are actually participating. Here funders say transmedia is lost money. My answer to this is no. For the simple reason that 99 percent of the audience is following that 1 percent as part of their audience group. They’re like “it’s one of us interfering with this story world.” So this 1 percent become representatives of a much larger audience. It’s very important to keep this in mind when you’re working with transmedia.
That brings me a little bit back to participation in theatre. I’ve had several meetings with Annette Mees, who used to work at Coney in the UK, who have done a lot of participatory theatre pieces. I really like how she talks about about levels of participation. It’s not that you’re interactive or you’re not—that you go in it, or you just sit and you watch it. There are very many levels of participation that you need to design. For the Coney project A Small Town Anywhere, they got everyone involved in participating onstage, so actors and audience were blended together and they had to reinvent society as part of their theatre world.
So whether you’re doing it in the theatre, or you’re using many audiences, transmedia is actually inviting audiences to become co-makers, co-creatives. This has a lot of consequences.
For example, if you consider every medium a stage, what does that mean for directing those stages? Directing a social media stage is much different than directing actors on a physical stage. It’s important to design these levels of participation, and to be aware of this 1-9-90 rule, to really invest in the 1 percent in order to get the 99 percent of the people being interested. But there are consequences. The consequence of co-creation with an audience is, of course, personalized feedback. Because if I want to participate, but I feel the system is responding with general answers, I don’t feel like I’m being talked to. So how to personalize that experience? That’s a big challenge.
Another thing is that the story needs to adapt because my behavior and response will be different from others. It means that you get not only multilinear stories, but you get hybrid stories evolving over time, so you also have to adapt the basis of your story over time based on what’s going on. And then of course we have authorship issues. I know a lot of theatres and a lot of playwrights don’t like these developments because they have to co-create and co-write, and that’s the same for directors.
Just to mention, there are a lot of technical platforms that facilitate these kinds of performances and these kinds of projects. Conducttr, for example, developed by Robert Pratten in the UK, is a platform where you can connect social media to the whole Internet of Things kind of thinking and mobile apps.
What is mixed reality? Well, it’s a scale. It’s not one thing or another. It’s a mix between the real and the virtual. What then is virtual? If you understand what virtual means—not real— mixed reality is the merging of real and virtual worlds and creating new environments, where physical and digital objects coexist and interact in real time. A mix of reality, augmented virtuality, and virtual reality. So if you make an experience, you can have this experience anywhere on this scale. And you can also move; you can have moments where it’s completely virtual, and moments where it’s completely real and everything in between, and over time, you can shift on that scale. I think I like scales because it’s not a black and white discussion. It’s something you can creatively move back and forth in.
You have probably heard of the Oculus Rift or HTC Vive. That’s sort of at the virtual end of the experience. In the ‘90s, when I went to school, I wanted this. And then it got quiet for twenty-five years for some reason, and now suddenly it’s happening. It’s interesting because if you compare these ‘90s photographs with the 2016 promise of the VR, it looks so similar.
For those of you from Holland, this is a 360-degree—huge—panorama painting hanging in The Hague by Mesdag. It’s a view of the beach in Scheveningen. People are standing in the middle, and they look around, and they experience a beach. It’s also a time capsule: a distilled time capsule of the 1880s.
This idea of immersion is of course much older. When I started my studies in the ‘90s, for example, we had Char Davies, already starting to make VR demos:
In this case, the breathing of the participant was navigating through some sort of virtual space. When you’re breathing, physically, a lot happens to your psychology. So people got really spaced out when they were in there for about fifty minutes. Which is what he wanted. And in the 1950s, there was the Sensorama by Morton Heilig, who was trying to make this totally immersive box. He said theatre is an activity that could encompass all of the senses in a very effective manner, immersing the visitor into the onstage activity. He called it experience theatre.
So what does this all mean? We don’t know yet. The whole film industry is moving into what they call VR. I don’t call it VR—it’s just 360 movies. There is a lot of development going on in post-production pipelines to put big movies in 360. When the first film camera was invented, the first thing recorded was theatre. The moment one person picked up the camera and started to move it, something changed. Filmmakers started to make close-ups. You could cut the film, playing with time. And so movies started. Then it wasn’t theatre anymore. And now I see moviemakers moving into the VR industry thinking it’s about movies as well. I will show you an example of theatremakers, and game designers working inside this new platform. I would say it’s a new platform, so let’s not be too fast in thinking it’s just another screen.
Have a look at this making of the video example from the Mill about a VR film called Help by Justin Lin as part of the Google Spotlight Stories project by Google ATAP:
Moviemakers think in a frame and direct in a frame; that’s how you “frame” your story. With a 360 helmet on, there’s no frame because I can decide what I look at and when I look at it. The example I just showed you is just action. There’s the monster, follow the monster, and they think that’s directing. I, immediately, am very tempted to not look at the monster and see what’s happening elsewhere.
So why use 360? I think this is useful for journalism. A journalist can frame three people burning a flag over here while other people are shopping and having their daily lives over there. With a 360 camera, the user gets a fuller picture. For journalism, this is an interesting development.
I think this VR demo from 2014 called Sightline the Chair is still one of the best I’ve seen when it comes to the promise of a new platform. Why? When I look at you, for example, I see the audience in a theatre; then I look away, and when I look back I see a forest. The interaction of the line of sight is a trigger for change but they put the change in image everywhere else, everywhere you are not looking in that moment. It is exploring the potential of VR and real time 3D.
We have been working in our lab with these things in a very low tech way. We put little streaming cameras with a parabolic mirror—we stream that through an output mirror where you see this red thing moving. That is actually the field of view of the person that is wearing the helmet. And then we experimented—how can we direct the attention of the person? How can we change the scene when they are not looking? What would happen when we move the camera around? This idea of how to always account for the user perspective is one of the creative challenges we are facing.
Performance and Presence
Performance and presence are probably terms you’re very familiar with. When technologists talk about performance, what they actually mean is if the technology actually does what they hoped it would do. I would say theatremakers would rather have successful performances where things start happening that they didn’t know of yet, so it is a different approach to the idea of performance. According to “Research on Presence in Virtual Reality: A Survey” by Martin J. Schuemie, presence is translated into something like this by technologists: “presence is a shortening of telepresence, which is a psychological state of subjective perception in which all of the individuals’ perception fails to accurately acknowledge the role of the technology in the experience.” If we make this system so perfect that you don’t see the system anymore, you’re totally somewhere else; we can transport you with our system somewhere else. Now, as theatremakers, you have to say, “We know better.”
You have a better idea of what presence means—and it’s definitely not about mediating or tricking the mind—it’s about being here, together. And theatre is live. So how do we translate this quality through technology?
For example, let’s have a look at a project called La Peri by Innerspace. We have dancers here that are being captured with a motion capture studio. So the dance is then put into the computer. The dancer is running around and there are a lot of cameras looking at her. Her movements are translated into this virtual 3D character. But this is called motion capture. The question is: is the presence of the dancer also captured? In the VR version you will have a helmet on so you’re sort of in the same virtual space as the dancer. You have two controls so you can make some visuals yourself. The idea of this is that you can dance with her. I have spoken to a few people who have been in this demo and they have said “I wasn’t quite sure what was going on, because she is just doing her dance—she wasn’t responding to what I was doing.” Well, there you go. In the best case there is some form of presence residue. There was definitely no interaction and there was no felt presence. To be fair, I think they are doing necessary groundbreaking work, I just want to point out the potential loss of presence as a quality of live performance when using these kind of technologies.
You can go there, put a VR helmet on, and you’re physically welcomed by this character who is also in the VR movie. Live and not live, real and not real is continually swapped.
Another thing to say is we are not going to translate everything into a virtual space, but we are going to combine the physical and the virtual. This is an example of the physical VR installation called VPRO Peepshow. You can go there, put a VR helmet on, and you’re physically welcomed by this character who is also in the VR movie. Live and not live, real and not real is continually swapped. And they are constantly playing with questions like what you think is happening and where you are. And the fact that she is there, physically present, around you, makes a huge difference. You just met her; you just shook her hand. So that changes the perception of the moment when you are in this 360 movie.
Immersive Virtual Reality
And this brings me to the wonderful piece by CREW from 2011 called Terra Nova. In the old days, way before the current VR helmets like Oculus Rift and the HTC Vive, they were already working with similar VR and live 360 streaming video technologies. They developed this technology themselves. What I find very interesting about their work is not only the fact that the audience is participating and observing each-other, but also that each audience member has a dedicated helper; a person who helps you through the experience. They are doing physiological manipulations with you—they place your body in a certain angle, they touch your arm when the movie is pretending to touch your arm—there is constant dialogue between the physical and the virtual. So although what you see is 360 pre-recorded video when these moments of live physical manipulation are synchronized with the VR movie it works really well.
We’ve been working at our lab with theatremaker Madeleine Matzer from Matzer Theatre Productions. She wanted to create a story about the inner dialogue of angels and demons based on the paintings by Hieronymus Bosch. We tried to create an installation that was partly theatrical whereby the person who would be inside the VR experience would also be placed in a theatrical setting that could be watched by another audience. What we created is a space with a lot of projection mapped environments where we would sometimes project what the person was seeing inside the VR experience and sometimes we would project completely different stuff so the people outside would experience a different story. Then of course, the person who is inside can become a chapter for the person who is outside.
Now the director, who had an acknowledged professional background in more traditional text-based theatre, was very brave in trying to figure this out. She made an interesting choice. She decided in the making process to sit in the middle of it, with the VR helmet on, which determined her perspective on the story. At some point during the lab-days she didn’t know how to proceed, she was literally lost in the process and in all these worlds and perspectives. Which changed immediately with the help of her dramaturg. She put the dramaturg in the middle with the VR helmet on enabling herself to take some distance as a director. This helped her to make necessary creative decisions. In designing these multiple perspectives for audiences, she as a maker also had to take on those multiple perspectives herself.
And now this promo video about THE VOID. It’s very American.
This is where the commercial industry is heading, right? This is called THE VOID and it is an entertainment center, a physical space in which you can walk around wirelessly with those VR helmets on and, of course, machine guns. They show in the VR helmet fantastic imagery that fits the physical space in which you are running around in real-time. So basically they are incorporating the whole physical experience in this virtual environment. What they are saying is that 300 of these parks will be built worldwide, a majority in China, but as far as I understand two in Holland.
Audiences are dying for good stories and meaningful experiences. The industry is currently pumping millions and millions of dollars into these technologies, and they don’t know what to do with it.
What they are creating is hardware, a new podium, in fact. But of course if you let Hollywood create the experience there are going to be monsters and dragons and robots, because they go for shallow action packed effect driven experiences. The thing is, I’m here in a room with theatremakers who try to tell much more layered stories, right? So I would ask you, what would you do with this?
If you think, and I can imagine you might, “Whoa, that’s way too high-tech and it’s probably going to cost a lot and be very difficult and counterproductive to my creative process.” Well, yeah. That’s partly true. But I would also say audiences are dying for good stories and meaningful experiences. The industry is currently pumping millions and millions of dollars into these technologies, and they don’t know what to do with it.
Have a look at this experiment we did at our lab with a similar approach as THE VOID. In the center you see a virtual space. I stole it from Unity, which is a 3D engine. We didn’t make that—Unity did.
The moment when the person with the VR helmet on and the “inside perspective” walks over to the virtual box and picks it up, he can physically feel it. And on the left there is another person, who has an “outside perspective,” who has a game controller that can operate the virtual robot arm you see in the middle. They have to work together. The person with the VR helmet on has to walk with the box to another part of the room and give the box to the robot arm remotely controlled by the other person. In the physical space to the right you see my colleague taking the physical box just at the right moment creating the illusion that the robot arm has actually picked up the box. I can tell you, the first moment where I felt the physical feedback with the virtual world, my brain said, “Okay, maybe this is real,” which is not normally happening all the time, but there is something to this tactility that is very interesting.
One big problem. If you have a virtual space that is huge, what do you end up with? With needing a huge physical space. If someone needs to run down a hallway of fifty meters, you need a fifty meter hallway in the real world. To solve this issue there are a lot of interesting contraptions being made to keep people in one physical place—we have for example all sorts of treadmills—so we all become hamsters soon.
THE VOID is using another solution called “redirective walking.” By showing a straight hallway in VR the brain can be tricked in thinking it is walking in a straight line while in fact the body is actually walking in a curved hallway. This enabled the makers of THE VOID to have an endless virtual space in a relatively limited physical space by literally letting the people walk circles. These ideas of combining the physical and virtual space are interesting. Because as you can see, all these examples are showing these very embodied haptic interactions with these virtual elements in a physical space, which is where we reside—our interface is our body.
Now let’s move on to the other end of the scale of mixed reality: augmented reality. You might know of the failed Google Glass project or the more recent development of Microsoft Hololens or the mysterious promise of Magic Leap. I like to begin with a warning. Do you want to be in a kitchen like this?
I think there are a lot companies that think this is really cool. This is the Internet of Things combined with social media, combined with augmented reality, right? But what is actually happening here? Well, it is just the environment telling you how to make a cup of tea. So again, augmented reality is not just about adding layers of information. And even big companies are now aware that storytelling is probably the right way to go about a lot of things.
So Julian Oliver’s work is interesting to look at. In 2008 he made binoculars that could actually recognize commercial billboards in a project called The Artvertiser. He worked with local artists so that when someone looked through the binoculars they would see all the billboards replaced with local art. He was reclaiming the public space as a gallery for the local artists by using commercials as tags. Another example is the work of a Dutch artist called Sander Veenhof who as an augmented reality specialist is reclaiming public spaces in other ways. For example he placed a virtual art piece at the GPS location of the US White House and told people to go there to see it. He could put his virtual art wherever he likes.
We can reclaim all sorts of spaces—both political and public spaces—by augmenting reality and asking questions. So, what does this mean for theatre? Well, theatre is mixed reality, right? And you have gone through post-modern times. And you have thought critically about it and there is this language for it called dramaturgy. And you are so incredibly aware of what you are doing that you are way ahead of other industries—so I think there are a lot of makers from other media and other disciplines that are dying for this kind of knowledge.
Well, theatre is mixed reality, right? And you have thought critically about it and there is this language for it called dramaturgy, and you are so incredibly aware of what you are doing that you are way ahead of other industries—so I think there are a lot of makers from other media and other disciplines that are dying for this kind of knowledge.
Is There Any Future For Theatre?
I think we need to ask questions about what we want to teach our young talents. Because of what we are designing— from the perspectives of dramaturgy and scenography—more and more we need to take into account multiple audience perspectives. It is not just one room with a frontal view. Right? So how are dramaturges and scenographers trained to design and mix physical and virtual spaces? If there are performances happening in many spaces, how is an actor trained to be present in multiple spaces? And of course there is the whole physiological aspect, the experience of reality is very much embedded into the interface of our bodies, how can we use that as a creative tool? Manipulate it. How do we use the physiology of perception in our designs and for our designs? Maybe we need to work with neurologists and psychologists.
Theatremakers understand mixed reality experiences as no other. And especially because we have gone through post-modern thinking we are perfectly capable of dealing with the simultaneousness of the play, as a virtual dimension of the actual experience, and the physical reality of the actual physical presence in a theatre building. This is what all the other industries find very problematic because all the other industries try to create the perfect illusion, which means they really want to transfer a person to a certain space and that should be consistent. You should forget about the technology. You should forget almost your body or your body should be transferred elsewhere. Here in theatre we know all these multiple layers of experience can be there at the same time. Maybe not in your primary perception, you’re shifting between those awarenesses. You’re now looking at me, if I become silent long enough, you will become aware of each other. But it is all there at the same time. So we can be in multiple space and places at the same time. We understand how we can design this, and we are completely relaxed about this. We are not looking anymore for the perfect illusion. We are way beyond that.
Theatremakers understand mixed reality experiences as no other.
To summarize. In response to the question “Is there any future for theatre?,” If everything will be connected to everything, to an infinite number of things; If audiences become co-creator in games, transmedia, augmented reality games—all sorts of participatory theatre. If we can be at several places at the same time, not here or there but at the same time.
“All the world’s a stage, And all the men and women merely players.” This is not just a metaphor for our way of life and for how we evolve as human beings but this is literal advice for you theatremakers. Through all of these technologies the world is your stage. The whole world. Everything is your stage. And I think that you are the kind of storytellers that need to start telling stories on different stages as well. I would even go as far as to say, maybe you should try to not get all these technologies inside of your theatre, maybe you should bring your theatre knowledge to other places. That could actually mean that you’re not making theatre, but you’re bringing your knowledge to game makers and to virtual experience builders and to all sorts of new media people. And start telling stories in a different way. I think there is a lot of talk about “How do we get new audiences?” Maybe by not trying to get the audience into this room but to go to places where the audiences are having their experiences. And layer that so we aren’t stuck with these dragons and robots all the time.
Thank you very much.