“Any sufficiently advanced technology is indistinguishable from magic.”

I’ll hazard a guess that few of you (save the real geeks among you) know that this quotation comes from British science fiction writer Sir Arthur C. Clarke’s “Hazards of Prophecy: The Failure of Imagination,” in which he posits three universal truths of scientific prediction. Yet whether or not you’re acquainted with Clarke’s Third Law, I imagine that it resonates with a moment you’ve experienced in the theatre, when a virtuosic stage effect goes so far beyond the realm of your technical understanding that it induces in you a mystic reverence, a kind of theatrical sublime.

While I’ve had many such magic moments in the theatre, none have rivaled my first experience with augmented reality. So, as with an overeager child observing a particularly impressive hat-trick, I have become obsessed with its mechanics. In the course of my research, I’ve discovered only two fully-fledged productions that have made use of AR. The first, Everyman: The Ultimate Commodity 2.0, debuted at Toronto Fringe Festival in 2007. The second, the Builder Associations’ Elements of Oz, had its premiere at Montclair State University last year, with an impending run in New York City this December.

Interestingly enough, both productions (and indeed, my pitch for this article) premiered before the release of the now ubiquitous app Pokémon Go, which rocketed the once-niche tech into mass public consciousness. Whether or not this delightful game ultimately proves a historical blip, the lasting impact of Go should not be underestimated. It’s already altering the commercial landscape, with bigwigs like Microsoft and Google diverting their resources away from VR (virtual reality) towards AR development, firmly establishing it as “the go-to paradigm” for future gaming.

Yet the most profound implication of Go may ultimately be sociological. Back in 2014, AR researcher Mark A. Livingston noted the major limitation that, “despite multiple cycles of media and industry attention, there has never really been a widely-accepted killer application for AR.” Yet the staggering success of Go (The New Yorker noted that “to say it has spread like wildfire is to exaggerate the power of wildfires”) has not only infinitely amplified AR literacy, but equally given us extraordinary data on when, why, and how we like to engage with augmented content.

For those unfamiliar with the technology, let me briefly recapitulate the differences between AR and VR. In virtual reality, a user engages with a wholly simulated 360-degree environment. By contrast, AR technology imposes a layer of virtual content on the real environment, captured through a live camera feed. Both technologies generally necessitate a degree of physical action on the part of the user, whether with a hand-held device or head-mounted display like the Oculus Rift. In simple terms, VR invites you into a new world, where AR layers content on the world as you presently see it.

The most widely accepted definition by AR researcher Ronald T. Amuza states that AR must cohere to the following three stipulations:

1. It must combine virtual and real imagery
2. It must align/register computer graphics with objects or locations in the real 3D environment
3. It must do both of these in real time

It should be noted that none of these aspects have really been optimized. Today, AR remains plagued by innumerable glitches, quirks, and bugs. As such, AR research remains necessarily structured around a success-failure dichotomy. This makes perfect sense in fields where applications have clear end goals and can be measured via metrics like visual acuity and temporal synchronization. But these criteria don’t necessarily apply to artistic AR, and despite growing interest in its creative applications, there remains limited critical engagement around its use in theatrical environments.

How can we build successful AR specifically for live performance, where technical considerations are superseded by aesthetic and dramatic ones? What makes for integrated, dramaturgically rigorous augmented content, beyond mere novelty or spectacle? And how can theatre practitioners, in observing the current trajectory of AR, stride alongside this rapidly evolving technology? To interrogate these questions, I propose to examine the dramaturgical considerations around theatrical AR, using Everyman and Oz as illustrative examples.

NTU-IERC’s Everyman: The Ultimate Commodity 2.0 (2007) at the Toronto Fringe. Photo by William Russel Pensyl.

In 2011, Stephen Fernandez asked what distinguishes Technodrama, a term widely applied to any aggressively multimedial performance, from other theatrical genres. He concluded that, despite myriad variations, Technodrama consistently foregrounds the tension between the simulated and the live, thereby reinforcing its own liminality as a theatrical practice. Moreover, Fernandez argued that the aesthetics of technodrama remained largely inextricable from technoculture itself. We might then speak of theatrical AR not as a rift or apotheosis, but rather the latest phase in the historical continuum of Technodrama, for which new technology is not merely a facilitative tool but a foundational dramatic question.

This is certainly true of the pioneering AR production Everyman: The Ultimate Commodity 2.0, which evolved not out of an initial theatrical conceit, but rather as a vehicle for technological experimentation. Head designer and media artist William Russell Pensyl explained that Everyman 1.0 was initially conceived at the Interaction and Entertainment Centre (NTU-IERC) in Singapore in 2007. Pensyl and his team were exploring ways of enhancing live-tracking systems for mixed reality environments in an attempt to overcome traditional limitations with spatial markers and 3D image components (and if that’s making your head spin, rest assured, I’ll explain in more detail later on).

Working backwards from the technology, the team recruited playwright Daniel Jernigan to explore what kind of theatrical narratives might serve their technological pursuits. Jernigan proposed an adaptation of Singapore author Gopal Barathaman’s “The Ultimate Commodity,” a dystopic fiction wherein the addition of a seemingly utopian chemical to Singapore’s water supply produces a nasty side effect; members of the population perceive that they are physiologically merging, over time becoming indistinguishable from one another.

To realize the merging, the team designed virtual masks that would be superimposed over the actors’ faces in real time. Creating 3D-mapped projections of the actors’ faces, the team then employed state-of-the-art AR and lip-synching technology to create uncanny talking masks. This content was displayed on a screen side by side with the playing space, allowing the audience to perceive both live and simulated faces simultaneously. Just as it was ambiguous whether the population was really being physically altered or the perceived merging was the result of psychological confusion, so too did the audience have to grapple with the parity of both visual fields. This uncanny juxtaposition gave way to larger questions about state control and the increased homogeneity of Singaporean identity. The AR content was not merely an aesthetic superimposition, but a key dramaturgical device.

As with Everyman, the Builders Associations’ Elements of Oz evolved from an interrogative impetus, one very much in line with the Builders’ established oeuvre of pursuing an ambitious hypothesis through imaginative theatrical methodologies. For Elements, the Builders set out to create a live reconstruction of the 1939 film The Wizard of Oz, which captured their interest as a sort of “technical motherlode” that would allow them to explore the interplay of technology, performance and the mechanics of spectacle.

When I spoke to designer Larry Shea and artistic director Marianne Weems, they explained that their decision to create AR was part of a broader investment in creating interactive content for mobile devices. For Oz’s 2015 run, Shea and his team built an extraordinary cross-platform app which could be downloaded by audience members before the performance. Then, during Oz’s key moments of spectacle, viewers were encouraged to raise their phones and view AR content, which ranged from digital poppies to Glinda’s bubble to a swarm of flying monkeys over the proscenium.

From left, Sean Donovan, Moe Angelos, and Hannah Heller in The Builders Assocation’s Elements of Oz (2015). Photo by Richard Termine for the New York Times.

While the AR for both productions was produced with proximate software, they presented their augmented content through different devices, with radically different image resolutions. Part of this difference is historical; Everyman premiered in 2007, right at the cusp of the smartphone boom. Yet the dissimilarity raises interesting questions about the future of theatrical AR. After all, one of the perceived limitations of current AR is its inability to approach photorealism, with developers fixated on visual optimization. Yet unlike AR in other fields, designers for the theatre must ask whether aping “real” objects or environments is a desirable outcome. Of course, with a conceit like that of Everyman, increased visual acuity might well enhance the storytelling. Indeed, when I asked Pensyl what he might change were he to redesign the show today, he noted the potential for using multiple camera positions “to create a more intimate experience of sensitive dialogue,” as well as the addition of recent facial recognition technologies.

Yet for the Builders, who stress an organic process of work creation, the necessarily low-fi aesthetic of the mobile AR actually served as a creative invitation. “The monkeys were never gonna look real, the poppies were never gonna look real,” explained Shea. “A single monkey would crash everyone’s phone.” Instead, the Builders explained, “we ended up building poppies on stage that mirrored what the poppies ended up looking like in AR,” creating colorfully kitsch flowers from plastic crockery. This gesture not only served as an homage to the film’s aesthetic, but an extension of the production’s central interrogations. Just as The Wizard of Oz blurs the boundaries of reality and fantasy, Elements of Oz complicated the traditional dichotomy of liveness versus simulation by allowing the AR content to inversely inform the “real” staging elements.

This tension is part of what makes AR so exciting as a theatrical device. Altering the visual “reality” of theatrical settings by forcing them to accommodate simulated imagery, AR complicates ontological distinctions between real and virtual environments, presenting both with equanimity. As such, theatrical AR is an evocative tool for interrogating falsely naturalized dichotomies—not just the fraught binary between liveness and simulation, but equal binaries of gender, power, class—to the very border of what constitutes live theatre itself.

Another key consideration for AR designers is the spatial configurations of the given theatrical environment. Presently, most AR necessitates the use of a fiducial marker or image target—a 2D or 3D object identified through the live camera onto which simulated content can be spatially aligned. This presents unique challenges in traditional performance spaces. Pensyl’s solution was to employ bulky head pieces with LED markers, which Jernigan then integrated into the diegesis. By contrast, because the Oz app needed to function for multiple camera perspectives, Shea’s solution was the creation of an enormous fourteen-foot image target, which would be projected onto the back wall of the theatre. This equally served as useful social conditioning for the audience, signposting when AR content was about to appear.

Both these solutions spring from a similar question: how do you prove AR’s liveness? Pensyl decided to foreground both realities, while Shea and team* granted the audience agency over their visual field. Yet it is equally worth noting that these productions occurred in theatres with static audience members. However, if we are capable of physical agility with the technology, as the meteoric rise of Pokémon Go suggests, then perhaps the tech is equally transferable—or arguably better suited—to immersive theatrical genres, such as site-specific or promenade theatre. This is certainly a present fixation for other artists and game designers, with visual art apps like Lapse and games like Ingress mapping augmented content onto found architectural spaces.

In pitting two visual realities against one another simultaneously, another key consideration for theatrical AR is the question of dramatic focus. Shea argued that constant AR would have been unsustainable in Oz, as holding up their phones would have proved physically tiresome for the audience. Yet when I asked Pensyl whether having two constant visual fields proved exhausting, he explained that because disorientation was key to the narrative, “we did not see issues with the audience’s ability to maintain focus on two simultaneous loci of action.” This highlights the need for rigorous dramaturgy around the “when” and “why” of AR content, as a necessarily dualistic mode of theatrical storytelling.

With exponential innovations in synchronization, facial recognition and, perhaps most excitingly, new vibration-detecting imaging out of MIT, the future of AR is bright. You’ll recall that I began by quoting Clarke’s Third Law, in reference to the magic that is AR. But to tell the truth, I’m a little more attracted to Clarke’s less touted Second Law, that “the only way of discovering the limits of the possible is to venture a little way past them into the impossible.” For when we rest on the summit of a technological precipice, one whose jagged edges shift precipitously beneath us, then it is surely our role as artists to not only harness the innovations of the now, but also to point to the potential of a yet imaginary technodramatic future.


Design team for Elements of Oz:
Augmented Reality Design: John Cleaner

Interactive Design/Programming: Jesse Garrison
Augmented Reality/Network Consultants: Larry Shea, Kevan Loney