Modes of Media in Digital Performance

In the late 1980s and early 1990s, theatre and performance artists such as George Coates, the Wooster Group, Troika Ranch, Mark Reaney, and Laurie Anderson began to explore the use of emerging digital technologies to expand the possibilities of live performance. Those who were scenic designers, such as Reaney, who established i.e.VR at the University of Kansas, conceived of digital media primarily as a way to enhance theatrical illusion through virtual scenery. By contrast, performers and directors, like Anderson and the Wooster Group’s Elizabeth LeCompte, exploited technology’s ability to probe the nature and significance of live performance itself.

These days, digital media is widely accessible to theatre practitioners and has become commonplace, which has led to higher and more consistent standards for technical proficiency. At the same time, its proliferation runs the risk of dulling its experimental and subversive edge—rendering it just another tool to create work that conforms to pre-existing theatrical conventions and audience expectations. At this juncture in the history of performance, as the novelty of media-enhanced performance is fading and a narrowing set of conventions is consolidating around media design practices (much like what happened during the early development of cinema as a narrative artform), it is important to remind ourselves of the many radically different ways that artists can and have incorporated media into their performance work.

Over the past twenty-five years as a director, installation artist, and teacher of digital performance, I have developed and refined a taxonomy of roles that media plays in live performance. The current version of the taxonomy distinguishes six basic modes: scene, prop, actor, costume, audience, and mirror. While some performances consistently call upon media to serve just one of these functions, many use media in a multiplicity of ways, sometimes simultaneously. Distinguishing clearly among these modes can help artists to resist calcifying expectations and clichés, move out of their comfort zone, and identify new creative possibilities. The taxonomy has proven invaluable in my teaching, enabling my students to define the use of media with more clarity and precision both in their own work and that of others.

Media as scene

Media functions as scenery to the extent that it defines the environment within which the performer is acting. For example, the physical set for The Animals and Children Took to the Streets (2011), produced by 1927, consisted simply of three white screens with small cutout windows. Meticulous animations in an expressionistic, hand-drawn style filled the screens, depicting a series of constantly changing environments, including the exterior of a tenement house covered with crawling cockroaches, domestic interiors, street scenes, a police station, and a park.

When the media functions as virtual scenery, the performer interacts with it by navigating through it. The spectator will interpret a change in the image as a jump to a new location and will interpret a shift in the perspective within the image as an indication of movement within the scene. In this way, the performer’s virtual or fictional location is anchored to the media.

For example, when Peter Pan and the children fly to Neverland in Threesixty Entertainment’s Peter Pan 360 (2009), designed by William Dudley, the performers are suspended in front of an enormous, 360-degree 3D animation of London sweeping rapidly beneath and around them. This effect is reminiscent of the piece de resistance of Lincoln J. Carter’s 1898 production of Chattanooga, one of the first uses of filmed scenery in the United States, in which two actors fought aboard a stationary locomotive while the filmed scenery raced toward the audience behind them.

Distinguishing clearly among these modes can help artists to resist calcifying expectations and clichés, move out of their comfort zone, and identify new creative possibilities.

Media as prop

At several points in The Animals and Children Took to the Streets, a live performer made sweeping gestures, which led to animated puffs of dust appearing on the screen. At such moments, the media is not simply providing an environment within which the character is acting, but it constitutes a virtual prop with which the character interacts. Any object in a virtual scene will function as a prop as soon as a performer uses it. When the media functions as a prop, the performer/character is the source of agency, manipulating, controlling, or triggering changes. The media’s relationship to the performer is reactive and instrumental.

For example, in a vignette from Golan Levin and Zach Lieberman’s Messa Di Voce called “Pitchpaint” (2003), two performers draw on the screen with their voices; when they sing descending notes, the lines on the screen curl clockwise, and when they sing ascending notes, they curl counterclockwise. Electronic musicians have created a wide variety of digital instruments that allow them to trigger synthesized or sampled sounds and images interactively. An early, well-known example among electronic musicians is a custom-designed instrument called lady’s glove, first created in 1991, which Laetitia Sonami embedded with an array of sensors, allowing her to trigger various sonic events—tones, voice samples, and sometimes images—by moving her fingers. Such digital instruments blur the boundaries between composing and performing.

Media as actor

Media functions as an actor when the live performer relates to it as an autonomous agent—that is, as a subject in its own right with sentience and volition. The relationship between the live performer and the virtual actor is responsive and interactive. In 1914, vaudeville performer and animation pioneer Winsor McCay performed live with a projected animation of Gertie the Dinosaur. At one point, McCay scolds Gertie, who responds with tears. To make amends, McCay tosses an apple at the screen—in reality a red cardboard prop that he had clandestinely stashed into his coat pocket—and into Gertie’s mouth. More recently, in The Animals and Children Took to the Street, one of the primary characters is a girl who appears exclusively as an animation on screen and who interacts with various live actors throughout the play.

In both of these cases, the animated sequences are pre-rendered and the live performers must adapt their performances to it, creating the illusion that the media is responsive. In other cases, however, a mediatized actor might respond dynamically to a live performer in real-time. In my own production of The Tempest at the University of Georgia in 2000, Ariel was a computer animation controlled in real-time using motion capture (to my knowledge the first use of real-time motion capture in live theatre); the mo-cap performer was visible on stage but never—until the very end of the play—acknowledged by any other actor.

Another way a truly interactive dynamic can be created is telematically, when a remote performer on screen interacts in real-time with a live one on stage. For example, during each performance of the Builders Association’s Continuous City (2008), the actor Rizwan Mirza, playing a character named J.V., engaged in improvised video chats with members of his real family in London and Virginia.

Media need not be humanoid to function as a virtual actor. In 1997, Flavia Sparacino, Glorianna Davenport, and Sandy Pentland, a team of researchers from MIT, presented a piece called “Improvisational TheaterSpace” in which a live performer interacted with text projected on the stage. According to Sparacino, Davenport, and Pentland, “the text is just like another actor—able to understand and synchronize its performance to its human partner’s gestures, postures, tone of voice, and words.” In 1988, jazz trombonist and digital composer George Lewis created a computer program called Voyager that improvises with live musicians by detecting melodic and rhythmic patterns in real-time performance and generates its own original music in response. Lewis describes Voyager as “a nonhierarchical, improvisational, subject-subject model of discourse, rather than a stimulus/response setup,” clearly articulating the distinction between media as actor and as prop or instrument.

Media need not be humanoid to function as a virtual actor.

Media as costume

Costumes in conventional theatre move in conjunction with performers, merging with them to define the character’s identity. In the same way, media that functions as a virtual costume is coextensive with the performer. For example, in D.A.V.E. (1999), Klaus Obermaier and Chris Haring projected video onto a man’s face and shirtless chest to give him a woman’s breasts and face and bulging frog-like eyes, and to create the illusion that he was squashing his face between his fists and peeling the skin off his arm, shoulder, and chest.

In some of the most sophisticated examples of media-as-costume, the media object and the live performer conjoin to create a cyborg identity. For example, at several points during the Wooster Group’s To You, The Birdie! (2002), a video monitor slides in front of a live performer’s head or legs to create an uncanny image of a whole body composed of half flesh and half video.

The costume function, as I am defining it, ranges from the use of media to enhance conventional costumes—such as by projecting images onto a fabric costume or embedding LEDs within it—to cyborg technology that conjoins multiple subjects, some live, some mediated, to produce new hybrid identities. In all of these cases, the media and performer(s) merge together in the imagination of the spectator to constitute a single fictional or virtual subject.

Media as mirror

David Rokeby, creator of the pioneering interactive installation Very Nervous System (1986–1990), proclaims that “an interactive technology is a medium through which we communicate with ourselves… a mirror. The medium not only reflects back, but also refracts what it is given; what is returned is ourselves, transformed and processed.”

Media functions as a mirror when it reflects the performer’s actions. The most banal example is the kind of live-feed video used to help spectators see close-ups of musical performances and athletic events in huge stadium venues—though, as the Wooster Group has demonstrated in a series of productions over the past fifty years, such technically straightforward live-feed video can be used to highly sophisticated aesthetic effect.

To function as a media mirror, however, the media need not visually resemble the performer at all. What matters is that the spectators perceive a continuous, reflective connection between the performer and the media object, and as a result can “read” the performer’s actions through the media. Synesthetic media, which translates a performance from one modality to another modality, exemplifies this mode: for example, a music visualization algorithm might automatically generate shapes that grow and shrink as the volume increases and decreases, that fill with warmer colors as the pitch rises and cooler colors as it descends, and/or that become rounder when the notes are sustained and more jagged when the notes are staccato.

A media mirror has no agency of its own, as a media actor does, but simply follows the performer’s actions. It does not merge with the performers, as costume media does, but runs in parallel with them. And it is not subject to the performer’s intentional control, as a media prop is. A media mirror, such as a live-feed video, reflects the performer regardless of the performer’s intentions, desires, or even, in the case of covert surveillance, knowledge. A performer’s relationship to his or her media reflection is fundamentally passive.

There is not and should not be a one-size-fits-all approach to incorporating media into performance.

Media as audience

Finally, the media can function as a virtual audience by responding to the live performance. While media-as-actor engages in a two-way interaction with the performers, media-as-audience engages in a one-way interaction. To adopt a term from film theory, it is non-diegetic, functioning out of the live performer’s frame. A basic example of media-as-audience is musical underscoring in the tradition of nineteenth century melodrama, which has become a ubiquitous convention in film. The same function can be fulfilled visually with washes of color and abstract images that convey mood. These examples of media-as-audience are purely affective, serving to model and manipulate the audience’s emotional responses to the performance.

Media-as-audience, however, can and often does assume a cognitive dimension as well, commenting on the live performance, as Erwin Piscator did in his productions in the 1920s that used analogue media such as documentary film footage. The Almeida Theatre’s 1997 production of The Cenci, with media created by Studio Azzurro, is one of many examples of a production that used digital media primarily as commentary on the live performance, with the projected images acting “as a visual counterpoint that does not attempt to explain the text but rather to open it up to different interpretations and comparisons.”

Digital Media in the Production Process

Distinguishing between these six very different modes of media helps to explain why media is often so difficult to integrate smoothly and effectively into the conventional theatrical production process. Most theatre companies clearly delineate responsibilities and timelines for each member of the production team, including the director and the scenic, costume, lighting, and prop designers. It is often unclear how media should fit into this process. In some cases, media is construed as an aspect of the lighting design. In others, it falls under the purview of the scenic designer, or of a separate projection designer who works in conjunction with the scenic designer. In other cases, media is introduced by the costume designer and has little or no impact on the scenic design.

In all of these instances, the performers typically do not interact with the media until technical rehearsals. Sometimes, however, directors conceive of media as an integral element of the production’s concept and content, and either create the media themselves or work intimately with a media designer throughout the process. In such cases, the director may try to integrate media as early as possible into rehearsals, fashioning and modifying it along with the work of the actors.

There is not and should not be a one-size-fits-all approach to incorporating media into performance. Digital media is protean and dynamic, and to exploit the prodigious and barely tapped potential of its use in performance demands a supple and creative approach—one that is adapted to the artists’ goals and the function of the media in each particular work.