Sound and Media Design
This blog series offers a multi-disciplinary approach to achieve the best practices for collaboration in the creative and production process of incorporating digital media into live performance.
Sound and video are linked. Think back to the AV Club in high school: Audio-Visual. Typically, if you have video, there is going to be some kind of audio—be it a person talking, music, sound effects, etc. This is important to remember when planning productions because it will affect everything in the production pipeline from content creation, recording, editing, output, and playback. Our goal is a smooth process and a unified design, which creates a dramaturgically meaningful and rich world. Audio (#TonyCanYouHearMe) is a key component in this process.
On most video productions, there is a sound department and a camera department that have an established production and post-production workflow. Yet this workflow does not exist in most live performance scenarios.
Who’s in Charge of What?
Who is responsible for creating the audio content for a video? Media designers come from different backgrounds including video production, animation, and lighting, so not everyone will have experience in audio. Even if a designer is from a video production background, this doesn’t mean she is a pro on how to set-up microphones, record, and perform basic audio editing. More often than not, the media designer’s strength is in visual storytelling and she works in audio simply because she has to; because of the A in AV.
Most theatrical sound designers will not automatically think it is their job to record, edit, and master dialogue captured in a video shoot. But someone must. What about sound effects or music added for an animation? Who does this? On most video productions, there is a sound department and a camera department that have an established production and post-production workflow. Yet this workflow does not exist in most live performance scenarios. It is up to us to agree upon a workflow between media and sound for each new production.
If the sound designer of live performance is only going to be responsible for the playback of audio during the performance, it is important to have whoever is in charge (of recording, editing, and mastering sound for all video sequences) be a key member of the design team. How the sound is recorded, edited, what types of effects, and the music chosen all play a role in meaning making. The sound artist/technician for video production needs to understand how the overall sound of the live performance functions. This is the only way that her audio and sound will work for both the video sequence, and fit within the greater world/context of the live performance.
Key Components and Workflow Between Audio and Media in the Production Pipeline
Below is a typical workflow between the audio and media department when there is any audio associated with a video.
- For the recording of any location audio during a video shoot (this can include dialogue, ambient sound, etc.), decide who will be in charge of recording and what gear will be used. Then, hire an additional sound technician if needed.
- Edit location audio. Path 1: Sound Department edits/equalizes all location audio and sends to media department to sync audio with video. Media Department edits picture with polished audio. [NOTE: After picture lock (no more changes to the video, which is usually a few days before the show opens), the Sound Department will need to Master the final video’s audio.] Path 2: Media Department syncs audio and video, then edits picture with audio. Once there is picture lock, Media sends the audio file to the Sound Department to Edit/Equalize.
- If music is used in a video, decide on music. Path 1: Media Department cuts picture to music. Original Music: Sound Department composes/records/edits music first. Existing Music: Either department acquires the rights to music. Path 2: Sound Department composes music to edited video. Original Music: Sound Department composes/records/edits music to picture. Existing Music: Either department acquires the rights to music.
- If Sound Effects are used in a video, the Media Department edits picture. Then, sends rough file to Sound Department. Sound Department creates all sound effects to picture. Depending on the production, the Media Department may have one more round of editing after sound effects are added.
- Master all audio once there is picture lock and all sound effects, music, etc., have been created, the audio needs to be mastered. This step is usually skipped in live performance productions because videos can change so frequently and last minute.
- Audio Playback in the Theatre: Who will control playback of the audio file? If it’s the Media Department, the video file needs to be outputted with audio for playback in the media server. Check the media server for optimal sound codec and settings. Then, an audio line out (either digital or analog) from the media server will need to be connected to an input on the Sound System Control. But if it’s the Sound Department, the video file will not need sound outputted with it. (Sometimes I output the sound as a reference.) A separate master audio file that matches the exact length of the video file needs to be outputted separately from the video by the Media Department. (This is an extra step in the process.) There may be a lag/delay between audio and video causing the audio and video to be out of sync. If it is only music, it may not be too noticeable. If it is a video of people talking, it may be very visible. Instead of a stage manager calling a video and audio cue while two board ops try to hit the go button at the same time, you may want to consider: Using Open Sound Control (OSC) or Musical Instrument Digital Interface (MIDI) to fire the audio and video at the same exact time. Even doing so, you may have to put a delay on the video or audio so they are in sync.
There are some truly amazing explorations of how sound and digital media work together to dazzle, delight, and shape stories and experiences in exciting, bold new ways.
Audio Playback in the Performance Space
In a lot of productions, systems like QLab have become the show control software of choice for many sound designers. This type of cueing and control program offer designers great flexibility and the ability to communicate directly with media servers—allowing for the option to send and receive different kinds of commands that act as triggers for events (cues). QLab is able to send and receive triggers via OSC, MIDI or via timecode, allowing a sound cue to trigger a projection/media cue, or vice versa. You will need additional gear and time to set-up a computer network so that the sound control computer and media server can communicate with each other.
At smaller venues and for productions on tighter budgets, QLab is an affordable control system that can cue/control media. If QLab is going to be used for both media and sound cues and one machine/one operator will control all sound and media cues, there must be thought given on how media and sound will cue during tech in terms of numbers, auto-follows, etc.; tech time must be allocated to integrate the sound and media cues.
Beyond Playback to Real-time Audio Reactive Environments
Audio engineers, designers, instrument makers, musicians, and audio software developers are often at the forefront of creating and adopting new technologies. They are fearless leaders, blazing new methods to create and control sound from 3D special sound environments to audio reactive installations. There are some truly amazing explorations of how sound and digital media work together to dazzle, delight, and shape stories and experiences in exciting, bold new ways. I encourage you to seek them out and incorporate them into your productions.
Up Next: Academia and Media Design