Pearl, the seventh interactive experience in Google’s Spotlight Stories program, premiered last month on YouTube’s 360-degree player and the Google Spotlight Stories iOS app (both of which are compatible with Google Cardboard).
The ambitious short, directed by Patrick Osborne, who won an Oscar last year for the Disney short Feast, attempts a more complex narrative than many virtual reality projects to date, telling a tender story about a father and daughter on the road that spans both time and place.
Cartoon Brew spoke with Osborne when the film premiered at Tribeca in April, in a roundtable discussion that also included Spotlight Stories executive producer Karen Dufilho, Spotlight’s creative director of music and sound Scot Stafford, and Pearl producer David Eisenmann.
In our conversation, we discussed how standard concepts in traditional filmmaking, like cutting and editing, can be huge technical challenges in vr, and explored vr’s new complexities, like sound, which must be more dynamic to take into account the immersive environment in which a viewer watches the film.
CARTOON BREW: Was the story always about a dad and daughter?
PATRICK OSBORNE (director): Not really. There was a version of the story where it was really following the car, like on a new lot and then through different owners. It started as a Giving Tree with a car where we would trash the car over time, but I ended up wanting to follow the same people. And then I’m a folk music junkie. I go to SXSW in Austin every year to take it in, all the singer-songwriters there. I thought it’d be cool to do that, but as a studio project, I don’t think it would ever be possible for me to do a folk-musical. These guys [at Google] were open for whatever creatively.
CARTOON BREW: Did you have a song or musician in mind?
PATRICK OSBORNE: We didn’t. I knew that Scot was their music director and had done great stuff. He did Presto at Pixar, and Glen [Keane’s Spotlight] short Duet. I talked to Scot a little bit about it, and I felt like we could relate on the music side. So we put a call out through his music collaborative Pollen for demos of songs, and about twelve songs were submitted as options, some from people that I was a mega-fan of, but Scot didn’t let me know who wrote the songs when I first heard them so I had to choose independent of the name.
CARTOON BREW: You could recognize the voice though?
PATRICK OSBORNE: He was like, ‘Do you know who that one is?’ Do you know who that one is?’ And it made me sad that I actually didn’t like the ones that I was a fan of as much as the people I didn’t know, but that’s good in some ways, too.
SCOT STAFFORD (creative director, music and sound): I think the key was that we had played the song back against some storyboards that Patrick had done so you could really see the characters and story against the music without any noise.
PATRICK OSBORNE: I sent storyboards out with temp music on them; Scot cut the temp music off and just had artists write to the visuals. They were pretty cool to watch, but it had to be dismantled later when we realized that in 360, you can’t storyboard that all. But I chose a song and then a different singer…I don’t know if that was going to be insulting to Alexis [Harte], who wrote the song, but he was on board for doing that. So we had two different singers who also wrote their own songs, but I kind of said that guy sounds like the dad, and this could be the girl, and let’s record it like that. The story structure is the same as a pop song, it’s a verse-verse-chorus-bridge thing. Alexis wrote way more verses than we needed.
SCOT STAFFORD: It was more than a dozen, which Patrick was able to reassemble. And it was really interesting thinking, Oh, this verse could be from the perspective of the dad. This could be from the daughter. Doing it early on in that process made everything else possible.
CARTOON BREW: You’re not a dad yourself, Patrick. Why did the father-daughter angle resonate so strongly?
PATRICK OSBORNE: That’s sort of for my dad. My dad was a toy designer at Kenner. When I was 10, Kenner was sold to Hasbro, and the company moved, but he decided not to move and not to continue doing what he was passionate about, so we didn’t have to move or change schools. Instead he went into teaching. He was the director of design at Kenner so he was very involved in everything and then all of a sudden, not. It kind of started from there. I think my dad feels that my career working out in the way it has has sort of satisfied that in him; that maybe he didn’t actually do his thing, but he passed on passion for something that is related.
PATRICK OSBORNE: People were talking about not doing edits in vr and I thought if you had a consistent environment, it’s probably okay and not disorienting to try and edit in a 360 vr environment. This was the first request to make a cut for Google [Spotlight Stories] so they had to figure out how to make it possible first, and then how to make it so you could see it and give notes on it.
CARTOON BREW: On Jan Pinkava’s Windy Day, when the viewer goes up into the tree and the scene changes, that wasn’t considered a cut?
KAREN DUFILHO (executive producer): It’s not a cut. It’s all our trigger-based technology so we kind of shifted sets when you weren’t looking at it. It’s just sleight of hand.
CARTOON BREW: So what did you have to request on this film?
PATRICK OSBORNE: The ability to cut.
KAREN DUFILHO: Which sounds really easy.
PATRICK OSBORNE: I thought it was going to be way easier than it was.
DAVID EISENMANN (producer): Aardman’s Special Delivery, the entire project is as big as one of the 40 shots that we have in Pearl.
PATRICK OSBORNE: It’s kind of boring and technical, but a shot doesn’t just cut. You have to load all the stuff before you can transition to it. We’re already making the film do too much by loading one scene…
DAVID EISENMANN: Because it’s all real-time.
PATRICK OSBORNE: …so loading the next scene became an issue and I actually don’t know how it’s solved. It’s just magic.
SCOT STAFFORD: It was a really risky thing because we were wondering, Can this work? Can you actually be in vr, which has the threat of being so disorienting, and do a cut into a completely different scene? The first cut [in Pearl] is a huge one; you’re going back in time. The risky thing was we couldn’t tell if we could get away with it until like three weeks ago because it took so long to load up the next scene. And so everyone was saying, “It’s not working. We have to rethink this whole thing.” And we just kind of stuck with it.
DAVID EISENMANN: But you know, there was a lot of investigation and discussion on our end about what the cut was. We tried blinks. We tried fades, dissolves.
PATRICK OSBORNE: They all looked stupid to me.
DAVID EISENMANN: All of those didn’t work. Then we sort of had to help the audience learn what those cuts would do and how to introduce them into that world.
KAREN DUFILHO: But it’s two different things: How does it look, and then how do we actually do that underneath?
PATRICK OSBORNE: A requirement for the song was that we would be able to record it in a way that it would sound like it’s sometimes sung in the car, sometimes it’s over the radio, sometimes it’s outside the car, and sometimes it’s a score. Scot actually went out and bought a fancy microphone just to do that, so we actually made the poor singer sing it so many times just standing in different places in a car to get that sound to feel right.
SCOT STAFFORD: Every set you see him singing, whether he’s driving the car and playing at the same time, or he’s outside busking, he had to sing just his voice all the way through the song, then play guitar all the way through, and then sing and play all the way through, in every single position, because we weren’t exactly sure how it would work.
CARTOON BREW: You can’t just alter the sound in post?
SCOT STAFFORD: You can, but it won’t be quite as believable, especially if you see the vr version of it. The way game audio works where you have these object-based point sources is not how sound works. Right now you hear my voice bouncing off all of these surfaces in the room, and if I was to rob you of that, you’d be really disoriented. It’d be strange to hear a really dead voice without any of the acoustics. So by recording it the way it really is, with a microphone that captures a spherical representation of sound that you can then rotate with a camera, it puts you there in a really special way.
CARTOON BREW: What continues to surprise you about doing sound on vr projects after you’ve done it on about a half-dozen films.
SCOT STAFFORD: This was the seventh, and each one has sort of asked a different question and come up with a different answer. By the end of [the fifth project, Shannon Tindle’s] On Ice, there was not a single technique that had been re-used. So I hadn’t had the opportunity to say, “OK, let’s take a look back and see what worked on that one and then we can improve on it.” It was like starting from scratch on each one, and eventually building tools that were capable of handling all of these techniques. Pearl was really the first time where I looked back on all of the previous Spotlight Stories, and did exactly what I just said: We can use this technique, we can use that technique, and I had some of that confidence of knowing that it’s going to sound like shit for months and months, but at the very end, when things come together, it’ll work. As an artist, you need some of that confidence, otherwise you’re not going to take any risks. For this one, the big surprise was how these complex audio techniques can result in something that feels very natural and simple, without explosions, fireballs, or things that sound like special effects. Well, deceptively simple, because under the hood, all of the complex techniques are working in service of that.
GETTING THE LOOK RIGHT
PATRICK OSBORNE: Oren Haskins and Tuna Bora designed the characters together, it’s a mix of the two. I really wanted to try to get close to what their design styles were; the artwork’s amazing. The problem with the characters actually was they’re articulated and rigged in a fairly detailed way—not as much as a feature, but more than like a typical game character would be, which means there’s a limit to how many of those characters can be onscreen at one time. I think it’s like four, sometimes five. But phones are fairly powerful now, enough to render that stuff out decently well.
The shaders are just a solid color. There’s geometry that’s invisible in the scene, and we have it rendered as black-and-white. It’s a mask, and it’s masking out light over dark. So every shader color has a dark and a light, and depending on if it’s behind the mask or not, which is an invisible piece of geometry, it lights. But it’s not like a light; it’s designed light, like an artist went into everything and made geometry that makes the light work. It was sort of my interpretation of the techniques that I’ve been using before on other things because I don’t like how toonshading looks. It doesn’t let the human design the light shapes and it’s nicer when a human does it. Cassidy Curtis, who came from PDI Dreamworks, did most of that himself during the the last couple months of patching it together and making it work.
CARTOON BREW: It didn’t feel toon shaded…
PATRICK OSBORNE: Not at all. It’s just using geometry to make a color; it’s the same thing you do in Photoshop where you make a mask and paint it with one brush stroke. It’s that kind of theory but over time. The characters are animated at 15 frames per second to be divisible evenly into 90 or 60 or 30, but the actual camera moves at the fastest speed it possibly can. So even in the Vive [the most immersive of the four versions that Google created], where you have the camera at 90 frames per second, the characters still move on twos essentially, at kind of a choppier frame rate, which I think looks a little more crunchy and human-made, and I just like how it looks.
The film version we ran into issues there because it’s at 24 and it doesn’t multiply into 15. I realize that maybe I should have animated at 12 because that does multiply into 60. So we did have to figure out how to get the actual rendered frames out without doing a 3:2 pulldown because I wanted an even frame rate. That was a bit of an issue when we went to the edit suite.
CHALLENGES OF PLANNING AND GIVING ANIMATION NOTES
PATRICK OSBORNE: It’s technically interesting and weird to make stuff like this because you can’t really storyboard it. One of the issues with this 360 thing that people are going to have to come to terms with and figure out how to make it work is that we’re still creating animation on screens for not-screens. I know there’s software like Tiltbrush that is starting to become a creative tool within vr, but I think you really should make it all within vr at a certain point because the biggest struggles you have are in the planning phases. There’s really no tools built for it. So we just had to make a few small things here and there to help make it possible. Like there’s no such thing an editor; you can’t really edit. You can storyboard and really painstakingly place drawings on 3D objects around you, but it’s not a workflow.
I think animating is basically the same except nothing is hidden. Nothing is just off-frame so you can’t really save any time there. Giving notes is really hard because they’re either shooting very wide where the character’s small and you can see the whole thing, or we’re close-up for details but then you don’t see the legs or something. So, for one shot, you need the animator to [output a version] up here, and then down here, so I can give notes on everything. Animating, it’s similarly kind of weird because you’re animating on a flat plane and you can only see some of the character. You can’t see the whole character at one time.
We were a Maya Shotgun pipeline; Shotgun is an organizational tool. We had artists all around the world and it’s really essential to have that place where I can type notes and draw notes over artwork for people. About 18 people worked in San Francisco, but then we had another 15 or so just wherever they were, which is cool now that animators can do that. Ken Fountain, a wonderful animator who did a bunch of stuff on this, lives in Virginia because he wants to live there, but he works on high-profile great stuff all around the world. He worked on The Peanuts Movie, and it’s neat to be able to live anywhere and make it work if you’re willing to be a workhorse. I don’t think leadership roles quite work that way, but getting animation work is pretty easy to do anywhere in the world now.
ADVICE FOR FIRST-TIME VR DIRECTORS
CARTOON BREW: Karen, everyone you’re working with is a first-time vr director because this is so new. What are some of the things that directors need to consider when they’re switching to this medium?
KAREN DUFILHO: I think really thinking and considering the space, the architecture, and the world-building of it. It’s this completely different back-and-forth between creator and audience in vr; it’s one-on-one. So there’s this new conversation that’s happening, and you have to consider the entire sphere from day one, you have to believe that that is true as you go through the process and think conceptually about what you’re doing. You really want to control it. Patrick, from day one, was, “Let’s see, if I’m in the car, I can frame out of the window, and that gives me some grounding.”
PATRICK OSBORNE: I was afraid of losing compositional control. People watch the films vertically; that’s, like, insane. But the car windows are still some sort of frame.
KAREN DUFILHO: People will do what they do. But the whole point of it is that [the viewer] has the freedom.
PATRICK OSBORNE: As a director, you can’t possibly see everything. You just have to kind of set a mood and a feeling that you want, and hope that you’ve got the team to build the right stuff, because you’re not going to see everything. It’s not as noodly; you can’t pixel-f it.
DAVID EISENMANN: Patrick and I both came from a place where previs was a huge part of the process, and so from the get-go on this one, it was a progressive refinement of every shot all the way through. They’re not boards, whatever you want to call them, the boards turned into 3D previs. At any given point, you’d end up with final shots in some shots, and really blocky bad looking stuff, but at least you can understand the framework of the world. I don’t think a lot of directors understand that right now because they either underestimate the amount of work or investment they’re going to need to make into that space. A lot of it is for things like animation. Like, guess what, our animators didn’t animate three-quarters of what we didn’t see when we were reviewing it. Same thing happens with everything you want to cheat. Our vr version is really cool as a demo, because we didn’t fix that stuff but if you really do poke around, all of the cheats that we did are busted.
KAREN DUFILHO: Yeah, they’re called easter eggs. [laughter]