Lytro And Rodrigo Blaas Collaborate on VR Short Using New Volume Tracer
In recent times, Lytro, the company known mostly for developing light field cameras, has been moving into vr. It has also announced a specialized renderer called the Lytro Volume Tracer, for rendering cg vr experiences.
The announcement is accompanied by a cg animated vr short called One Morning, directed by Trollhunters showrunner Rodrigo Blaas and made using the Nimble Collective platform.
Cartoon Brew looks more at light field technology, and at the making of One Morning with Blaas and Lytro visual effects supervisor John Berton Jr.
What is a light field?
Before we get into Lytro’s volume tracer, let’s look first at light field technology. A light field is essentially all the light that passes through an area or volume. A normal camera just captures the rays of light coming through its lens and aperture which then hits the film or sensor to produce an image.
But a light field camera effectively has multiple lenses, or is made up of an array of cameras, each at slightly different positions. That means it can capture light from multiple vantage points and then work out different things about that light such as intensity and angular direction. Processing that information in a certain way means you can replicate the captured light field in a virtual space, and do things with it.
What can you do with it? Well, Lytro had initially been selling both pro and consumer light field cameras. One selling point with these cameras was that photographers could take a photo of a scene and then with special software re-focus onto different areas in the image. But in 2015 Lytro changed tack after realizing it could do more things with captured light fields. It jumped into the cinematic side of light fields, and in particular, into vr.
Recently, the company has been showcasing its Lytro Immerge system – a light field camera array, server, editing, and player set-up – to shoot and produce light field films. This involves shooting live action with a large light field camera array, but the system also facilitates the compositing of cg imagery into the live action. Lytro showcased the result in the vr film Hallelujah. This behind the scenes video, below, shows how Hallelujah was made.
Lytro argues that its system is a better enabler of creating vr experiences with six degrees of freedom movement, or 6DoF. Basically this means letting a user experience a full range of motion while wearing vr goggles (high-end goggles such as the Oculus Rift and the HTC Vive help with range of motion by allowing for positional tracking). Being able to look around everywhere and move in vr with 6DoF is said to enhance the experience because it’s basically how we move in real life – that’s why some vr experiences that are not 6DofF feel limiting. (For a more detailed explanation of the six degrees, read this.)
Volume tracing
OK, so what then is the Lytro volume tracer all about? Firstly, it’s part of Lytro’s push into more immersive vr experiences; in this case, fully-cg ones. The volume tracer is an addition to, rather than a replacement for, current 3d tools and current 3d render engines (including game engines) to produce a light field representation of a 3d scene.
“That light field then represents not only just the three dimensional space, but also all the light within that three dimensional space that was created by the synthetic lighting and the reflections, and so on,” explains John Berton Jr., who previously supervised the visual effects on films such as The Mummy, Men in Black II, and Charlotte’s Web. “We re-build it using the same technology that we use to build three dimensional worlds out of two dimensional images in photographic space, using our light field camera.”
To create live-action light field experiences that work in vr, Lytro has to do a lot of ‘clever’ things with its camera array and Immerge system. But for fully-cg experiences – this is what Blaas’ short One Morning is – the data for creating a three-dimensional experience is already there in, say, a Maya, 3ds Max, or Houdini file. The Lytro volume tracer is simply a way of re-rendering the scene to create a light field version of it.
“Once we have a light field,” says Berton, “we push it through our system that we have already, and that gives us the opportunity to look in on an animated movie the same way that we look in on our photographic material. That means that you have six degrees of freedom to move around a volumetric space, and explore the world in front of you.”
But why is Lytro’s volume tracer necessary in the first place? Berton Jr. says, predominantly, it’s about fidelity in creating light field-based vr and mixed reality experiences, but without impacting on how a creator makes their material.
“We just take the data, and it goes into our system,” comments Berton Jr. “That’s what we did with Rodrigo. He went out and built his system in Maya, with the Maxwell renderer, and we took his data, and his renderer, and we made One Morning with just that information. We didn’t touch the data in the animation files. We didn’t touch the shaders. We didn’t touch any of the parameters of the lighting. We just read it out of the file, put it into our system, and produced the piece you’re seeing in the headset.”
Making One Morning
To showcase what the Lytro volume tracer could be used for, the company commissioned Blaas and his El Guiri Productions to make One Morning as a demo. The short is about an errant robot bird looking for food. In the film, the viewer (wearing vr goggles) has an encounter with the robot bird.
“I just wanted to explore this idea of what happens to older versions of robots,” Blaas told Cartoon Brew. “What is going to happen with your iPhone 5 or 4? I feel like there’s going to be this idea of discarded robots because they’re not lasting generations.”
The film was made via the cloud-based animation platform Nimble Collective. A team of artists from around the globe collaborated on it, something Blaas was keen to explore further. He was also keen to look into vr films, but his previous experience with vr had been that most projects he looked at were decidedly low resolution.
“I wasn’t interested in vr until now because I’m always trying to push the quality of the work that I do in animation,” Blaas said. “The thing that attracted me was when Lytro came back saying, ‘We don’t want you to change your process. We want you to keep all the detail – the really vibrant and high quality imagery. You create an animation and we’re going to be the ones to translate that into a vr experience.’”
Like a number of filmmakers who have worked in vr, Blaas had to adapt his style for a medium in which the viewer can look anywhere. The bird is animated to look directly at the viewer and also gaze elsewhere, which the viewer can follow along with.
“The tricky part is knowing how to direct the viewer to what you want to see and what you want to tell him in that environment,” said Blaas. “You have to be way more aware of what the sound around you is doing, and the visuals, and how to simplify the story you’re telling. You really concentrate the viewer on one thing.”
“One of the things that we started putting in originally was a lot of detail around us,” adds Blaas. “Then we started stripping it, in the way that we wanted to get the attention on one place so you can follow the story of this little bird.”
Lytro’s light field future
Other companies have been diving into this area, too, such as OTOY with its Octane renderer. But it’s clear that getting vr and light field content out there for audiences to see is going to be an important part of mainstream adoption.
Lytro acknowledges that it is still early days in terms of getting creators on board with its tools. Still, Berton Jr. says, many have been asking them about mixing more live-action light field footage with cg light field footage.
“That’s something that we’re very well aware of, and once you’ve got everything in light field space, then we’ve got a game to play,” the visual effects supervisor said. “We’re absolutely talking to those people, and we’re looking for great projects that we can continue to develop our pipeline with.”
One Morning was featured last month at both VR on the Lot and Animation is Film festival, both in Los Angeles. Lytro has a form on its website to request viewings, and tells Cartoon Brew there will be further screening locations announced soon.