Top Story: A Historic Box Office Weekend: Five Of The Top Ten Films Are Animated Features

What if you could sketch a pattern or design on a piece of paper and have all the hand-drawn intricacies of your work instantly preserved in the texture of a 3D model?

Well, it’s already a possibility thanks to StyLit, a new development in non-photorealistic rendering by Czech Technical University, in partnership with Adobe. Cartoon Brew saw the tech at SIGGRAPH this year and immediately wondered whether it could also be translated to animation. So we talked to one of the lead researchers.

What is StyLit?

First, let’s get a handle on what StyLit does. Using a specialized algorithm, the app stylizes the rendering of a 3D model based on an example style drawn by the user. To do that, the tool not only analyzes color information but also information about light propagation in the scene.

That’s the killer part of this tech: the analysis of light lets StyLit reproduce—in real-time—specific illumination effects such as specular highlights, shadows, glossy reflections, color bleeding, and indirect illumination, “for which artists typically use different stylization techniques,” according to research co-author Daniel Sýkora, an associate professor in the Department of Computer Graphics and Interaction at Czech Technical University in Prague.

So how does all this work? To get the whole pipeline running, two components are necessary. One is a global illumination render which allows StyLit to separate individual lighting effects (such a render can be done using NVIDIA’s Iray renderer, for instance). The other component is what Sýkora describes as “our novel guided image synthesis algorithm that enables us to transfer patches from hand-crafted exemplar to corresponding locations in the target rendering.”

Real-world applications

But will artists be able to get their hands on StyLit, and is it useful? Right now, the StyLit method is patent pending. Sýkora says there are plans for his university to release a simple executable demo which would allow others to play with the app for research or other non-commercial purposes.

The original 3D rendering (a) and examples of how it was stylized: (b) tonal drawing, (c) colored pencils, (d) oil pastel, and (e) comic drawing. Note how the lighting effects on the sphere are transferred to a similarly illuminated location in the target 3D rendering. Rendering by Daichi Ito (b), Pavla Sýkorová (c, d), and  Lukáš Vlček (e).
The original 3D rendering (a) and examples of how it was stylized: (b) tonal drawing, (c) colored pencils, (d) oil pastel, and (e) comic drawing. Note how the lighting effects on the sphere are transferred to a similarly illuminated location in the target 3D rendering. Rendering by Daichi Ito (b), Pavla Sýkorová (c, d), and Lukáš Vlček (e).

People in the digital arts community certainly seem keen to see what StyLit can do. Matt Wallin, for example, is a former visual effects artist at Industrial Light & Magic and Weta Digital, and now associate professor at Virginia Commonwealth University in the School of the Arts, and says many of his students are traditional illustrators by trade.

“For them to be able to use their traditional drawing skillsets along with their original styles and aesthetic sensibilities to create one of a kind 3D textures and looks is really powerful,” Wallin said. “It has the potential to bring so many more artistic sensibilities into the world of 3D graphics.”

StyLit could find use in animation, too, and that seems like one of the most exciting possibilities for the tool. The researchers have already produced animation out of StyLit by rendering individual animation frames and letting the algorithm stylize them independently. Although the results exhibit a degree of temporal noise, Sýkora notes this can actually be a desirable feature, citing a similar kind of look in Bill Plympton’s hand-drawn animation, for example.

Importantly, StyLit was not just developed inside a research vacuum. Six artists co-operated with the researchers and provided feedback. Of course, Adobe Research researchers are also co-authors on the StyLit project.

So does that mean we might see the app as part of some kind of Adobe product? Sýkora could not say, but the idea of an executable demo sounds promising and something to keep an eye out for in the meantime.

To learn more, read the full text of the StyLit research presented at SIGGRAPH 2016.

A comparison of StyLit's rendering algorithm against the commercially available DeepArt.io.
A comparison of StyLit’s rendering algorithm against the commercially available DeepArt.io.

Latest News from Cartoon Brew