An Artist Has Used Machine Learning To Turn Animated Characters Into Creepy Photorealistic Figures
Please don’t show this to the Walt Disney Company lest they get any ideas.
Nathan Shipley, a technical director from San Francisco with experience in vfx and motion graphics, has taken to Twitter to post “realified” versions of characters from famous animated films. He’s applied the technique to protagonists from Pixar’s Up, Coco, The Incredibles 2, and Sony’s Spider-Man: Into the Spider-Verse, as well as virtual being Lil Miquela and paintings like the “Mona Lisa.”
Today's reverse toonification experiments with art from @Pixar for Incredibles 2, Up, & Coco.
This framework from @EladRichardson and @yuvalalaluf quickly finds a "real" human face in the #StyleGAN FFHQ latent space. Adding some style randomness too.
More Pixar in thread! ⬇️ pic.twitter.com/uI2v4Tn96m
— Nathan Shipley (@CitizenPlain) October 15, 2020
Reverse toonification!
Here's what one of @robhrr's drawings of Miles from #Spiderverse might look like as a real person. Source on left / result on right.
Using @yuval_alaluf and @EladRichardson's #pixel2style2pixel amazing framework. Big thanks! 🙌 #styleGAN
More in thread: pic.twitter.com/se4XM5S4J7
— Nathan Shipley (@CitizenPlain) October 13, 2020
This posse of uncanny faces comes courtesy of Pixel2Style2Pixel, a machine learning framework developed by Elad Richardson and Yuval Alaluf. It builds on the StyleGAN model, which has been used to generate highly plausible artificial faces (you can sample some at the website This Person Does Not Exist).
Richardson, Alaluf, and their colleagues explain how the framework works in a blog post. The software can be applied to a range of image-to-image translation tasks, including the “toonification” of real faces. Shipley has effectively done the reverse.
Toonify with pSp! Initial results of the pSp framework @EladRichardson and I released this week over @Norod78's and @Buntworthy's toonify StyleGAN. pic.twitter.com/33VoMghpMY
— Yuval Alaluf (@yuvalalaluf) October 6, 2020
Shipley’s “realified” faces smooth down some of the original designs’ cartoonish extremes, but their proportions remain caricatured. One user asked whether the faces could be made even more realistic, and Shipley obliged, although he didn’t specify how he carried out that second translation:
Good idea – here's playing with Dash to make him more realistic – this isn't the same technique as above but it's interesting! Thanks for the suggestion! pic.twitter.com/VVBcQGkZcW
— Nathan Shipley (@CitizenPlain) October 16, 2020
These experiments with design are interesting because they mirror trends in the entertainment world. One is a growing tendency toward photorealism in some animation and vfx; Shipley’s transformations look almost like a parody of Disney’s approach to its remake of The Lion King. A closely related phenomenon is the rise of synthetic influencers, who are meant to look at once idealized and naturalistic. Lil Miquela looks “realified” to begin with, but Shipley cheekily submits her to the same process as the other characters:
What if @lilmiquela was real? Using #MachineLearning to create three plausibly real versions of a CG version of a real girl. pic.twitter.com/pRpQ5Ff1Cv
— Nathan Shipley (@CitizenPlain) October 13, 2020
Image at top from Nathan Shipley’s Twitter account.