An Artist Has Used Machine Learning To Turn Animated Characters Into Creepy Photorealistic Figures

Reverse toonification!

Here's what one of @robhrr's drawings of Miles from #Spiderverse might look like as a real person. Source on left / result on right.

Using @yuval_alaluf and @EladRichardson's #pixel2style2pixel amazing framework. Big thanks! 🙌 #styleGAN

More in thread: pic.twitter.com/se4XM5S4J7

— Nathan Shipley (@CitizenPlain) October 13, 2020

This posse of uncanny faces comes courtesy of Pixel2Style2Pixel, a machine learning framework developed by Elad Richardson and Yuval Alaluf. It builds on the StyleGAN model, which has been used to generate highly plausible artificial faces (you can sample some at the website This Person Does Not Exist).

Richardson, Alaluf, and their colleagues explain how the framework works in a blog post. The software can be applied to a range of image-to-image translation tasks, including the “toonification” of real faces. Shipley has effectively done the reverse.

Toonify with pSp! Initial results of the pSp framework @EladRichardson and I released this week over @Norod78's and @Buntworthy's toonify StyleGAN. pic.twitter.com/33VoMghpMY

— Yuval Alaluf (@yuvalalaluf) October 6, 2020

Shipley’s “realified” faces smooth down some of the original designs’ cartoonish extremes, but their proportions remain caricatured. One user asked whether the faces could be made even more realistic, and Shipley obliged, although he didn’t specify how he carried out that second translation:

Good idea – here's playing with Dash to make him more realistic – this isn't the same technique as above but it's interesting! Thanks for the suggestion! pic.twitter.com/VVBcQGkZcW

— Nathan Shipley (@CitizenPlain) October 16, 2020

These experiments with design are interesting because they mirror trends in the entertainment world. One is a growing tendency toward photorealism in some animation and vfx; Shipley’s transformations look almost like a parody of Disney’s approach to its remake of The Lion King. A closely related phenomenon is the rise of synthetic influencers, who are meant to look at once idealized and naturalistic. Lil Miquela looks “realified” to begin with, but Shipley cheekily submits her to the same process as the other characters:

What if @lilmiquela was real? Using #MachineLearning to create three plausibly real versions of a CG version of a real girl. pic.twitter.com/pRpQ5Ff1Cv

— Nathan Shipley (@CitizenPlain) October 13, 2020

Image at top from Nathan Shipley’s Twitter account.