How ILM Turned ‘The Irishman’ Into A Game Changer For De-Aging Effects
Nestled among the usual action and sci-fi epics in this year’s vfx awards race is a very different triumph of effects: The Irishman. Martin Scorsese’s mob saga has made waves by turning the likes of Robert De Niro and Al Pacino back into thirty-something men.
The film spans the second half of the 20th century, and by the time Scorsese came to make it, his desired actors were too old to play their characters’ young incarnations with makeup alone. De-aging effects were the only solution, but Scorsese was leery of the markers and helmet cameras traditionally used in this process. He thought they would interfere with the performances.
Enter Pablo Helman of Industrial Light & Magic (ILM), who previously worked with Scorsese on 2016’s Silence. As vfx supervisor on The Irishman, Helman spearheaded an unprecedented de-aging pipeline that eliminated physical restrictions. The result satisfied the director’s demands: to convincingly “youthify” the cast, while preserving their freedom to ad-lib comfortably and play off each other on set.
Helman and his colleagues have discussed their work with, among others, Indiewire, Wired, and Befores & Afters (to whom Helman gave an excellent in-depth interview). They also detail their approach in a featurette released by Netflix, which produced the film (see below). We’ve collated the most interesting insights below.
The crew devised a custom camera rig dubbed “the three-headed monster.” The director’s camera was flanked by two infrared cameras (which ILM developed with camera company ARRI and cinematographer Rodrigo Prieto). Together, they captured vast amounts of volumetric data on the cast’s faces. This setup led to unforeseen issues: e.g. the windshields in the old cars used on set contained lead, which blocks infrared light — so they were all removed and recreated digitally.
The cameras’ data was fed into a proprietary software called Flux, which interpreted how the faces were lit and shaded. This volumetric data was used to modify, frame by frame, digital models of the actors as they are now (which ILM created with Disney’s Medusa Facial Capture system). Having thus captured the performances, ILM was ready to de-age them.
Scorsese didn’t want De Niro as he really looked 30 years ago — he wanted De Niro’s character in The Irishman to look 30 years younger, which isn’t the same. In other words, some design was involved. ILM spent two years creating vast libraries of stills and clips of the actors throughout their career, which served as detailed reference. The team then incorporated the “young” facial elements appropriate to the shot they were working on.
Machine learning was used in the de-aging process. Having de-aged the actors for a specific shot, ILM let artificial intelligence scan the libraries and find the real shots or footage that most resembled the rendered shot (according to a variety of criteria: lighting, angles, eyes, etc). Thus the team were able to check their shot against real images of the young actors, and correct any discrepancies.
During a shoot, an actor’s appearance isn’t consistent. “In 108 days [of shooting on The Irishman], a person changes,” Helman told Befores & Afters. “Sometimes they look great and sometimes they had a bad night or they show up on set a little bloated.” This complicated the effects artists’ task, as they had to ensure consistency in the way the cast looked.
No animators were involved. The actors’ performances were in no way modified or recreated through key animation. “I think you could change the performances,” Helman told Befores & Afters, “but if you did on this project, you’d get fired!” Adds ILM’s Douglas Moore, “An animator isn’t going to get all the detail that we [got] from these tools.”
This pipeline could be a game changer for character dramas. To date, de-aging technology has mostly been used in action films, where the emphasis is on spectacle. ILM’s less intrusive technology is well suited to more intimate character studies, where the minutiae of facial acting are all-important.
“The Irishman” is available to watch on Netflix.