Disney Research Studios has published a paper and video that it has prepared for the upcoming SIGGRAPH conference happening December 6-9. In its presentation, Disney demonstrates how a tool called FRAN is a production-ready solution for re-aging actors. Disney has already used re-aging in many of its projects spanning from Tron: Legacy (2010) where it de-aged actor Jeff Bridges, to numerous Marvel and Star Wars movies and streaming shows.
Photorealistic digital re-aging of faces in video is becoming increasingly common in entertainment and advertising. But the predominant 2D painting workflow often requires frame-by-frame manual work that can take days to accomplish, even by skilled artists.
The researchers explain in great detail how they trained U-Net, a convolutional neural network developed by the University of Freiburg for biomedical image segmentation, to analyze and alter subject images. By using upsampling and downsampling techniques combined with many other computations, they used FRAN (Face Re-Aging Network) to create the new images. They have some fairly technical PDFs on the matter that can be found here and here.
“The U-Net architecture of the proposed Face Re-Aging Network (FRAN) takes as input a 5-channel tensor with the RGB image to be re-aged and two additional channels indicating the current and target age of each pixel. Optionally, a pre-trained face segmentation network (BiSeNetV2 [Yu et al. 2021]) can be used to limit re-aging to skin areas and to set localized input and output age values. We use blur-pooling [Zhang 2019] in both FRAN and the discriminator to accommodate small shifts in the positions of wrinkles and other high-frequency details.“
Image Comparisons
“We demonstrate how the simple U-Net, surprisingly, allows us to advance the state of the art for re-aging real faces on video, with unprecedented temporal stability and preservation of facial identity across variable expressions, viewpoints, and lighting conditions. Finally, our new face re-aging network (FRAN) incorporates simple and intuitive mechanisms that provide artists with localized control and creative freedom to direct and fine-tune the re-aging effect, a feature that is largely important in real production pipelines and often overlooked in related research work.“
As Disney demonstrates, FRAN continues to evolve at re-aging actors and it is constantly being refined and trained.