NVIDIA’s Omniverse Audio2Face is a tool that uses RTX AI tensor cores for syncing audio to video animations. It is free to download and requires an RTX graphics card with at least 6 GB VRAM and Windows 10 (build 1909 or higher). Audio2Face is the latest expansion to NVIDIA’s Omniverse open platform for virtual content collaboration.
Audio2Face simplifies animation of a 3D character to match any voice-over track, whether you’re animating characters for a game, film, real-time digital assistants, or just for fun. You can use the app for interactive real-time applications or as a traditional facial animation authoring tool. Run the results live or bake them out, it’s up to you.
NVIDIA has also provided tutorial videos on its on-demand page. Users can learn how to incorporate the software and view many examples. The included 3D character model is called Digital Mark. Some examples show that audio quality plays a factor in the ability of the software to correctly sync to video.
Charlie Chaplin’s uncharacteristically famous speech from the film, The Great Dictator is featured in one video and the movements just don’t seem as clear as they do with crisper, modern recordings. That being said, unlike some audio quality, this tool is likely to only get better with time.
Omniverse Audio2Face Demos
There are demo videos showcasing a number of features such as face swap, data conversion including blendshape conversion, and also blendweight export along with export options for Unreal Engine 4. More in-depth videos show off the complexity of more detailed facial animations such as aliens and completely recreated scenery and from Mount & Blade II: Bannerlord. The software can also work with any language.
Audio2Face is said to process any language easily and the team is continually updating it with more and more languages. The tool can also be used to animate stylized characters or humanoid aliens, for example.
There’s even a video of NVIDIA CEO Jensen Huang interacting with an AI weather bot using the software to create its facial animations and synced audio.