NVIDIA has announced NVIDIA Avatar Cloud Engine (ACE) for Games, a custom AI model foundry service that aims to transform games by bringing intelligence to non-playable characters (NPCs) through AI-powered natural language interactions. The service basically enables gaming experiences in which the player can have intelligent, unscripted, and dynamic conversations with NPCs with persistent personalities that evolve over time, including accurate facial animations and expressions, although a 4K demo video that was released to showcase the platform suggests that the resulting dialog could sound a bit stiff and unnatural. The video, which comes in a ridiculously wide aspect ratio and stars Jin, a ramen shop worker, was created by the NVIDIA Lightspeed Studios art team and rendered entirely in Unreal Engine 5, using NVIDIA RTX Direct Illumination (RTXDI) for ray traced lighting and shadows, with DLSS for the highest possible frame rates and image quality.
“Generative AI has the potential to revolutionize the interactivity players can have with game characters and dramatically increase immersion in games,” said John Spitzer, vice president of developer and performance technology at NVIDIA. “Building on our expertise in AI and decades of experience working with game developers, NVIDIA is spearheading the use of generative AI in games.”
NVIDIA ACE for Games AI Foundation Models
- NVIDIA NeMo, which provides foundation language models and model customization tools so developers can further tune the models for game characters. This customizable large language model (LLM) enables specific character backstories and personalities that fit a developer’s game world. Developers can better align player interactions within the context of a scene through programmable rules for NPCs with NeMo Guardrails.
- NVIDIA Riva, which provides automatic speech recognition (ASR) and text-to-speech (TTS) capabilities to enable live speech conversation with NVIDIA NeMo.
- NVIDIA Omniverse Audio2Face, which instantly creates expressive facial animation for game characters from just an audio source. Audio2Face features Omniverse connectors for Unreal Engine 5, so developers can add facial animation directly to MetaHuman characters.
From an NVIDIA GeForce post:
The Kairos demo leveraged NVIDIA Riva for speech-to-text and text-to-speech capabilities, NVIDIA NeMo to power the conversational AI, and Audio2Face for AI-powered facial animation from voice inputs. These modules were integrated seamlessly into the Convai services platform and fed into Unreal Engine 5 and MetaHuman to bring Jin to life.
Game developers and startups are already using NVIDIA generative AI technologies. For instance, GSC Game World is using Audio2Face in the much-anticipated S.T.A.L.K.E.R. 2 Heart of Chornobyl. And indie developer Fallen Leaf is using Audio2Face for character facial animation in Fort Solis, their third-person sci-fi thriller set on Mars. Additionally, Charisma.ai, a company enabling virtual characters through AI, is leveraging Audio2Face to power the animation in their conversation engine.