NVIDIA is attempting to lift people’s spirits even higher ahead of the Christmas holiday with Toy Jensen, an entirely AI-driven avatar of NVIDIA’s CEO that sings Jingle Bells, developed using the company’s Omniverse Avatar Cloud Engine and inspired by a recent, viral event in which the executive was recorded requesting/singing Lady Gaga songs, including “Always Remember Us This Way” and “Hold My Hand,” the theme song for this year’s $1.48 billion blockbuster, Top Gun: Maverick.
To make it happen, NVIDIA’s team used a recently developed voice conversion model to extract the voice of a professional singer from a sample provided by them and turn it into TJ’s voice – originally developed by training on hours of real world recordings. They used the musical notes from that sample and applied them to the digital voice of TJ to make the avatar sing the same notes and with the same rhythm as the original singer.
“It’s a really amazing technology, and the fact that we can do this is phenomenal,” said Cyrus Hogg, an NVIDIA technical program manager, who noted that Toy Jensen’s backing track was developed in collaboration with Jochem van der Saag, a composer and producer who has worked with Michael Bublé and David Foster.
“We have van der Saag composing the song, and he’s gonna also orchestrate it for us,” said Hogg. “So that’s a really great addition to the team. And we’re really excited to have him on board.”
Jensen’s appearances with Lah Yileh Lee and Xinting Lee have been linked by NVIDIA, with the first showing the CEO joining in on the singers’ serenade as they recorded Lady Gaga’s “Always Remember Us This Way.”
Another comprises a follow-up video from the pair, who sang Lady Gaga’s “Hold My Hand,” a song that Huang originally requested.
Both videos quickly went viral after they were shared online, according to NVIDIA.