Creating Immersive Experiences with Digital Humans, Avatars, and AI

The intersection of AI, motion capture, and digital humans is critical for creating believable interactive avatars. Inference and machine learning are important for optimizing rendering pipelines and creating fast content creation. The advice for those interested in the field is to focus on shipping rather than perfection.

• AI, motion capture, and digital humans are critical for creating believable interactive avatars
• Inference and machine learning are important for optimizing rendering pipelines and creating fast content creation
• Shipping is more important than perfection

The speaker shared an example of needing to recapture a dialogue or interaction with an actor who did not have scheduled time due to their availability. Synthesizing data that is correlated to the motion capture was necessary to create a believable result for the digital avatar.

Eric Rachlin, Body Labs’ Co-founder, and Natalya Tatarchuk, Distinguished Technical Fellow and Chief Architect at Unity, join Vivek Muppalla, Scale's Director of Synthetic Systems, to discuss the current and future state of digital humans, avatars, and AI. Rachlin and Tatarchuk describe how to create more realistic digital humans with methods including traditional motion capture, motionless capture, and recent machine learning techniques like Instant NeRF.They also discuss supplementing models with synthetic data for improved performance. They outline how text prompting will evolve from state-of-the-art diffusion models to more powerful digital creation tools, allowing non-developers to create and animate realistic digital humans and to control and customize every element of digital environments to create more immersive experiences.The panel addresses Metaverse standards such as Universal Scene Description and other machine learning methods to provide an "intelligent glue" to help provide smooth cross-application interactivity and realistic object interactions for avatars. Rachlin and Tatarchuk share various applications for digital humans, including virtual try-on, inclusive and diverse digital assistants, and more realistic digital avatars for gaming. They address mitigating bias, distinguishing between real and fake identity, and protecting privacy. They end with thoughtful advice for anyone who wants to get started in the field.