AI Chat Avatar
Real-time conversational sales assistant with speech, viseme-driven lip sync, and character animation
Personal Project
2023
Solo Developer / Technical Artist
Context
A real-time AI sales assistant prototype built in Unreal Engine 5, integrating speech recognition, OpenAI-generated responses, Azure text-to-speech, viseme-driven lip sync, and character locomotion.
My Contribution
Designed and implemented the complete prototype independently, connecting speech input, conversational AI, synthesized voice, facial animation, and locomotion into a responsive UE5 character pipeline.
Impact / Outcome
Demonstrated a complete real-time conversational character workflow that unified cloud AI services with UE5 facial and body animation in a solo prototype.
Outcome Summary
AI Chat Avatar
AI Chat Avatar is a real-time sales assistant prototype I built independently in Unreal Engine 5 in 2023. The project connects conversational AI services with a fully animated digital character, turning spoken user input into a synthesized, lip-synced response inside UE5.
Table of Contents
Overview
The goal was to prototype a believable interactive avatar that could support a sales-assistant experience. I owned the complete implementation, including speech recognition, OpenAI API integration, synthesized voice output, facial animation, and body locomotion.
The result was a unified real-time pipeline rather than a collection of disconnected service demos. Each stage passes usable timing and state data into the next so the character can listen, respond, speak, and move as a coherent experience.
End-to-End Conversation Pipeline
The interaction follows a complete voice-driven loop:
- Capture the user's voice and convert it to text through speech recognition.
- Send the transcription to the OpenAI API and receive a generated response.
- Pass the response text to Azure AI Speech for voice synthesis.
- Play the generated speech while applying the accompanying viseme data to the character's face.
- Coordinate the speaking state with the character's body animation and locomotion in UE5.
This integration required coordinating asynchronous cloud-service responses with real-time character playback inside Unreal Engine.
Viseme-Driven Lip Sync
Azure AI Speech provides viseme events alongside the synthesized audio. I used that timing data to drive the avatar's facial animation, aligning mouth shapes with the generated voice instead of relying on generic talking animation.
Keeping the audio and viseme playback synchronized was central to making the response feel like it came from the character rather than from a separate voice system.
Character Animation
Beyond facial performance, I integrated character locomotion through an Unreal Engine Animation Blueprint. The animation system coordinates conversational behavior with the character's movement state, creating a more complete interactive avatar presentation.
