Multimodal Perception
PerceptionAgent converts raw audio-visual input into a structured percept with transcript, speech emotion, optional visual cues, dialogue history, and input metadata.
Abstract
This paper presents EmpaAva, to our knowledge the first open-source, agentic 3D-avatar empathetic chatbot, which carries empathetic response generation from text-only exchanges into live, face-to-face interaction. Through a video-call-like interface, a user speaks to a 3D digital human that reads their affect from speech and optional vision, and replies with emotional speech, lip-synced facial motion, and photorealistic 3D Gaussian avatar rendering.
At its core, an LLM coordinates a Tri-Agent Architecture in which perception, empathetic response planning, and embodied rendering form a closed loop. A Response Planning layer compiles each LLM reply into an executable multimodal plan, aligning voice, expression, rendering, and avatar behavior with one empathetic intent.
Video Demo
Demonstration of a user speaking with EmpaAva through the booth UI.
Method Overview
Empathetic Reply Planning
ResponseAgent reasons over the user state and emits a structured reply plan that specifies the text, emotion and tone, avatar, voice, background, and supporting evidence.
Affective Speech and Motion
RenderAgent synthesizes emotional speech and predicts frame-level FLAME parameters for jaw, lips, head pose, and expression.
3D Avatar Rendering
A FLAME-to-Gaussian transfer applies motion to a rigged 3D Gaussian avatar and renders the response in a realistic call scene.
Qualitative Avatar Renderings
Case Study Demo
Multi-turn examples show EmpaAva tracking academic stress, self-doubt, interpersonal conflict, and emotional invalidation while grounding each response in emotion, cause, and strategy.