aditi

Designing for Accessibility using AI 3D-Model

How I leveraged user-centered design to create an engaging AI-driven speech therapy app for children with Apraxia, improving accessibility.

Overview

Fumo is an AI-driven speech therapy application to help children with Childhood Apraxia of Speech (CAS). The app uses NVIDIA Omniverse Audio2Face to create responsive avatars that provides real-time feedback during speech exercises.

Results

From V1 to V2, my teammates and I redesigned Fumo. Through research, ideation, and countless iterations of prototyping, we transformed the application into a more accessible and engaging experience. Between two phases of usability testing we saw a 86% increase in user engagement.

My contributions

Designing an end-to-end mobile experience

Visual Design, UI Design

Prototyping, User Testing

What is Childhood Apraxia of Speech (CAS)?

Key Challenges

✅ Introducing the NVIDIA OMNIVERSE 3D MODEL

Our research indicated that visual cues could significantly enhance speech therapy outcomes. We implemented a 3D model as the core interactive element, designed to visually demonstrate proper mouth movements and articulation patterns for children with apraxia.

We developed a high-fidelity avatar using meta human character technology, carefully designed to appear friendly and approachable. Throughout the design process, we strategically avoided the "uncanny valley" effect by balancing realism with stylized features that children would find engaging rather than unsettling.

The development of 3D model

The 3D model displaying different emotions and prosidies

Voice Integration - NVIDIA Audio2Face

Design Solutions

Building upon our research findings and addressing the identified challenges, we developed a structured, four-step approach to speech therapy practice. This solution directly responds to the needs uncovered in our interviews and observations, particularly addressing engagement issues and the need for consistent practice routines. The systematic progression ensures both therapeutic effectiveness and sustained user motivation.

Four-Step Practice Framework:

Step 1

Warm-up Words: Users begin with foundational practice

Step 2

Emotional Expression: Practice words with 4 different emotions

Step 3

Story Integration: Spontaneous learning through storytelling

Step 4

Reward System: Earn badges for progress

Accessibility First Approach

These standards align with WCAG 2.1 guidelines while specifically addressing the unique needs of children with speech and motor planning difficulties.

In our initial release, we implemented fundamental web accessibility features based on early research and development priorities. As I delved deeper into conversations with Speech-Language Pathologists (SLP) and conducted additional user research, I uncovered more sophisticated accessibility requirements.

Our Version 1 is focused on essential features like simple navigation patterns, clear visual hierarchy, and basic visual feedback systems (fig: UI with labels). Through continued engagement with the SLP community and observing real-world usage, we identified several advanced accessibility considerations.

These upcoming enhancements include customizable text-to-speech speeds, support for alternative input methods, flexible exercise controls, and adaptable font settings. These improvements, planned for future versions, demonstrate our commitment to evolving the platform based on expert input and user needs.

(fig: UI with labels)

Old Solution
(reduced video quality due to file size limitations)

Current Solution
(reduced video quality due to file size limitations)

Technical Innovation

Children with Childhood Apraxia of Speech (CAS) face significant barriers to receiving optimal therapy. Current research indicates children typically receive therapy only once per week, while the ideal frequency is 4-5 sessions weekly. Traditional therapy methods, though effective, present challenges in maintaining engagement and accessibility, while existing digital solutions lack the necessary personalization and immediate feedback for effective practice.