HeatherAI

• 

  System Design

Inspiration

Imagine slowly forgetting the people and moments that made your life meaningful. More than 10 million people are diagnosed with dementia every year, and that number continues to grow. It's heartbreaking, especially for families experiencing dementia for the first time. With HeatherAI, I wanted to bridge the gap so families don't have to bear such a heavy burden alone, while ensuring patients still receive meaningful companionship and care.

What it does

HeatherAI is a Voice AI Agent designed to help patients in elder care, specifically those with dementia, reconnect with their memories through the simple act of speaking with their younger self.

Before diagnosis: HeatherAI helps you plan and prepare by recording your life story through natural text-to-speech or speech-to-text conversations. It learns your history as you go and creates a voice clone that sounds just like you.

After diagnosis: When dementia sets in, you can open up and converse with HeatherAI. You might or might not realize it's your younger self speaking—but either way, it provides comfort, familiarity, and connection to your past.

Core Features:

• Voice Cloning - Preserves your voice before memory loss begins
• Memory Lane - AI-powered recall of personal history and cherished moments
• Smart Tools - Emergency calling, safety monitoring, and dynamic agentic tool creation for personalized needs

How we built it

HeatherAI is built on a robust stack of cutting-edge technologies:

• ChromaDB - Vector database for semantic memory storage and retrieval
• LiveKit Agents - Voice AI framework with advanced VAD and turn detection
• ElevenLabs - Voice cloning and natural text-to-speech
• Twilio - Emergency calling capabilities
• Mailjet - Email notifications for caregivers
• Groq - High-speed LLM inference for natural conversations

Challenges we ran into

Multi-voice switching: I wanted users to hear from multiple loved ones—not just their own voice, but also their children, spouse, or friends. While I successfully created two voice clones with ElevenLabs, I couldn't figure out how to dynamically switch between different voice models during runtime.

Hallucinations and background noise: The agent initially struggled with accuracy and was sensitive to background sounds. I solved this by refining prompts to be clear and concise, providing specific examples, and adjusting the temperature parameter. I also attended LiveKit's workshop and learned about their multilingual semantic turn detector, which layers on top of voice activity detection to filter out background noise.

Connectivity issues: WiFi challenges at the hackathon made testing difficult, but persistence paid off!

Accomplishments that we're proud of

✨ Getting it working end-to-end - Connecting everything to a LiveKit room and hearing the voice model respond for the first time was magical

🛠️ Dynamic tool calling - The agent can intelligently select from a variety of tools during conversation, making memory recall seamless with ChromaDB integration

🎭 Voice quality and empathy - Testing different TTS models and seeing the voice agent display genuine empathy and understanding for users

🤖 Agentic tool creation - The most ambitious feature: when a patient needs something not covered by existing tools (like checking weather or creating reminders), our LLM dynamically creates new tools on-the-fly. This means HeatherAI can adapt to each patient's unique needs without requiring code changes.

What we learned

I discovered a passion for voice AI and its incredible potential to make a real difference in people's lives. I learned about LiveKit's system design—how voice models work from VAD to STT and beyond. I gained hands-on experience with ChromaDB for building RAG pipelines with vector databases. Most importantly, I learned that technology can preserve what makes us human: our memories, our voices, and our stories.

What's next for HeatherAI

• Multi-voice support - Enable switching between different loved ones' voices (children, spouse, friends)
• Caregiver dashboard - Real-time insights and alerts for family members and medical staff
• Mobile app - Make HeatherAI accessible anywhere, anytime
• Advanced memory triggers - Use photos, music, and video clips to enhance memory recall
• Clinical trials - Partner with elder care facilities to validate HeatherAI's impact
• Multilingual support - Help dementia patients worldwide, regardless of language
• Integration with medical records - Seamless connection with healthcare providers for coordinated care

Built With

• python