LumiCue - Project Documentation

Inspiration

At the end of her life, finding meaningful connections with my Grandmother was difficult, and the silence between us was painful for both of us. We both sought out connection, but it was often elusive as she found it harder to gather her thoughts and her memories.

It turns out, we were not alone. In America, 12 million caregivers provide unpaid care for those with dementia at home (Alzheimer's Association, 2024). And that number is expected to grow. By 2050, the number of people living with dementia could exceed 152 million (Alzheimer's Association, 2024).

The inspiration for LumiCue came from the recognition that a tool that Qloo, a tool that makes personalized experiences possible with the power of AI, could help bridge the communication gap.

What it does

LumiCue is an AI-powered app that uses Qloo's Taste AI to help caregivers and those they care for spark meaningful conversations and have meaningful experiences. By combining cultural intelligence with personalized recommendations, LumiCue provides a simple-to-use tool for caregivers to help spark meaningful moments and bridge the silence.

The LumiCue dashboard provides a theme of the day and four carefully curated content cards, each designed to engage different senses and encourage interaction between the caregiver and the person they care for.

Music Card: Offers culturally relevant songs with YouTube integration that only allows Creative Commons licensed music. Each selection is tailored to the person with memory loss and loaded directly into the app.
Recipe Card: Features safe, microwave-only recipes that connect to cultural traditions—perfect for sparking memories of family cooking without safety concerns
Photo of the Day Card: Presents culturally appropriate images tailored to the person's background and interests.
Nostalgia News Card: Delivers LLM-generated personalized historical and cultural stories that create meaningful bridges to the past

In addition, an easy-to-use feedback mechanism allows caregivers to note what the person they care for likes and dislikes, allowing the daily suggestions to improve over time.

How I built it

Frontend Architecture

I built LumiCue using React with a component-based architecture that prioritizes simplicity, accessibility, and ease of use. The frontend features:

Dashboard Interface: Clean, intuitive grid layout
Detail Views: Dedicated pages for each content type with embedded media and conversation starters, so the user does not need to leave the app to engage.
Profile Management: Secure profile system that requires minimal information, uses first name or nickname only, and puts privacy first.
Fallback System: Ensures the app works even when APIs are unavailable.

Backend: 8-Agent Pipeline

Behind the simple interface is a sophisticated eight-agent pipeline that takes the profile and feedback information in anonymized form,

Information Consolidator: Verifies anonymized patient data and selects the theme of the day that will be used to guide selections
Photo Analysis: Uses Google Vision for initial processing of theme-appropriate images (In future state, this will include photos that are unique to the individual for better personalization).
Qloo's Taste AI: Connects the anonymized profile and data gathered above and uses it to suggest deep personal preferences that serve as grounding data for our Gemini Flash LLM and content selection
Music Curation: Fetches Creative Commons songs with the YouTube API
Recipe Selection: Chooses microwave-safe comfort foods from our curated database
Photo Description: Creates culturally relevant stories around selected images
Nostalgia News Generator: Weaves everything into personalized stories using Qloo's preference grounding
Dashboard Synthesizer: Assembles the complete experience

Technology Stack

Frontend: React, Tailwind CSS, Axios
AI Integration: Google Gemini AI, Qloo Cultural Intelligence API, YouTube Data API, Google Vision AI
Architecture: Google ADK Framework
Safety Features: PII compliance, anonymized data processing
Deployment: Cloud Run, FastAPI, Firebase Hosting

Challenges I ran into

Technical Challenges

API Integration Complexity: Orchestrating multiple AI services (Gemini, Qloo, YouTube, Google Vision) required careful error handling and fallback strategies
Data Privacy: Ensuring complete PII compliance while maintaining personalization quality

Domain-Specific Challenges

Safety First Design: Every feature had to be evaluated through the lens of dementia care safety (microwave-only recipes, safe conversation starters)
Caregiver Usability: Creating an interface simple enough for stressed caregivers while sophisticated enough to deliver meaningful results and ensuring it could work on screens of any size
Memory Care Appropriateness: Ensuring all content would be engaging but not overwhelming for various stages of memory loss

AI/LLM Integration Challenges

Grounding LLMs with Cultural Data: Learning to effectively use Qloo's taste intelligence to ground Gemini's responses in cultural truth rather than generic outputs
Maintaining Consistency: Ensuring the eight-agent pipeline produces coherent, themed content that works together and avoids cultural bias
Fallback Strategies: Building systems that gracefully handle API failures without breaking the user experience

Accomplishments that I'm proud of

Social Impact

Addressing Real Need: Created a solution for 12+ million home caregivers who seek out more meaningful connections with those with memory loss and communication issues. This is the app I wish I had with my Grandma.
Safety-First Design: Every feature prioritizes the safety and dignity of those with memory loss.
Cultural Sensitivity: Built a system that respects and celebrates diverse cultural backgrounds
Accessibility Focus: Designed for users with varying technical abilities and communication challenges

Innovation

Novel Use Case: Used cultural AI for memory care and caregiver support
Multi-Modal Integration: Successfully combined music, recipes, photos, and stories into a cohesive experience that allows users to engage their senses
Privacy-Preserving Personalization: Achieved personalization while maintaining complete data anonymization

What I learned

About AI Integration

Cultural Grounding is Powerful: Qloo's taste intelligence transforms generic LLM outputs into culturally relevant, personally meaningful content
Multi-Agent Orchestration: Complex AI systems require careful coordination, error handling, and fallback strategies

About User-Centered Design

Simplicity is Complex: Creating an interface simple enough for memory care required sophisticated backend systems
Feedback Loops: User preference tracking and feedback systems are crucial for improving AI recommendations over time

What's next for LumiCue

Immediate Roadmap

Enhanced Personalization: Expand the feedback system to create more sophisticated preference learning
Content Expansion: Add more content categories (art, nature sounds, gentle exercises) and expand available offerings (currently limited to public domain and Creative Commons licensing)
Better Use of Google Vision AI: Add the ability for caregivers to upload their own photos to better use Google Vision AI and allow more personalization and meaningful connections. Currently, the demo version relies on preloaded images sourced from Creative Commons.

Long-term Vision

While this hackathon project demonstrates the core concept, I envision LumiCue evolving into a comprehensive platform that could support millions of caregivers with one, sole purpose: personalizing experiences for those who may not be able to easily communicate preferences and memories on their own.