Al.Ai

Inspiration

One of our group members, Sivaditya, once came across an elderly lady who was heading to the train station from the mall, but she seemed lost, bouncing back and forth. Sivaditya went up to the lady to offer help while crossing the street. He asked her where she wanted to go. She said she wanted to go home. Sivaditya asked, “Where is home?” She replied, “Brazil.” Confused, Sivaditya continued inquiring about people he could contact and how to help her get home. After nearly an hour of conversation filled with her uncertainty and distress, he suspected that the lady likely suffered from a form of sustained memory loss, such as dementia.

Reflecting on Sivaditya’s anecdote, we realized that he could have been better able to assist if:

He had more guidance about the lady’s condition,
And a way of connecting her back to her family/support system.

After doing some research, we learned some unfortunate facts:

People with Alzheimer’s can experience sundowning (a phenomenon that occurs during the evening hours, where a patient suffering from such a cognitive impairment tends to become more confused and agitated), which leads to wandering
Around 60% of people with dementia are at risk for wandering (Alzheimer’s Association)
When a person with dementia goes missing:
- 72% were found alive the next day
- 17.5% after 2 days
- Less than 5.5% after 3+ days

A lot of these situations are tragic, and it can make close ones think, “I wish I were there at the right time.”

Seeing this, we wanted to make a product that would improve such circumstances. So we came up with the idea of a voice-first AI companion that helps people with cognitive impairments stay safe, calm, and connected while empowering caregivers with contextual, real-time insights. We wanted something that would allow the user, or even bystanders (like Sivaditya), to help people with dementia return to safety. We wanted to empower caregivers to be there at the right time.

What it does

The fundamental principle of our application is to use an AI agent to bridge the communication and attention gaps that naturally show up when caring for Alzheimer’s or dementia patients to keep them safe. This product enables efficient communication with an AI agent that prompts bystanders to contact certain people and provides caregivers with the patient’s location information, so they can share the most relevant facts with people who can help those who can’t help themselves.

These are the main functionalities of our program:

Patients would wear a wristlet with a QR code, which would come with the product and would get linked to their accounts, and they could chat with our voice AI agent. The agent should seek to calm them down, give answers to questions, and most importantly, prioritize, through gentle affirmations and speech, that these patients should seek help among their “caregivers” or with the people around
The public bystander would scan the QR code on the patient’s wristlet and would be taken to a website with no sign-in necessary, and could access the first name of this patient. The chatbot should not divulge specific information of this person, but should answer questions related to this person if they are required to help. That said, some guardrails prevent the divulging of their address, full name, or medical history. They communicate with the AI chatbot agent, which also leads to notifying the caregivers.
Caregivers are the people chosen to have broader access to the user’s account through a sign-up and authorization process on the patient’s end. They have a broader suite of information, including location trends, identifying whether this patient has left a “caregiver” set radius around their home, etc.

How we built it

We built Al.Ai using a combination of different tools and interfaces.

We used Flutter as a way to have a single codebase that concurrently implements our application on a web and mobile platform (made possible due to the ability for native compilation). This allowed us to have a very quick and clean frontend development phase.

We used FastAPI in our backend mainly for its ability to leverage asynchronous programming for high performance, specifically the I/O-bound operations.

We also used the combination of the Elevenlabs API and OpenAI API to create our realistic, real-time voice-first AI companion agent. Specifically, we used the Elevenlabs API for the humanized AI vocal conversations and interactions. Then, we used the OpenAI API to use a model for processing texts (whether it was messages or the speech-to-text transcript), and it generated the conversational language used. This model is post-trained on previous research and verified articles to prioritize truth and good speech habits when having these conversations, allowing for proper guidance to users.

Challenges we ran into

Originally, we wanted to provide a way for a person feeling disoriented to be able to request a call to a caretaker to receive advice from a more familiar person. Additionally, we wanted to provide a way for bystanders to be able to scan a QR code that a patient is wearing and have the ability to contact a caretaker as well.

The way that we planned to implement this was to use the ElevenLabs API for a realistic and comforting voice for the user. When the user requests to talk to a caretaker through the ElevenLabs API, a call should be forwarded to the caretaker. However, general phone numbers do not easily take automated calls from the ElevenLabs API. Therefore, we wanted to use Twilio to interface with the ElevenLabs API, specifically, ElevenLabs sends a call request to a Twilio phone number which then can call the designated caregiver, ideally without issues.

When actually implementing the interaction between these two APIs, we were able to call the Twilio number from some mobile device and receive the call on another device (caretaker cell). Additionally, we could call the ElevenLabs voice agent from some cell device and interact with it over the phone. Through the dashboards of both ElevenLabs and Twilio, we were unable to configure the settings such that the ElevenLabs API could call another device (caretaker cell). To address this challenge, we simplified our app design to allow users to contact caretakers through notifications pushed to them on the app.

Accomplishments that we're proud of

We’re especially proud of the fact that we were able to make an application in flutter, which can be accessed which is both friendly to access on PC and as an app on mobile devices. Additionally, we’re proud that we were able to incorporate the ElevenLabs API into our flutter implementation of the project. Getting these voice agents to work was not an easy task and we’re glad to have implemented it. Finally, getting the QR code generation for patient info as well as scanning the QR code on the bystander side of the app and having a specialized conversation about the specific person was a feature that we were very proud to implement.

What we learned

We learned the importance of maintaining a clean and modular codebase. This meant we also learned that there needs to be effective communication between teammates, especially when developing the frontend and backend separately.

What's next for Al.Ai

As we discussed in the challenges section that we explained before, we had to compromise on the original vision of our product which included the ability for patients of the app to request to contact their caretakers with voice call through the AI agent as well as the bystanders to interact with the caretakers through voice call as well. In a future iteration of the Al-Ai application, we would certainly try to more thoroughly research how the Twilio and the Eleven Labs APIs work and how they would interact with one another in the hopes of adding the originally intended voice feature back.

Since one of the primary purposes of this app is enabling patients to interact with the comforting, reassuring AI voice agent, it is crucially important that they always have access to the voice agent. With the app at the moment, the phone must be turned on and the dementia patient must open the phone and sign in for any of the patient based functionality to work. Undoubtedly, many people with Alzheimer's may not be in a state to carry out the steps listed above. The functionality is limited to only execute the app when the phone is powered on, which poses a significant problem because phones turn off with normal frequency, but the user needs to be able to access the voice agent at all times. Ideally, finding a way to maintain not only an always on display mode for the app, but also having it run in the background at all times certainly provides these patients with dementia. Additionally, we could try our best to optimize the code and display as few features on the always on display mode to save power consumption, battery life, and let the app provide unlimited guidance to the patients for longer periods of time.

Finally, a main component of our product involves the wearable that the patient with dementia wears. Our next step involves finding an optimal material that would make for a comfortable wearable, modeling this wearable, and fabricating it using whatever method is suitable for that material. We also need a way to ensure that the QR code displayed using the wearable does not easily fade away or get worn away through use.

Overall, we hope to proceed with Al-Ai by aligning the product closer to how we envision patients with dementia and bystanders interacting with the application and building a prototype of the wearable itself.