HelloMind.ai

Real time emotional sentiment analysis in video, voice and text for improved therapy session insights for both AI and/or real-world therapists. Both users and therapists can track patient growth.

Comment

Inspiration

I'm motivated to create more empathetic human-machine interfaces and experiences. For AI to become truly useful as a therapy bot, it needs to interact with humans in a more holistic way, taking the emotional content from human communication into account. This aims to explore how people express their emotions visually, verbally and through text while providing real-time insights to both users and their therapists, clinical and/or AI to track progress over time. This can help an individual and a therapist with deeper reflection of their personal practices while providing clear insights into triggers and areas of growth. It is also designed to give the user/patient more autonomy over how they want to interact with a therapist and what stylistic or treatment therapies will work best for them. I think it's essential to provide choices for both patients and clinicians so they can tailor their use of technology to best serve their needs and comfort levels.

Lastly, it is meant to be inclusive of clinicians, who can also train and oversee models that will work best for the patients based on their own therapy style and patient notes. Therapists can customize their own therapy bots or use a premade AI to support patients in between sessions. This can both bring down the cost of treatment for patients as well as increase potential revenue time for clinicians, while personalizing supervised therapy for individuals. The analytics of patient growth and auto-suggestions for growth plans can help clinicians take a deeper look into their overall metrics for their own practice goals.

What it does

The app provides real time emotional insights to therapy sessions that can be provided either through clinical therapists or through AI therapists, that users can customize according to both the style and likeness they feel most comfortable with. It also let's therapists customize AI avatars through adding their own likeness, voice and therapy style to them for interim sessions between regular therapist/patient meetings. Either patient or therapist avatars range from serious clinical to more playful bots using the theories of Socrates, Marcus Aurelias, Confucious, Lao Tzu and more.

For patients, the dashboard includes uploading medical histories, goal setting and a wellness journal that tracks everything from water and food intake to daily summaries of events and feelings. Daily emails with inspirational affirmations and goal reminders are sent to help users prioritize their own health as well as track their progress, so they can have greater insights as to what is working or not working for them. For those struggling with addiction, there is a daily sobriety counter that sends them positive messages each day. There is also a resource library with role-playing potential that users can define based on where they feel they could benefit from support.

For clinicians, it provides scheduling options, daily, weekly and monthly summary reports, crisis notifications, the ability to use a premade AI avatar or create their own clone or creative avatar. If patients choose to share their journals and AI sessions, it sends session notes with insights and summaries and makes autosuggestions for next line therapies that therapists can add to progress plans or reject based on their clinical expertise. It is a clean interface that allows clinicians to glance quickly at session notes and patient information so they can review each time prior to sessions. They also have a resource library and role-playing activities they can use for either their own practice or to assign to patients as suits.

How we built it

I used several therapy data sets and a multimodal facial and voice emotion detection approach from Hugging Face to train the Tavus avatars as well for the generation of line by line speech and session summary reports. I also used Stripe, Eleven Labs, Pica, Canva, SupaBase and of course- BOLT!

Challenges we ran into

Kept getting told we were out of credits even though had paid for Pro plans- within minutes (literally) our credits evaporated. This happened each time we upgraded so we finally contacted customer service and did not hear back. As such, we would have built out many more features and further fine-tuned the avatars.

Accomplishments that we're proud of

Despite being out of time and credits, the app itself has a lot of functionality. I really hope it will help both individuals and clinicians in better serving people to grow holistically. I also hope it elevates the mindfulness of both healers and those working on healing.

What we learned

How far BOLT and Vibe Coding has come! Only a few months ago- this app likely would have not been able to have been built as quickly. We also learned that we can auto-train AI models with BOLT via Anthropic and use a wide range of datasets to improve overall end product user interfaces and experiences.

What's next for HelloMind.ai

We'll refine the app in the upcoming weeks to better fine tune Tavus avatars, improve UI/UX (wanted to add a Tavus avatar welcoming visitors to the landing page) and start beta testing!

Built With

Share this project:

Updates