Serenade

Inspiration

Our inspiration for Serenade comes from the high-pressure environment we found ourselves in as college students. There were countless instances, especially during midterms and exams, where we felt overwhelmed, stressed out, and emotionally worn out. In these moments, we found peace in music – from calming classical tracks to the latest pop songs—after all, who hasn't listened to their favorite song at a stressful time? We began to realize the immense therapeutic potential of music and how it could be harnessed towards improving mental health. Meanwhile, we were also becoming deeply intrigued by the capabilities of AI and machine learning. We decided to combine these two interests and thus, Serenade was born.

Anyone struggling with mental health can use Serenade, particularly those without access to costly therapy. Serenade can also supplement therapy for a quick at-home remedy during downtime.

What it does

Serenade uses AI-driven programs to implement its unique music therapy approach. It's designed to take input from a user's audio, effectively analyzing speech prosody to understand the user’s emotions and needs. Additionally, it integrates with Spotify to discover the user's favorite songs, delivering a personalized therapeutic experience. Serenade synergizes these elements to craft a custom-made therapeutic audio tailored to the user's needs.

How we built it

We used Convex and Clerk to sign in to Spotify and get the user’s top recommended song (because it’s probably a song that the user likes). We created an immersive emoji picker to gauge the user’s mood in a natural way. GPT-3.5 combines these signals and instructs MusicGen, an open-source music generation model by Meta, to generate specially-formulated snippets of audio for therapy. We also take a clip from the user’s top song (from YouTube) to make MusicGen generate something that reminds the user of their favorite music.

Challenges we ran into

Implementing all the novel features of our app into a coherent, user-friendly flow was quite the challenge. We had to carefully integrate systems such as GPT-3.5, MusicGen, Replicate hosting, Convex, and Spotify authentication. Each feature had its own unique requirements that needed to be met for seamless functioning.Finding the most effective prompts for our music generation - tailored to specific emotions - was another considerable task: what kind of music do "happy" people want vs "sad" people? What about specific use cases?

Accomplishments that we're proud of

The use of the Convex model exposed us to new techniques for good API architecture.We were able to develop a more structured, reliable, and secure API system for Serenade. Another major milestone was incorporating social OAuth that simplified the sign-in process significantly, offering users a seamless, effortless entry into our service. We learned of the various services that Convex and Hume.ai can provide, and enjoyed integrating these products into our webapp. For Convex, we used its features to provide an authentication method for users and host their profiles. We learned about techniques for good API architecture through the Convex model, as well as how to easily do social OAuth for easy sign-in with providers like Spotify.

What's next for Serenade

Beyond CalHacks, we plan on integrating Hume.ai's streaming facial recognition technology that operates seamlessly in real-time. Utilizing this advanced technology, the ability to read and analyze facial expressions will add another layer of emotional discernment to our platform. It will significantly augment the way we detect nuanced moods and emotions that may not be entirely captured through voice alone. By combining audio and facial cues, Serenade aims to create a more personalized and enriching therapeutic journey for the user.