Soundpair

Inspiration

Soundpair.ai was inspired by one of our teammates, Michael Chang, and his deep passion for music. To fulfill the final requirement for his Yale Computing and the Arts B.A., Michael submitted a basic implementation of a live accompanist that listens to audio input and plays paired backing chords in real-time.

As AI’s applications, particularly in generative music, exploded we were inspired to create Soundpair. We hoped that we could build out an MVP to validate a valuable tool for independent musicians to create full musical compositions without having to build every instrument from scratch.

What it does

Soundpair generates accompanying instrumental tracks perfectly tailored to your vocals.

How we built it

We used:

React and Next.js to call our hugging face model from the front-end
Hume.ai to measure emotional quality of vocal recordings
Groq to pass the emotion tags into a prompt refiner that we pass on to fb-music-melody-large
Remotion.dev, a front-end library that helps build video & audio-editing timeline interfaces
fb-music-melody-large hugging face model to assist with guiding instrumental generations

Challenges we ran into

Technical challenges included: 1) Identifying emotional qualities from vocals 2) Taking those tags and refining our instrumental generation prompt 3) Creating instrument-specific generations that matched the beat and tone of the user recordings 4) Formatting the output as a DAW-compatible (ex. Logic, Audacity, etc) file that users could then further edit and customize in their audio editor of choice 5) Wrapping that into a clean and familiar audio timeline editor interface

Accomplishments that we're proud of

We solved and implemented everything except #4, but what we learned through the hackathon will help us build out export formats easily and launch this tool out for indie musicians to try out by Wednesday of this week.