Lyrics2Mood

Make the music follow your singing lead, like it's always been supposed to be,

Comment

Inspiration

When doing improvisations, singers can lead their performance with their voice, playing an instrument or interacting with their motion. What they can't do is inspire and lead the performance with their words and the sentiment behind it. At least, they couldn't so far.

What it does

Our demo takes as input the voice of a singer, it extracts the lyrics using a Speech to Text, and then it performs Sentiment Analysis to understand what's the value of valence, i.e., positivity or negativity, expressed by the lyrics. From here, the applications are endless, from getting the music background follow your emotional lead, to change a guitarist effects with the sentiment you convey, up to record short demos of your music ideas, that the system will integrate with a proper musical arrangement.

How we built it

We used Processing to have real time recording, processing and contacting the Microsoft Cognitive Services for Speech To Text and Sentiment Analysis. Processing was also useful for interacting with an Eventide H9 pedal effect and tell it to switch among presets and values of parameters. Lastly, we used Python to develop a web-based APIs that receives a solo singing track in input and returns a properly arranged audio track (with singing, of course).

Challenges we ran into

Real-time development was quite challenging, probably we should have focused to something more artistic and offline; dealing with latencys from 2 API calls was not easy.

Accomplishments that we're proud of

Working in five is harder than working in 4; we struggled with repos - but succeeded. The kernel of our tool has a lot of artistic applications and we develop a clear and quite clean code -we think.

What we learned

Sometimes a good idea can be simple, and sometimes is better to properly develop a single simple idea that trying to extend it too much without a clear user scenario. We learned that it's always good to record progress during the project, because under Murphy's law, it's clear that the APIs will stop to work exactly when you'll decide to "wrap up and prepare the presentation". We also learned that studio1 is amazing, but its floors are not the best ones to sleep on.

What's next for Lyrics2Mood

The musical model is not complex as we would have liked to. We would like to extend it with algorithmic composition and music generation with neural networks. Also, we intend to further explore the application scenarios, from automatic composition of video soundtracks based on the dialogues to storytellers like theatrical monologues automatically triggered by the workds of the storytellers.

Built With

Share this project:

Updates