Chords & Art

A one stop solution for all your music generation needs -- whether you want to write lyrics, compose music or generate cool cover art for your music piece - we can help you with all of it :)

Comment

Inspiration

We're all aware of the tremendous progress made in the space of generative AI recently. We thought our project, Chords & Art will be a good way to catch up with recent advancements made in this area and apply it to the domain of Music to build something cool and hopefully useful.

What it does

Chords & Art is the perfect companion for any music artist whose muse is being uncooperative. It helps address different needs that someone may have with respect to generation of music -- whether it is to write lyrics for a song, compose music or even generate catchy cover art for your music piece.

We think this will definitely help people especially who aren't really music artists but still require music for their projects or apps to quickly create whatever suits their needs.

How we built it

It was a team effort with each of us looking into various modules that interested us and pitching in to build the app.

  1. Nitin Premanand - handled everything related to the development of the frontend
  2. Harshini Krishnamurthy - took care of setting up the backend for the application as well as working on setting up the lyrics generation module along with integrating Replicate's openjourney model for cover art generation.
  3. Shivalika Singh - worked on fine tuning stable diffusion using Dreambooth for creating a custom model for cover art generation along with setting up the music generation module of the project using SymphonyNet

Challenges we ran into:

  1. Figuring out how to fine tune stable diffusion for generating cool cover art with no prior experience using stable diffusion took us a little time. Thankfully this was less intimidating because of lot of useful resources created by Hugging Face such as Dreambooth training.
  2. We were also working with a recent music generation model called SymphonyNet for the first time. Apart from understanding how to use the model, it also required some domain knowledge like what are midi files or sound fonts and how to convert midi files to mp3 files, etc in order to integrate the model with our app.
  3. Figuring out right settings for good lyrics generation for using OpenAI's GPT-3 also took a bit of experimentation.

Accomplishments that we're proud of

  1. Our fine tuned model for music cover art
  2. We were able to finish all the 3 modules we planned

What we learned

We learnt a lot about how to leverage recent state-of-the-models for both unlocking human creativity and solving critical problems.

  1. With respect to backend and UI integration, working with audio files was a new thing for us. SymphonyNet model outputs midi files which we were having trouble playing using our app frontend. We had to figure out how to convert midi to mp3 files on the backend so that the audio can be rendered on the frontend.
  2. Figuring out how to leverage models like GPT-3, Stable Diffusion and SymphonyNet for our use case was the core of what we learnt through this project and also about so many other applications that are feasible now thanks to such generative models.

What's next for Chords & Art

At the moment, our lyrics, cover art and music generation modules are independent. Going forward we wish to integrate them together so that any music artist can create songs or compositions as a whole easily via the app.

Built With

  • css
  • fastapi
  • gpt3
  • huggingface
  • openai
  • python
  • react
  • replicate
  • stablediffusion
  • symphonynet
  • weightsandbiases
Share this project:

Updates