Inspiration
With two of our team members being musicians in the jazz and classical world since middle school, we've seen time and time again an overemphasis on enforcing a standardized approach toward teaching and assessing all student musicians. Yet we believe that with each student's musical journey being unique, so too must be the process of musical education. Currently, with an explosion of personalized education through tools like Duolingo for language acquisition and Khan Academy for academics, there's a surprising gap for personalized teaching curricula for music.
More specifically, we know that music lessons are not cheap. We wanted to build a product that would allow students to independently assess themselves and receive real-time feedback to improve their skills, especially for students who may not have access to private lessons due to financial or geographical barriers.
What it does
Polyhymnia.ai is an application that allows users to gauge their skills with their choice of musical instrument and be given personalized sheet music based on their skill level so that they can improve at a gradual pace and be given individualized attention. It consists of two parts, first a generative section which constructs sheet music based on a Markov chain of note probabilities and another section which judges how well a student played the music in terms of pitch, rhythm and time.
How we built it
We used Next.js and MongoDB for our front end and database. For the backend, we used a variety of music libraries (Lilypond, Midi, etc) as well as traditional AI libraries like TensorFlow.
Challenges we ran into
Standardization of sound: minor differences in recorded frequency and expected frequency caused a hassle. There was quite a lot of difficulty in developing an algorithm which could systematically and consistently map imperfect audio recordings to musical symbols with the text of music being the chosen method of comparison.
Setting up API routes to communicate with the database and the currently logged-in user. It was the first time building a full-stack application for all three of us.
Accomplishments that we're proud of
A key challenge we faced was developing a judging algorithm in the face of many potential sources of student mistakes, i.e. how do you compare the weights penalties such as a missed note compared to a note that was played in the wrong pitch or wrong rhythm? After deep consideration, one of our teammates realized that the Needleman-Wunsch algorithm he learned in his algorithms class for sequence alignment in DNA was surprisingly applicable for finding a method to standardize comparisons between the correct sheet music and a student's submission. We're particularly grateful to have the opportunity to apply what was previously theory in class to real world problems; seems like all those hours spent studying for class are useful after all!
We were also pleasantly surprised that our initial approach of infinite music generation with random walks through Markov chains; given some loose constraints with the key, this method allowed for unlimited possibilities for material for the student to experiment, play and assess with.
What we learned
Since it was our first time building an application, we learned frontend with Next.js and backend development with Flask. We also learned about mathematical applications of Markov Chains and experimented with random walks. Perhaps most interestingly, spending hours translating and fiddling with the audio to notes pipeline has left us with not just a newfound appreciation for hashmaps, but also the unexpected repercussion of an ingrained map in our minds between notes and their frequencies.
What's next for Polyhymnia.ai
We aim to expand Polyhymnia.ai beyond its current form, with the current short four-measure-token being just a proof of concept for future projects which may be longer, more complex, and able to handle harmonization. We hope to scale Polyhymnia.ai to an overall teaching assistant, able to generate personalized lesson plans in scales, exercises, and songs.
Log in or sign up for Devpost to join the conversation.