Drolg

AI DJ curates parties using real-time video analysis.

Comment

Inspiration

Parties just haven't been the same since COVID hit. I mean, think back to 2019. House parties and frat houses were insane - people everywhere, everyone going wild. Now? It's like everyone forgot how to have fun.

That's why I made Drolg. I wanted to bring back that party energy, but make it even better.

What it does

  1. Before the party, everyone adds songs they want to hear to a shared playlist.
  2. The AI DJ looks at these songs and makes a plan for the best order to play them, based on how long the party is and the vibe the host wants.
  3. During the party, the AI watches the crowd through a video feed. It changes up the music to keep the mood just right, based on how people are reacting.

How we built it

Step 1: Collaborative Playlist Building

  • Frontend: HTML, CSS, JavaScript, Bootstrap
  • Backend: Flask, SQLAlchemy, SQLite database
  • Spotify API integration via Spotipy

Step 2: AI DJ Analysis

  • Music Recommendation System analyzes songs
  • RAG (Real-time Atmosphere Generator) plans ideal flow
  • Hosted on Flask API

Step 3: Real-Time Party Analysis

  • ESP32 cameras stream video via WebSockets
  • YOLOv9 model (custom-trained) for object detection
  • BMF Framework processes video for sentiment analysis
  • Deep SORT tracks crowd movements
  • Body language and facial expression analysis
  • Playlist updated based on crowd mood and host preferences

Challenges we ran into

One major challenge was the lack of high-quality training data, which limited the performance of the AI model. Creating synthetic data took a lot of time and effort. Additionally, I struggled to find a timely solution for handling overlapping individuals in the crowd, which forced me to ignore those with overlap and negatively impacted the accuracy of the crowd sentiment analysis. Another significant challenge was not being able to test the system in a real party setting, which made it difficult to properly evaluate its performance.

Accomplishments that we're proud of

Building Drolg alone in just a week of nights after work, I hit some big goals and learned a lot:

  • Real-time video processing with ByteDance BMF for the AI DJ
  • ESP32 camera integration, my first time working with hardware
  • Custom AI training data for understanding party scenes
  • A web app that's almost ready for real-world use
  • A new way to analyze crowd mood that I couldn't find anywhere else

What we learned

  • The ins and outs of API integration
  • How to deploy a fully functional web app using Flask framework
  • Real-time video streaming using ESP32 cameras - trickier than it sounds
  • Combining multiple computer vision models to create a single unique model
  • Creating customized synthetic data for training computer vision AI

What's next for Drolg

The next step for Drolg is to improve and train the AI model to recognize more subtle cues and enhance mood prediction accuracy for the playlist, as there was limited time to work on this aspect initially. Better training data could make a significant difference. Improving the front-end is also a priority, ideally turning it into a user-friendly PWA application. I want to take Drolg from a proof-of-concept to a viable MVP and share it with people to see if it's something they'd be interested in using.

Built With

Share this project:

Updates