Ink-Keys

Our project turns drawings into instruments using a webcam and hand tracking, offering a portable, affordable way to make music. Accessible for kids, beginners, and creatives alike!

Comment

Inspiration

The idea for Ink-Keys came from the desire to make music accessible, affordable, and portable for everyone. The hefty price tag on pianos often makes music education inaccessible, discouraging beginners. Ink-Keys combines a child's love of drawing with the joy of playing music, fostering creativity and learning through fun. We wanted to eliminate the barriers by allowing users to draw their own piano and bring it to life anywhere, anytime.

What it does

  • Uses a webcam, phone or any camera available to detect a printed or drawn piano on paper
  • Allows users to learn, create and play around with a piano system, with realtime feedback

How we built it

  • Used ReactJS and TailwindCSS to form an easy-to-follow hierarchy showing the many different components that are used and reused around the website
  • Used Flask to create a SocketIO stream, updating key presses in real time
  • Used OpenCV to access the webcam and detect keys
  • The keys were detected by finding all the contours on the video and filtering by shape and aspect ratio to detect just the keys
  • TensorFlow and MediaPipe to track hands and fingertips for easy interaction with the virtual keyboard
  • Python for backend logic/integration in key detection, hand tracking, and input recognition

Challenges we ran into

  • Integrating the frontend with the backend while keeping the real-time functionality necessary for the project. Our intial system to constantly ping the backend was replaced with the efficiency of a Flask SocketIO stream
  • Noise issues in detecting the keyboard keys and separating them based on the lines drawn, which neeeded a combination of overlap detection, rectangle detection and shape filters to solve the issue

Accomplishments that we're proud of

  • Getting to make our keyboard fully interactive and responsive
  • Website’s display reflecting the keys being pressed
  • The use of OpenCV, Tensorflow, and Mediapipe for our hand tracking and piano detection/key separation mechanisms

What we learned

  • We learned how to use openCV, mediapipe, and tensor flow for object detection
  • We learned how to use Flask streamIO to connect the back and frontends

What's next for Ink-Keys

  • Add a community based feature, where users are allowed to save their info to allow people to play and learn them
  • Precise hand tracking to detect whether a find is pressed down or not for a key press
  • Adapt into an App to create a user-friendly UI for mobile users

Built With

Share this project:

Updates