vike | Devpost

Editing the keyboard config
About Vike
Mouse detection with CV
Virtual keyboard with custom configs

Inspiration

All of us are students who have been blessed with the fortunate opportunities given by modern technology. However, not everyone around the world has equal access to these gadgets that we take for granted. Realizing the need to address this issue, our team worked to create a budget friendly alternative to those unable to afford keyboards and mice.

What it does

With nothing besides a laptop webcam and a piece of paper, we built a tool that is capable of replacing the computer mouse and keyboard by performing the same tasks at a small fraction of the price. Our project enables those unable to purchase certain technology appliances to have easy and convenient access to them wherever they go.

How we built it

Our project revolved around the use of computer vision. With the mouse, we tracked the location of the paper mouse as well as the movement of the pointer and middle finger, detecting a click whenever the finger went up and then back down onto the mouse. The keyboard was built with an on screen keyboard that could be interacted with by fingers. Each finger would have a round circle at the tips of each that would grow larger the closer it came to the camera, all enabled with the use of computer vision.

Challenges we ran into

While we initially planned to utilize a naive Mediapipe implementation and wrap up the computer vision aspect of the application within one day, this process ended up being a lot more grueling than expected. Mediapipe had issues when detecting the back of our hands versus the front of our hands, and we had no idea how to combat this. After a few hours of failed debugging, we ended up pivoting our entire approach and focusing on detecting contours with certain color values. This proved to be extremely valuable and helped us successfully implement our application shortly after.

Accomplishments that we're proud of

We’re proud of maintaining grit throughout the debugging process; there was one point halfway through the hackathon when we had considered restarting because nothing was working. However, we managed to clutch up, and we are happy with the end result! Additionally, while we are obviously proud of the application’s detection logic, our personal favorite aspect of the application is its frontend. It’s minimalistic yet functional, with a few hidden gems, and we thoroughly enjoyed creating it.

What we learned

The challenges we came across with using a naive Mediapipe implementation ended up forcing us to end up learning a lot more about restrictive color spaces, HSV format, and how contour detection can be used as a valuable heuristic for real object detection. These skills were critical to our success in developing the app.

What's next for Vike

We're hoping to merge both executables so that users can simultaneously use the virtual mouse and keyboard, which we believe would be a substantial improvement in the quality-of-life for programmers around the world.