GestureInk

GestureInk: The future of notetaking
Cute elephant that we drew

Inspiration

Ever spent your precious time and energy commuting to school only to realize you forgot your writing utensils/ nothing is charged? What if you injured your hands and can’t hold a pen? Our team developed GestureInk to mitigate these unnecessary obstacles to learning!

What it does

With AI finger detection, our camera can track your hand as you draw in the air, saving the line art into a virtual notebook. GestureInk has the following features:

🚨RBG pen colour selection + LED lights to indicate pen colour changes 👋3+ hand gestures to switch between drawing, undoing, and colour change, and buzzers to keep user’s hands within camera bound

The only downside ? You now have no more excuses to skip classes with this convenient gadget!

How we built it

OpenCV is used to access AI finger detection libraries that is coded to track finger drawings. Other gestures such as two fingers for hold and three tapping fingers for undo are coded as additional features.

Live video feed is captured from the webcam which will track the user's gestures in front of them and be processed in OpenCV. From the hardware end, switches are used to indicate a user’s selected pen colour, with LED lights to signify which colour is activated. These red, green, and blue LED switches can be turned on and mixed simultaneously for a total of 8 colours (red, green, blue, yellow, cyan, purple, white, black). Arduino receives these signals and MQTT is used to relay the information to the python end point where the displayed ink colour is changed. Buzzers are used to alert the user that the hand is out of frame from the camera. This involves the python OpenCV code to relay information back to the glove Arduino to the buzzer through MQTT.