Signado

I love you
Hello

Inspiration

We aims to bridge the communication gap between hearing-impaired individuals and those who don't understand sign language.

What it does

This web app utilizes the webcam to capture hand gestures, recognizes the corresponding sign language symbols using machine learning models, and displays the result on the screen.

Features

Real-time hand gesture recognition
Supports standard ASL
Intuitive user interface
Cross-platform compatibility (iOS and Android via web browsers)

How we built it

We use Hand Pose Detection Model and Fingerpose to detect hand and its correspond gesture from webcam. For frontend, we use ReactJS and Vite as the build tool and serve our website on Netlify. There is no backend since we embed the models on client side.

Challenges we ran into

Model Choice

We originally used google web ML model to trained with data that contains many noises such as the person and the background. This implies that the prediction will be heavily biased on objects that are not hands. Next, we thought about labeling the image to emphasize on hand. That leads to another bias on some hand gestures are weighted more, so the model tends to predict certain gestures even when we did not pose it that way. Later, we discover to use hand landscape to recognize hand only and joint pose to recognize the gesture which gives much better prediction.

Streaming

In order to let the webcam worked for all device(mobile and desktop) of different specs and screen ratio, we struggle to find a way to enable full screen on all of them. First solution was to hard code the width and height for one device but that was hard to adjust and limited. During try and error, there's another issue that raised, the width and height is applied horizontally for mobile devices, so to work on both mobile and desktop, we dynamically check the user's screen ratio. To solve full screen issue, we used progressive web app to capture the device window.

Accomplishments that we're proud of

Active and accurate hand tracking from webcam streaming
Finding ways to translate different gestures from ASL to English
Being able to use across mobile and desktop
Intuitive yet functional design

What we learned

Learned about American Sign Language
Deploy progressive web app
How machine learning model takes inputs and make prediction
How to stream from webcam to inputs to our model
Variations of machine learning model and how to fine-tune them

What's next for Signado

Next step, we plan to add two hand and motion based gestures support since many words in sign language require the use of these two properties. Also to improve on model accuracy, we can use the latest [handpose]{https://blog.tensorflow.org/2021/11/3D-handpose.html} model that transform the hand into a 3D mesh. This will provide more possibility and variation to the gesture that can be performed.