ASL-RPG

Inspiration

Coming into VTHacks, we wanted to do something with computer vision as it was always a field that had been interesting to us. We decided to focus on ASL since we both wanted to learn how to detect hands and gestures. We included the game aspect as both of us are avid gamers and having most recently played Baulder's Gate 3, we wanted to try our hand at creating an expansive RPG.

What it does

American Sign Language is a dying language with many high schools not even offering the class anymore as a foreign language. However, the amount of deaf people is rising according to the World Health organization.

ASL-RPG seeks to teach the basics of American Sign Language to people worldwide by "gamifying" learning. Play as a lone adventurer on your way to save the kingdom. On your way, you will be stopped by monsters and bosses, and using your sign language capabilities sign as quickly as you can to score critical hits and kill the monster. Upgrade your stats and save the kingdom!

How we built it

The backend was built using Python. The model was trained overnight on Friday using the Sign Language MNIST dataset through tensorflow. Using opencv and Google's mediapipe library , a live "translation" from ASL into English can be attempted.

The game itself is created using Pygame and public assets.

Challenges we ran into

We ran into a lot of problems finding a proper open source ASL dataset that wouldn't take days to train as creating our own data would be tedious and not as robust. In addition, trying to get through that initial "hurdle" of learning what computer vision actually is. We also ran into problems getting the camera to open using opencv.

Accomplishments that we're proud of

Getting a good model generated from a dataset and the camera to correctly identify a hand. Learning good game design principles.

What we learned

Learned how to train a model manually instead of using a prebaked library. We learned how to process inputs from a webcam into a python program. We learned how to use google mediapipe and tensorflow. We also learned what goes into making a good game worth playing.

What's next for ASL-RPG

Game design and accuracy of the model.

Built With

• machine-learning
• mediapipe
• opencv
• pygame
• python
• scikit-learn
• tensorflow