Sign Master

• 

  Landing Page

• 

  Instructions

• 

  Real Time Object Detection within Game

• 

  End Screen

Inspiration

We were inspired by our interest in learning sign language. We thought it would be cool to be able to teach sign language through an interactive game.

What it does

Sign Master helps players sharpen their sign language skills by detecting their hand gestures and labeling them with our pre-trained model. It requires players to sign a specific word given a prompt to win the game.

How we built it

Sign Master was created using Python and TensorFlow. These were used to create our trained model. Which we needed to convert to TensorFlow.js in order to integrate it into our website which we created using HTML/CSS and Javascript.

Challenges we ran into

The largest challenge we face was training our model in Python and TensorFlow due to manually taking pictures to label for each of the 5 sign language gestures. As well as trying to train the model itself. We were unsuccessful in training the model due to errors we were unable to fix in time.

Accomplishments that we're proud of

Our team is proud of the experienced we gained working on a team. For many of us, this is our first hackathon so we were able to learn how to coordinate with one another given our time zone differences.

What we learned

For most of our team, object detection was a topic we knew very little about. So we all were able to learn a little bit about how to create a model for our program to detect specific objects. In our case, specific ASL hand gestures. We also learned the importance of team coordination since we split our team into two sections. One for the web development and one for the object detection portion. Each section worked on their respective part of the project by coordinating with one another to combine the two parts.

What's next for Sign Master

Sign Master currently only detects static versions of specific ASL words. We would love to be able to train our object detection model to detect certain hand movements since that would be more accurate to ASL. As well as incorporate more words and phrases into our game.

Built With

• css
• html
• javascript
• python
• tensorflow
• tensorflow.js