Smart Sign

Smart Sign's Beginnings
The life of our data.
Multi-threading diagram.

Inspiration

There are 70 million people in the world who are deaf. Many of these people struggle to communicate with others because of a lack of global education of sign language.

What it does

Smart Sign uses flex sensors sewn into a glove that record signing motions. These sensors send data to an Arduino Uno and Raspberry Pi 3 where the movements are translated into words and letters. Raw translations are displayed in a friendly user interface.

How we made it

We split our team into different strengths. Kirsi soldered the flex sensors, gave electrical engineering advice, and sewed the sensors into the glove. Zain read, parsed, and translated the data. Natalie created the front-end for the tutoring aspect of Smart Sign. Faheem took the data from the flex sensors and put it into a buffer to be sent for translation. Holly came up with the idea, assisted with hardware integration, and provided sign language translations.

Challenges we ran into

The flex sensors were very challenging. The only sensors we could find were apart of a previously constructed circuit and had to be de-soldered for use. Calibration the reading of the flex sensor of the hand positions and having that being computed all at once so multi-processing. Determining which piece of hardware was going to do the computing.

Accomplishments that we're proud of

Deseret mapping of physical sign language positions to flex sensor output from the glove. We are using Arduino so we can talk to multiple sensors and use it as buffer before it is sent to the pi. Translated raw data with the flex sensors into tangible "thumb flexed" and "thumb relaxed".

What we learned

Soldering. Python Flask. Raspberry Pi communication handling. Learned a lot about multi-processing. Coding one handed while wearing the glove. And everyone learned a little bit of sign language!

What's next for smart sign

As Smart Sign is in its infancy, it does not have full knowledge of ASL yet. We would like to integrate a second glove for the full range of ASL. We also see applications in the future with IoT devices such as Amazon Echo to enhance home life for the deaf and hard-of-hearing.