translateASL

Inspiration

Bridging the gap between sign language and verbal language. Like Google Translate on steroids.

What it does

Bidirectional ASL - speech translator. At the moment only handles letter signs.

How we built it

We trained a CNN for ASL sign detection using our own custom dataset. Speech synthesis and recognition is primarily done using Web Speech API

Challenges we ran into

Converting a custom trained Python model to a TensorFlow.js version had serious challenges with dependencies and version conflicts for different tensorflow libraries. Agony was caused.

Accomplishments that we're proud of

Creating the whole web-app without using a server in ~1k line file.

What we learned

How to chain different APIs and reduce latency. It was also a good opportunity to learn some ASL!

What's next for translateASL

In the future, we would like to enable translations for words along with alphabets and add a fluid gesture support. We would also want to create an avatar that generates the ASL signs on screen when converting from voice to ASL.