Sign collaborative

Simple, lightweight, inclusive, AI app to help raise awareness and help translate ASL/speech to text for deaf/mute/blind people.

Comment

Inspiration

We have people in our family who have hearing impairments and use ASL to communicate however, some of us realise that we do not know ASL properly and are in the process of learning. We wanted to make this app to raise awareness for people to learn ASL to bridge the gap between ableism and disableism.

What it does

This app uses a convolutional neural net using keras to recognise the image in front of it and convert ASL to text and then if the user wants, convert that text to speech for more convenience. The app also helps convert speech to text for people who are visually impaired as the app recognises background noise, adjusts for it and then processes the audio adjusting for the background noise to recognise the audio and convert that to speech. We also have system verification in place for the user to avoid the hassle of dealing with downloading dependencies.

How we built it

We built the model using keras and convolution neural net with softmax activation for multiclass classification and adam loss. This model is then used on a software made using the following libraries/frameworks: tkinter, pyttsx3, cv2 and speech recognition. These helped us show the camera feed, use those frames for recognising the ASL, converting text to speech and speech to text. This was all made to ensure cross-platformness among different devices with Python.

Challenges we ran into

The biggest challenge we ran into was ensuring that tkinter works cohesively among different layouts and make the os paths work relative to the current path. There were other issues with threads and deadlocks to ensure that speech recognition and text to speech worked on different threads.

Accomplishments that we're proud of

Being able to make a working product that fits our requirements of the apps

What we learned

AI and models, how to implement the model prediction in real time with our app, solve major git merging issues,

What's next for Sign collaborative

Integration into different API environments, a more accurate machine learning model

Built With

Share this project:

Updates