Talk2Hands

Our Slideshow
The actual translation
Our home page

Inspiration

Our inspiration for this project is to promote equal communication opportunities and create a more inclusive world for the deaf and hard-of-hearing communities. By breaking down communication barriers and providing a tool to facilitate effective communication, we aim to reduce instances of bullying and promote a more accepting and diverse society.

What it does

This web application utilizes Whisper to transcribe speech to text, which is then translated into American Sign Language (ASL) and displayed on screen. While it can convert English, Spanish, and Italian, it currently only supports displaying the translations in ASL.

How we built it

For the web-page, we utilized Nextjs, while the Flask server powered the Whisper model to enable accurate translations. Additionally, we leveraged MongoDB to store all past data obtained from the app, ensuring an efficient and seamless user experience.

Challenges we ran into

As this was our team's first experience working with audio files, we encountered challenges around sending and receiving audio files from the client. Similarly, our use of Whisper was also new, and we had to familiarize ourselves with the documentation and conduct extensive testing and debugging to ensure the functionality of our code.

Accomplishments that we're proud of

Our team takes great pride in the potential impact of this app, as it has the power to break down communication barriers for individuals who are hard of hearing. By providing speech-to-sign language translation, our web-app has the potential to promote inclusivity, understanding, and equal communication opportunities for all. We are honored to contribute to a more accessible and accepting world, and we are committed to expanding and improving our app to maximize its positive impact.

What we learned

Throughout the development of this app, our team gained valuable experience and knowledge. We learned how to effectively utilize Whisper for accurate speech-to-text translations and how to seamlessly send and receive audio files between the front-end and back-end of the app. Additionally, our team had the opportunity to learn and practice sign language, furthering our understanding of and appreciation for the deaf and hard-of-hearing communities.

What's next for Talk2Hands

Our vision for the future of this app is to increase its accessibility and functionality. We plan to develop a mobile version of the app to enhance ease of use and portability. Additionally, we aim to expand the sign language word database beyond the alphabet to include common phrases and words, and to increase the dialects of sign language, including British Sign Language. We are also working towards the development of an ML model that will enable the app to translate sign language to text, making it even more inclusive and accessible.