TalkingHands

• 

  Learning Numerical Sign Language made easy with automated sign language teaching methods

• 

  Learning Alphabetical Sign Language made easy with automated sign language teaching methods

• 

  Real time image detection using machine learning models.

• 

  Home page for the website.

Inspiration

The inspiration for TalkingHands stemmed from recognizing the communication barriers faced by the deaf and hard-of-hearing community. Despite the existence of sign language interpreters, there are situations where immediate translation is not feasible. We wanted to create an accessible tool that promotes inclusivity and empowers individuals by enabling real-time interaction between sign language users and non-sign language speakers.

What it does

TalkingHands captures American Sign Language (ASL) gestures through a camera and translates them into text or speech in real-time. This allows seamless conversations between sign language users and those who do not understand ASL, facilitating effective communication in various scenarios such as classrooms, workplaces, and social settings

How we built it

We developed TalkingHands using a combination of computer vision and machine learning technologies. The project leverages: OpenCV for video capture and real-time image processing. TensorFlow and Keras for training a deep learning model to recognize ASL gestures. Python for integrating the modules and managing the backend logic. Flask for creating a simple user interface for demonstration purposes. The ASL data was gathered from publicly available datasets and enhanced with custom-captured gestures to improve model accuracy and performance.

Challenges we ran into

Building TalkingHands came with its share of challenges: Data Quality and Diversity: Ensuring that the model was trained on diverse data was difficult. We had to supplement existing datasets with custom data to cover more variations in hand shapes and movements. Real-Time Processing: Balancing model accuracy with speed for real-time performance required optimization techniques and model pruning to achieve low latency. Gesture Complexity: Differentiating between similar-looking gestures without context posed a challenge, leading us to implement additional context-checking algorithms.

Accomplishments that we're proud of

We are proud to have successfully developed a working prototype that accurately translates common ASL gestures into text and speech. Achieving real-time performance while maintaining accuracy was a significant milestone. We’re also proud of creating an intuitive user interface that enhances usability, making the tool accessible to a wide audience.

What we learned

Throughout this project, we learned about the intricacies of computer vision and deep learning, particularly the challenges of processing real-time video feeds. We gained experience in optimizing machine learning models for low-latency applications and deepened our understanding of the importance of dataset diversity and quality. Additionally, we explored how user feedback is essential for iterative development, allowing us to refine the tool for practical use.

What's next for Untitled

Looking forward, TalkingHands will expand its gesture vocabulary to include more advanced ASL signs and phrases. We aim to incorporate support for dynamic, sentence-level translation and improve contextual awareness using natural language processing (NLP). Additionally, integrating TalkingHands into mobile platforms as an app would make it even more accessible to users on the go. Future versions will focus on enhancing user experience and accommodating various regional ASL dialects to further promote inclusivity and communication equity

Built With

• cnn
• css
• html
• javascript
• keras
• mongodb
• numpy
• opencv
• pandas
• python
• scikit-learn
• tensorflow