Logo

# EchoHands: AI-Powered Interactive ASL Learning

Inspiration

The idea for EchoHands stemmed from the need for an interactive and real-time American Sign Language (ASL) learning tool. With the rise of remote work and evermore virtual interactions, accessibility for individuals with hearing disabilities has become even more crucial. Surprisingly, leading language platforms like Duolingo still don’t offer ASL courses. EchoHands fills this gap by leveraging AI and pose estimation to create an engaging and accessible learning experience for ASL learners of all levels.

What It Does

EchoHands is a cutting-edge educational platform designed to make ASL learning intuitive and effective. Its key features include:

Real-time hand tracking using AI-powered pose estimation for instant feedback on ASL gestures.
Step-by-step learning modules designed with scientifically backed teaching methods.
Personalized AI tutoring to help users improve their accuracy over time.
Seamless integration with digital platforms, making ASL learning available in video conferencing and educational settings.

How We Built It

To bring EchoHands to life, we combined several powerful technologies:

Frontend Development: Built using Next.js, Tailwind CSS.
Backend & AI Models: Trained using PyTorch, TensorFlow, Keras, Huggingface and Cloudflare AI API for Lllama 2 model.
Machine Learning Integration: Used retrieval-augmented generation (RAG) with OpenAI and Google Search APIs to enhance language learning.
Real-time Pose Estimation: Leveraged MediaPipe to accurately track hand movements and evaluate ASL signs.

Challenges We Faced

Computational Limitations:
Mapping ASL Sentences: Recognizing full ASL sentences is extremely hard - couldn't achieve it.

Achievements Proud Of

Successfully trained custom AI models for ASL recognition using PyTorch.
Developed a real-time ASL learning interface that integrates pose estimation into an intuitive UI.
Implemented multiple AI-driven learning methods, gaining insights into how AI can enhance education.
Built a unique accessible language-learning platform that fosters inclusivity and engagement.

Lessons Learned

Pose estimation and computer vision can significantly enhance ASL learning experiences.
AI-powered language tools can make education more accessible for diverse communities.

Tech Stack

AI & ML: PyTorch, TensorFlow, Keras, LSTMs, Random Forests
Development Frameworks: Next.js, Flask
Databases & APIs: OpenAI API, Google Search API, Cloudflare API for llama2

Future Roadmap

Expand from word recognition to full ASL sentence translation using advanced NLP models.
Integrate EchoHands into video conferencing platforms to enhance accessibility in virtual communication.