EasySign

Inspiration

We were inspired by the lack of attention given to Indian Sign Language. We wanted to create a simple and accessible way for people to learn ISL. The project was especially designed to be a companion for the curriculum in orphanages and schools for underprivileged students with needs.

What it does

EasySign is a gamified, AI assisted Indian Sign Language coach. Using the camera of your computer, the platform teaches and provides feedback on signs. The AI quiz feature reinforces the lessons and ensures that the user retains the information about the signs.

How we built it

The platform was built using Next.js and Firebase. The AI camera detection features were implemented using MediaPipe.

Challenges we ran into

• Managing React state update bugs with Mediapipe
• Feature extraction and dataset creation
• Finishing before deadline
• Responsiveness of pages

Accomplishments that we're proud of

We are proud of the implementation of MediaPipe to automatically detect the alphabet that the user signs and grade it in the quiz feature.

What we learned

• Integration between ReactJs and Mediapipe models
• Firebase auth implementation

What's next for EasySign

Native implementation on mobile: The project will be truly inclusive when implemented on mobile since most of the Indian population accesses the Internet using mobile phones.

Expansion of lessons: The next steps would be to increase the available lessons from alphabets to words and phrases such that the user is able to learn conversational ISL through the app without the need of any supplemental material.

Open source and open access: The end goal of EasySign is to provide access to orphanages and schools to allow them to use the app along with their curriculum. We would make the app open source to allow other developers to extend the functionality of the app and increase the reach to even more individuals.

Built With

• firebase
• javascript
• mediapipe