Braillearn

Braille refreshable display
Braille refreshable display
Home screen with ability to choose mode
Braille quiz mode, waits for user speech input and verifies correct answer and keeps score
Braille practice mode, waits for user speech input and verifies correct answer
Braille learn mode, waits for user speech input to select character and displays on hardware
About page
Instructions page

Inspiration

Only 10% of blind individuals in the world can read Braille, and only one in three blind individuals is employed. Proficiency in Braille is crucial for visually impaired children to help them succeed academically and help the visually impaired enter the workforce. There is a high barrier to entry to be able to learn Braille. Braille refreshable tablets are not affordable ($4000+) and learning Braille often requires the assistance of a sighted person.

What it does

Braillearn aims to make learning Braille accessible and immersive, enabling more visually impaired individuals to learn Braille. There are three modes to choose from:

Learn. In learning mode, users can select a character to display on the Braille display through text or speech input. The display will refresh, and a user can learn how the character feels.
Practice. Our platform generates a random character for the user to feel via the display. A user will then say what they think the character is, and receive audio feedback for whether they are correct or not.
Quiz. A user can test their knowledge of Braille by selecting a number of questions they would like to try, and similarly to Practice mode, guess what the current character is. After a quiz attempt, they can choose to either test themselves on all of the characters or test themselves only on the characters that they got incorrect last time or haven't seen yet.

How we built it

Speech Recognition

Our React frontend utilized the Web Speech API for speech recognition through React component libraries. When frontend receives a user input through speech, we send that character to the backend for processing, and later verify whether the character is correct.

Character Processing

Our Node.js backend integrates Cohere API in order to process the user input. For example, if the user intended to say the letter "C", Web Speech API may interpret that as the word "sea" or "see". To solve this problem, we engineered a prompt in order to get the correct character back. We then send the single character back to the frontend for final verification.

Speech Synthesis

We wanted users to get audio feedback after they input the character. To accomplish this, we took advantage of built-in browser speech synthesis.

Braille Refreshable Display

To build the physical display, we used six servo motors that each represent a single dot in a 2x3 grid which represents a single Braille character. A straight pin is attached to each arm of the servo motor, so that when the arm is raised or lowered, the pin protrudes through an external layer which provides a tactile experience that aligns with an actual Braille character. Our display is controlled by an Arduino Uno, using the Serial Monitor to read and write to the Arduino from the backend server.

Challenges we ran into

We struggled with the placement of the pins on the servo motors. We wanted the pins to align when they are raised, and be close enough together to resemble a real Braille character. This was difficult given the dimensions of our straight pins and the layout of our servo motors.

In addition, we had issues with the Cohere API, sometimes providing a slower response time and inaccurate processed characters.

Accomplishments that we're proud of

We're extremely proud to have a complete project, including the frontend, backend, and hardware. This was our team's first time using hardware for a hackathon, and we learned a lot through trial and error.

What we learned

It is not simple to power six servo motors with one Arduino Uno
Hardware components, how to use the Arduino IDE, use the Serial Monitor to read and write
Prompt engineering
Speech recognition and synthesis

What's next for Braillearn

Support multiple languages and more characters including numbers, punctuation symbols, and short words
Extend the display to show more than one character at a time
Further incorporate AI to use more advanced algorithms for learning