Inspiration
ClearHear was inspired by the ongoing struggles faced by my best friend who is hard of hearing. In class she was unable to sufficiently focus due to her having to take notes, read the board, and read the professors lips all in tandem. It was jarring to observe the struggle for something I had taken for granted to be easy. Thus, I had to do something about it to quench this injustice.
What it does
ClearHear is able to take audio files and output raw transcripts along with accessible notes using Gemini AI so that students who are hard of hearing or are faced with any number of disabilities are able to learn along side other students without having to juggle a number of unseen tasks just to keep up.
How we built it
I built it as a Streamlit web application using Python and Google Gemini. Gemini is able to perform transcription, tone indication, quotes from the speaker and the respective timestamp, along with structured note generation. I created a text-based output schema using delimiter markers so that the formatting remained consistent. Then the notes were made to be visually appealing using highlights, boldening, colour coded tone indication, and key concept emphasis. After this, the final product was able to be downloaded with either a .txt format or a .docx format.
Challenges we ran into
One of the biggest challenges I ran into was setting up my virtual environment and learning how to use Gemini. Beyond that, I did struggle with formatting JSON and plain text. There was a massive leap from testing locally to deploying with Streamlit.
Accomplishments that we're proud of
I am beyond proud that I've created a tool that can help my best friend without having the added costs that most AI powered notetaking tools are associated with. I am pleased to know that the struggles that my friend faces in the classroom may be resolved and that she is able to learn at her full potential.
What we learned
I learned how much time and effort goes into making applications for those with disabilities. After I had implemented a feature I was further reminded by another struggle she had faced and thus had to keep implementing features. It opened my eyes to a world I was previously unfamiliar and led me to gain a new found respect for those who have invisible disabilities.
What's next for ClearHear
Next, I hope to add a live transcription mode so that my friend is able to take notes as the professor speakers along with an added camera mode that is able to read the board and take note of that too so that all bases are covered in terms of gathering notes. I further hope that I am able to detect different voices that are in play and make distinctions on the given position that said person holds so that the notes are able to truly capture the classroom experience.
Log in or sign up for Devpost to join the conversation.