EyeSurfer

Inspiration

With the COVID-19 pandemic, we have been more reliant on the internet. However, the internet isn’t all friendly to everyone—especially to the visually impared. There are over 285 million visually impared people around the world, and with their disability it is very hard to search on the internet. EyeSurfer aims to tackle that problem using voice activated searching, so that the visually impared can surf through the internet, but with their voice.

What it does

EyeSurfer allows users to use the internet, all with their voice. For example, the user can say the command “I want to see vaccine progress,” then Eyesurfer would get the key word (vaccine) and search that through a news website and read out the relevant titles. Due to the time constraints, we had to limit our search to news-related commands.

How we built it

This application was built using python, using the tkinter library for simple GUI display for user interaction. We used the Google Speech-to-text API to convert voice to text, or text to voice and analyzed the text received using the Google Natural Languages API. The web automation/scraping is done through Selenium python scripts, which get triggered based on the analyzed voice input from the user.

Challenges we ran into

Designing the overall architecture of our project, and integrating the parts together. Figuring out how to deliver our service to users.

We were able to use the Google APIs to interact with the assistant, and we were able to use selenium to navigate through the website. The problem was, HOW to put the two and two together. There were several options: creating a chrome extension, a web application, or a GUI application. Thinking of the limited time frame and our skillset, we decided to use a GUI app to host our service to our users! Using the GUI app also made it MUCH easier for us to integrate our selenium script with our text to speech and speech to text scripts!

Accomplishments that we're proud of

The fact that we finished it! We are all beginner hackers, for most of us this is our first hackathon!

What we learned

Definitely linking all the bits and pieces together. From the Google NLP API to Google Cloud API to finally Selenium, it was tough to bind all libraries into one project. Most of us only had scripting experience and the fact that we had to build some kind of full stack application was a big hurdle.

What's next for EyeSurfer

To be available for all websites! Right now, we chose to tackle a specific browser experience - reading the articles on a new site. However, we would like to extend this further and provide a voice assisted interactive browsing experience to perform other basic online tasks such as shopping for products, and writing emails! As well, we want to train our custom model using Google TensorFlow to create a more robust interactive voice bot!