Inspiration
43 million people in the world are blind, whether they are born with it, had an accident, or due to aging. Our product WanderSafe is the solution for this problem. They can wander with freedom, while staying safe.
What it does
WanderSafe is an app built for the visually impaired. It turns computer vision into a sort of replacement vision that transforms the visual information into an auditory form so the visually impaired can understand. The app has features that describe the general scenery with a focus on any potential dangers. It also has more specific object detection that assists in avoiding the predicted potential collisions. The distance to potential dangers is also synced with the intensity of the phone vibration.
How we built it
We built some of the analysis of the frames from the camera from Gemini API. As for the general image-related operations, we used OpenCV and YOLOv8 to help train a machine learning model in detecting and classifying objects. We used Python mostly for backend and Flutter for frontend.
Challenges we ran into
Some challenges we ran into were training the model in detecting objects and controlling the voice-overs. As our model initially had voice prompts after every update in frames, the voice prompt would continuously be played without being able to finish. To overcome this challenge, we set a cooldown for the voice updates; therefore, it wouldn't harm the user experience.
Accomplishments that we're proud of
We managed to get most of the features we envisioned for this product implemented. With all the challenges along the way and debugging we had to do, the final product turned out pretty well.
What we learned
We learned some Prompt engineering. Incorporating Gemini API in our project to analyze frames, we need to craft good prompts that accurately describe the situation from the perspective of an AI assistant for the visually impaired.
What's next for WanderSafe
Going forward, we hope to see this product being used by the public. We will continue to improve the responsiveness of the app and enhance the user experience.
Log in or sign up for Devpost to join the conversation.