Illustro

Inspiration

We noticed how difficult it is for the visually impaired to live through their everyday life. According to WHO, “Globally, at least 2.2 billion people have a near or distant vision impairment.” Not only is it hard for these individuals to maneuver, they can also easily get hurt as they don’t have a clear picture of their surroundings.

This can also help people that are getting old and losing some eyesight, this will be a great assisting tool.

Our grandparents have presbyopia and their eyesight is deteriorating. For one of our group members, her whole family has immigrated to Canada but her grandparents, because of age, decided to stay in China. Her grandparents live by themselves and still have to cook for themselves, this made her family really worried about their wellbeing and safety. Therefore our group decided to build a program that can help people who have eye problems in their daily lives. In this way, people don't have to be always worrying about their loved ones as they know they are safeguarded.

What it does

Through utilizing a camera, it can detect the objects shown in the frame. When a dangerous object is detected, such as a knife, scissor, sharp edge etc., the program would send out audio warnings to alert the user.

How we built it

NodeJS, React, Tensorflow cocossd Used Command Prompt to create new folders and clone already existing GitHub repositories needed for object detection Created a virtual environment using the command prompt in Windows Accessed the tensorflow cocossd api to access the machine learning database with models through imports and Command Prompt command (npm install) (seen in App.js file, seen in lines ~1-7 and packages.json file of dependencies) Coded functions for getting input from a webcam, setting camera frame size, getting detections and predictions, drawing rectangles around objects, styling, and web-app styling. ** steps 5-6 all used extensive code to complete Ran trial tests, and updated the code along the way. Uploaded the final product to Github repository

Challenges we ran into

-Figuring out real-time collaboration on Jupyter Notebook. -Downloading the necessary software to setup machine learning (need to install numerous softwares that we never used before, and oftentimes the commands in command prompt did not run as expected) We used online resources, including Google and Youtube tutorials, and at last found a solution. -The language used, javascript and css, were not languages we were too familiar with, so it was a tad difficult to navigate. -Had to transfer to Visual Studio because of the extensive issues we faced with Jupyter notebook (including, but not limited to, constant kernel disconnections, real-time collaboration errors, and difficult user interface)

Accomplishments that we're proud of

Using Command Prompt for “pip install” for quick installations. This was very different from how we installed softwares before, but it was way more efficient. Researching about machine learning and its useful applications in real life. We are most proud about our success in real-time detection! It is fascinating to all of us as it is our first time using Tensorflow and writing an object detection code.

What we learned

How machine learning works (using test models to train machines), how to setup a virtual environment to protect local data, and most importantly, how to overcome obstacles that are in our way

What's next for Illustro

For future developments, we would link the app to smart devices like cleaning robots to create a more accurate mapping of the house and to provide more safeguarding functions. Linking with other users would also be developed, so that family members can check on the user through the same app. The mobile app would also link to security cameras for family members to have real time info on the user and their safety. A tracking system would then be implemented, so that families could come to help if the user gets lost. In severe cases, for instance, a burning microwave/pan, the app would have the function to call 911. The Mobile App will be linked to Apple ID for easier user login and setup, as once users download it, they will be automatically registered. In this case, the app platform would also be registered on IOS. This would be beneficial as IOS has the function to aid visually impaired people to navigate through the phone