I have often thought about the challenges that the visually impaired face on a daily basis as one of my elementary school friends was blind. I tried to see how machine learning technology could be used to solve these challenges. While writing a research paper on how object identification could be used to aid the visually impaired, I found that most models used were for general cases and couldn't help when looking for a specific type of object. This led to me developing this project.
¢ounter accesses the user's camera to take a picture and return the total value of the coins in that picture. To do this, it transfers the image from the javascript to a python file via a combination of ajax and flask. cv2 is used to detect circles within the image which are then extracted from the image. These segments are each sent through an image classifier to check whether they are a type of coin or a false detection. After the classifier returns the value of each coin detected, the values are added and the total is returned to the js and displayed on the html page.
In the beginning, I tried to make an object detection model to detect the coins in the image, however, I found this process to be too difficult, so I opted for an image classification model instead. After creating the model, I copied the model folder and added it to my actual project so I wouldn't have to run the model training script every time I used the application. Using an image classifier led to the second major problem which was that an image classifier only labels an entire image as a single object making it useless if there is more than one object in the picture. In order to overcome this, I used the cv2 python library to detect circles in the image to find potential coins and cut each circle detection out of the image. These image segments were then sent through the classification system and the total value was returned.
Another big problem I ran into was website design. For some reason, using flask to run the HTML through the python script made images on the site stop showing up, so I had to make the background plain grey and use text symbols for the website logo.
During this project, I learned how to create an image classification model and how much harder it is to make an object detection model even though they both just detection objects. I also learned how to connect my backend script to my frontend script which I hadn't been able to do before.
I am planning to add a file upload feature to ¢ounter. I tried to do it, but I couldn't get the image file to be transferred to backend and ran out of time.
Log in or sign up for Devpost to join the conversation.