ScanEats

An app that scans a food item and determines if it is stale or not.

Comment

Inspiration

When deciding on an idea for this hackathon, we brainstormed by jotting down problems that we face in our life. Then one of our members pointed out that oftentimes when we are cooking or grocery shopping, we question whether or not something has gone stale. For example, before cooking a meal you might see an item and might think that it has gone bad, but at the same still unsure. Or when you're browsing the fruits section of a grocery store and you can not tell if the banana you plan to buy in front of you is at ideal ripeness. This problem inspired many branches of ideas as it has been an issue experienced by all our group members and their families. Although the human sense of vision is remarkable in its own right, it also often misjudges things. In particular, one of our group members had an issue deciding if his banana is rotten, showing that when judging these sorts of things, our eyes become too ambiguous for an accurate decision to be made. Machines, on the other hand, run purely on what it was taught. Thus, provided that the machine is given the proper training and data to learn from, it should be able to discern whether a food item is stale or not.

What it does

Our project is a web application with a camera integrated within it. When entering the website, users will be prompted to enable their camera, and upon doing so, the program will begin detecting specific fruit items such as a banana and an apple. It will also detect for various other things such as yourself, a person, but will only output a result of whether something is fresh or not after it detects the object in the camera's vision. The camera then takes the object's screenshot and runs a script on it which evaluates whether it is fresh or not. The result is then displayed on the screen back to the users.

How we built it

The project was split into a frontend and a backend. The frontend was built using React Javascript, HTML/CSS and Tensorflow JS. The UI components were created using React functional components that worked in unison with HTML/CSS to design the user interface of the website. The camera portion of the frontend was implemented with Tensorflow JS in which pre-trained models were used to detect what the camera is seeing. For example, the camera would draw a red outline around detected objects such as a person, fruit, etc. Afterwards, the camera would take a screenshot of the fruit, save it as a blob image, then send it to the backend via an API connection using the Python framework Flask. The backend, which is made from Python, Tensorflow, OpenCV, Scikit-learn, Numpy, and Flask has an endpoint setup for the frontend to send POST requests. Using this endpoint, the backend takes the screenshotted blob image and sends it to the classification script which runs a pre-trained model. To train the model for classification, we had to use Tensorflow and Scikit-learn with a dataset holding thousands of different fruit images. The model was based on a Binary Convolutional Neural Network for image classification, and would have an Epoch value of 100-the network would evaluate the dataset 100 times. After the script classifies the image, the result--fresh or stale--is returned and sent to the frontend via a GET request handled by Flask. Following this, the frontend updates with the result displayed on the screen. Finally, we used heroku to deploy our web application for anyone to use.

Challenges we ran into

There were several challenges throughout the hackathon. First of all, since none of us have had much experience in neural networks and machine learning, the process of learning how to train our own model using Tensorflow and Scikit-learn was a challenge we had to overcome through many hours of learning through online resources. Furthermore, although we knew we had to use a technology such as Flask to connect the backend with the frontend, when implementing this we ran into many conflicts between what the frontend gave the backend and how the backend processed it. The solution was to peer code with each other and help each other debug the issue, and ultimately after editing the classification script in the backend, we were able to successfully link the two ends.

Accomplishments that we're proud of

Since this project had many concepts and technologies we did not have much prior experience and knowledge of, we knew it would be a difficult endeavor. That being such, we are proud of successfully deploying our project with all the major features working properly in light of us having to learn almost all the technologies from scratch.

What we learned

This project was a new beginning for technologies we had never had much experience with before. Hence, we were able to learn about how neural networks work, how machines learn from a dataset of images. It was also our first time using the Python framework Flask and learning how to use it to connect the frontend and backend of an application. One of our members who had not used React JS before learned a lot about using it to build web applications and integrating it with a camera using Tensorflow JS.

What's next for ScanEats

The next steps for ScanEats are introducing more fruits and foods to be able to be scanned and classified. Furthermore, making a version for a mobile device is also a next step for our project.

Built With

Share this project:

Updates