SnackSnap

Simple welcome page
Questionnaire to learn more about your dietary preferences
ML to identify food label and map dietary restrictions ..
also gives useful info like nutritional values and related recipes
Live camera preview of your food photo

Inspiration

We wanted to create an app that would help people come together in these divided times and enjoy what they hold most true to their hearts: their food. We wanted to help people make good use of their leftovers, and share their cultures with each other, by showing people creative ways to turn their leftovers into healthy and hearty meals for them and their families.

What it does

SnackSnap is an iOS app that makes personalized meal preparation easy for you. By taking a picture of any food item, a user can get tons of useful information such as dietary restrictions, nutritional values, and related recipes. SnackSnap makes sure that all the displayed information matches the user dietary restrictions so it streamlines the entire meal preparation process.

How we built it

Overview of the System

Computer Vision

We used a custom-trained PyTorch model to evaluate what foods we are looking at. We used the Fruits 360 Dataset to train, as it is very thorough. One downside of this dataset is that the data is too clean, and doesn’t match real world use cases. Because of Covid, we couldn’t meet up to test the app, so I had to create many training pipelines to take this overly clean data and simulate real world imbalances. Some of these pipelines include scripts to add blurring to the data, randomly crop and scale it, randomly rotate it, change the brightness and hue of it, add aberrations, and replace all pixels over a certain threshold with random noise. I used a Convolutional Neural Network model to generate these predictions, and the code can be found in the repository. The model has 98% precision and 99% recall on average of all classes (11 total). The images passed in looked something like this:

And the goal of these transformations were to make the overly clean data look more like real world data, messy and unclean.

ML Ops

During the development and integration phase, we used Azure Machine Learning collaborative notebooks to ensure that we had a centralized repository to build and deploy faster. Once our training algorithms were developed, we used Azure Machine Learning to actually train and manage the ML models using Microsoft’s infrastructure. We used Azure VMs to serve the ML endpoints, and to train our models upon. If we had more time, we would set up integration would data monitoring services such as Azure Monitor to ensure data cleanliness and consistency throughout the entire cloud platform.

Back end

We used Python’s Flask framework to host our REST API serving metadata information on food items. We utilized GCP Compute Engine as the underlying infrastructure of SnackSnap’s backend and used the Edamam nutrition analysis / recipe API to extract data about dietary restrictions, nutritional information, and related recipes. All this information is stored in the highly-scalable Google Firebase DB which is used for further analysis on the user’s eating patterns and personalized insights. In the near future, we plan to add a caching layer mapping active users with their saved recipes so that they could refer back to them.

Challenges we ran into

The dataset used to train the food classifier was not realistic, and overly cleaned out. We had to artificially try to dirty the dataset up to make the dataset even remotely usable in real world contexts. This was made even more difficult by the fact that we couldn't meet up in real time to test things out, so we had to improvise with testing. Setting up all the ML pipelines also took a long time, and we also were very sleep deprived and did not sleep during the contest at all. Trying to develop a real-time computer vision algorithm in a virtual environment brought many difficulties.

What we learned

We learned tons about scalable backend architecture patterns to serve high-performing and accurate computer vision models. We took advantage of using proven ML data ingestion pipelines to update the computer vision parameters in real time as more users sign up on SnackSnap. In the client side, we used various Swift and iOS optimization techniques such as asynchronous HTTP calls.

What's next for SnackSnap

Launch app on appstore with ML personalization outside just dietary restrictions. Also make deals with colleges so that students like us can learn to cook cheaper and healthier dishes.