Inspiration
We were inspired to build this app because we are all new to cooking and are learning recipes every week. Since we are new we always face three key problems during our experiences.
Having to watch youtube videos for new recipes which creates problems for messy dishes. If we have to pause the video or go back to a part, it messes with the flow if we have to wipe/wash our hands.
Organizing existing ingredients and finding substitutes for those we don't have. My mom is extremely good at figuring out alternatives for ingredients but I just do not have the experience yet for that. I always wanted someone or something that could tell me if it is still possible to make a dish even if I do not have every single ingredient in the recipe.
Shopping! Shopping is very annoying and requires a lot of studying of recipes. We wish that there was a quick and easy way to get a list of materials/ingredients without having to watch numerous videos and read blogs.
What it does
To solve our three pain points we built Kitchennaire. This is a kitchen assistant which aims to make cooking more beginner friendly and seamless.
- Our key feature is gesture controlled video playback for those who can't touch their phone with dirty hands, but missed a part of the video or need to pause it for whatever reason. Using Machine Learning we detect gestures and map them to different controls for video playback.
- Another feature we have is automatic Pantry detection. Take a photo and instantly see what you have and how it relates to the recipe you want to cook.
- Automatic shopping cart. Using features such as the pantry detection and automatic transcription parsing. The app can read transcripts of youtube cooking videos and provide you with ingredients and steps before even having to watch the video! On top of that populate a shopping list for you making your next trip much easier.
How we built it
We built this on top of Expo, a mobile development framework that relied heavily on React. Our backend was Nodejs and FastAPIs. We utilized machine learning in fine tuning a model to detect gestures and used the Youtube API to embed the video and control playback controls native to the app. We also integrated openAI LLMs to our application to read video transcripts and get cooking duration, steps and ingredients immediately before watching the video. Furthermore, we are able to use LLMs to parse an image that you can send of your pantry to see what you have, what you will need and what to shop for.
Challenges we ran into
We ran into challenges with the gesture detection. Instead of our webcam, we wanted to use our phone to detect gestures but not only did we have to deal with api calls to send the model outputs to expo, the phone camera had a very different quality, exposure settings and shutter speeds causing many problems.
Accomplishments that we're proud of
We are really proud of the ML and AI integration in this project, especially the gesture detection. It feels like magic controlling the video with only hands.
What we learned
We learned about training and tuning ML models, embedding videos, mobile development and GitHub development. This is the largest scale application we have every made
What's next for Kitchennaire
What's Next? We have so many possibilities with transcription data, such as live instructions and remarks during video playback, measurements, and more quality of life changes. Right now this is a proof of concept but we have so many more ideas.
Built With
- css3
- expo.io
- html5
- javascript
- openai
- python
- tensorflow
Log in or sign up for Devpost to join the conversation.