MenuLens

Our web app translates menus, displays pictures and descriptions of food, and features an AI chatbot that filters dishes based on dietary restrictions—ensuring seamless, personalized dining experience

Comment 1

Inspiration

The idea for MenuLens was born out of the challenges travelers face when navigating foreign menus. Many visitors want to explore local restaurants to immerse themselves in a country’s culture and cuisine but hesitate due to language barriers. Ordering unfamiliar dishes can feel intimidating, especially when currency differences add to the confusion. To solve this, we designed a platform that not only translates menus and converts currencies but also enhances the dining experience with dish images, allowing customers to visualize their meals. Furthermore, our AI-powered chatbot lets users input dietary restrictions, automatically filtering out unsuitable dishes—ensuring a stress-free, safe, and enjoyable dining experience anywhere in the world.

What it does

MenuLens is a web application that enables users to upload restaurant menus, which are then automatically processed. The app translates menu text into the desired language, converts prices to the selected currency, and provides dish images to help customers visualize their meals. It also displays key details such as descriptions, ingredients, and allergy tags, ensuring diners fully understand their choices. Additionally, an AI-powered chatbot allows users to enter dietary restrictions, filtering out unsuitable dishes. To enhance convenience, MenuLens includes a shopping cart feature, enabling customers to compile their orders, view an estimated total price, and effortlessly show their selections to the waiter for easy ordering.

How we built it

We built MenuLens using a combination of technologies:

  • Frontend: React.js and CSS for a responsive and intuitive interface.
  • Backend: Flask for handling requests and interacting with external APIs such as OpenAI (for text processing).
  • APIs: We integrated several APIs: Pexels API (image fetcher), Google Search Console API, Currency Freaks API (up to date currency exchange rates, and OpenAI (image processing).

Challenges we ran into

  • Image Processing: One of the biggest challenges was accurately parsing text from images, especially with varying fonts and quality. We had to fine-tune the OCR process to ensure better accuracy.
  • Translation Issues: Translating menu items automatically presented challenges in ensuring contextual accuracy, especially with complex ingredients or dish names.
  • Frontend Design and Integration with Backend: Balancing a clean design with integration with the backend. We wanted the app to feel intuitive but also be functional.
  • Fetching Relevant Images: Utilizing an API to obtain appropriate food images based on the extracted text from the original menu image.

Accomplishments that we're proud of

  • We successfully integrated image recognition technology to extract text from images and used external APIs to translate and convert currencies.
  • We built a fully functional web app with both frontend and backend working seamlessly together.
  • We successfully implemented image retrieval technology, allowing customers to view relevant dish images based on their selected menu items.
  • The project came together in a short time frame and was ready to be showcased at McHacks 12.

What we learned

  • We learned how to integrate multiple APIs effectively to solve a real-world problem.
  • We gained hands-on experience with optical character recognition and translating complex data.
  • We honed our skills in working with React and Flask, learning how to efficiently connect the frontend and backend of a web app.
  • We learned to iterate quickly, balancing speed with functionality to deliver a working product under time constraints.
  • We acquired valuable knowledge on prompt engineering for vision models and learned how to craft and design prompts more effectively.

What's next for MenuLens

  • Improved OCR: We plan to enhance the OCR accuracy to better handle menus with different fonts, languages, and quality.
  • Multi-language Support: We want to expand our language options, allowing users to upload menus from even more regions.
  • User Accounts: Adding user accounts and a history feature for saving favourite menus or cart items.
  • Mobile App: We hope to develop a mobile app version for easier menu scanning and on-the-go usage.
  • Restaurant Partnerships: Eventually, we aim to collaborate with restaurants to provide them with MenuLens-powered digital menus for their customers.

Built With

Share this project:

Updates