🛍️ Project Story: ShopMate AR — Augmented Reality Shopping Assistant

💡 About the Project

ShopMate AR transforms Snapchat Spectacles into a hands-free, AI-powered shopping companion. By simply looking at products, users get instant nutritional insights and can manage their shopping list in an AR panel — making healthy choices and organization effortless.

🌟 Inspiration

We were inspired by the challenge of making everyday shopping smarter and healthier. Many people struggle to interpret nutrition labels and remember what they need. We envisioned a world where augmented reality and AI work together to make these tasks seamless, empowering users to make informed decisions in real time. The idea of combining Snapchat Spectacles with Large Language Models (LLMs) to deliver instant, contextual information felt both futuristic and practical.

🚀 What It Does

Visual Recognition: Uses Spectacles to capture product images in-store.
Nutrition Analysis: LLMs interpret product details and generate easy-to-understand nutrition summaries.
AR Shopping List: Users can add/remove items from a persistent shopping list displayed in AR.
Intuitive Interface: All interactions are hands-free, with information and controls appearing naturally beside the user.

🛠️ How We Built It

Data Capture: Leveraged Spectacles for real-time image and context collection.
AI Reasoning: Integrated LLM APIs to identify products and summarize nutritional information.
AR Interface: Built dynamic panels for nutrition facts and shopping list management using Snap’s AR development tools.
Backend: Used Python and Flask to orchestrate data flow between devices, LLMs, and the AR UI.
Testing: Iterated on UI/UX using feedback from both the Working ai-playground and only-nutrition prototypes, refining nutrition features and visual clarity.

🚧 Challenges We Ran Into

LLM Latency: Ensuring AI responses were fast enough for real-time AR use.
Context Management: Handling continuous streams of product data without overwhelming the LLM or the user.
AR Design: Creating a minimal, readable interface that works in diverse lighting and store environments.
Integration: Bridging hardware (Spectacles), software (LLMs), and AR frameworks smoothly.

🏆 Accomplishments That We're Proud Of

Built a working prototype that recognizes products and displays nutrition info in AR.
Created a robust shopping list feature that updates live in the AR panel.
Achieved a clean, intuitive user experience despite technical constraints.
Learned to optimize LLM queries for speed and relevance in a real-world setting.

📚 What We Learned

How to combine computer vision, LLMs, and AR for practical, real-world applications.
Techniques for minimizing latency and maximizing clarity in AI-powered AR interfaces.
The importance of user-centric design in emerging tech — making advanced features feel natural and helpful.

🔜 What's Next for ShopMate AR

Expand Nutrition Database: Integrate more detailed health metrics and dietary recommendations.
Voice Commands: Add voice-driven controls for even greater hands-free convenience.
Personalization: Tailor nutrition insights and shopping suggestions to individual preferences and goals.
Retail Integration: Partner with stores for seamless checkout and product recommendations.

ShopMate AR is just the beginning — we’re excited to keep pushing the boundaries of what’s possible with AI and augmented reality in everyday life!