LEGO Vision

Landing Page
Recipe Suggestion Page
Piece Detection Viewing Page
Instructions Page

Inspiration

This project is inspired by having a bunch of leftover legos from miscellaneous kits, mixed bulked bag, and that one childhood LEGO bin that you refused to throw away. We wanted to build something that makes those loose pieces useful again by helping people figure out what they can build without needing to sort everything manually.

What it does

LEGO Vision use computer vision to detect and classify the scattered lego pieces from an uploaded image of the user's choice and use those detected pieces to suggest small build ideas.

How we built it

The backend uses a 2-stage computer vision pipeline: Roboflow for detecting pieces in the image (bounding boxes), then our YOLO-based classifier to predict the LEGO part ID for each crop. We then convert the detected inventory into a list of top matching “recipes” and show instructions + a visual detection view in the frontend.

Challenges we ran into

The biggest challenge was getting accurate LEGO piece classification, since many pieces look extremely similar and lighting/background differences can cause misclassification. Training also took a long time on CPU, and we had to debug a lot of real-world issues like dataset structure, image formats (PNG/HEIC/RGBA problems), file paths, and API responses between frontend and backend.

Accomplishments that we're proud of

We successfully built an end-to-end demo where users can upload a LEGO pile image and instantly get detections + predicted piece IDs, along with a highlightable detection view and a recipe matcher that recommends builds based on available pieces. We’re also proud that we integrated multiple tools quickly (Roboflow + YOLO + FastAPI + React) and got a working full-stack prototype running locally across different machines.

What we learned

We learned how hard real-world computer vision is, especially for fine-grained classification tasks like LEGO pieces. We gained experience building ML pipelines, training YOLO classification models, debugging dataset + preprocessing issues, and connecting everything into a clean API that a frontend can use. We also learned how important good data, consistent labeling, and realistic training images are for model performance.

What's next for LEGO Vision

Next, we want to improve classification accuracy by reducing class confusion, adding more real-world training images, and fine-tuning with better augmentation + higher resolution training. We also want to expand the recipe database, improve UI/UX for browsing build suggestions, and add a smoother instruction experience (step-by-step previews). Longer term, we want LEGO Vision to support more piece types, better counting, and a more reliable “buildability score” for any set of leftover LEGO parts.