Bob.ai

Inspiration

In general, there are massive amounts of wastage from throwing away electronics (and other items) just because people don't know how to repair them. Truth is, if people knew how easy some fixes are, it would take less than 10 minutes, and we could save tons of money and prevent wastage. For example, Battery problems can be fixed easily and yet most of the time, people just go out and buy new laptops. This tool aims to let anyone become their own 'Bob the Builder' and fix their repair problems cheaply and without friction.

What it does

This tool live streams video allowing you to chat with an AI assistant in real time, and make the entire repair process intuitive by guiding you through the process step by step both visually with indicators on your video as well as voice and text.

How we built it

We used a segmentation model (SAM 2) to identify specific items, draw bounding boxes, and used Google Gemini in order to set up live chat for real-time feedback and assistance. We used React to build the frontend as well and Typescript and Node throughout.

Challenges we ran into

Finding the right segmentation model and getting it to work was tough. For a while, nothing could spot tiny things like screws both accurately and consistently. Once we found one that worked, our next headache was getting the tool to function in real time. Since the point of Bob.ai is to help while you’re live streaming, we needed clear, instant indicators for exactly where to do things—like which screw to unscrew. At first, the model only worked on pictures, so making it handle live video with constant feedback was a challenge. Getting the model to locate items and then draw bounding boxes in real time was another big hurdle. We had to fine-tune a lot of parameters to fix inaccuracies and make sure the right items were tagged. Making the AI agent talk back and forth in real time, and syncing that with the segmentation, took a lot of trial and error. One of the last big challenges was lowering the latency between the segmentation model and the frontend, so indicators and tracking happened instantly. After plenty of tinkering, we finally got it working.

Accomplishments that we're proud of

We’re proud that Bob.ai does real-time segmentation—not just uploading a video and getting feedback later. We got it to accurately highlight items in real time, and managed to keep latency low so you’re not seeing delayed indicators. Instead of waiting for feedback, you get help as you go and can see exactly where the problem is.

What we learned

We learned that real-time computer vision is way harder than it sounds, especially when you’re tracking tiny objects like screws with low latency. Getting the AI agent and segmentation model to communicate quickly and accurately in real time was a huge lesson for us.

What's next for Bob.ai

Next up is making Bob.ai even smarter, so it can handle more types of repairs and devices. We want to push the segmentation accuracy even further and get the whole thing working on mobile. The goal is to make Bob.ai so intuitive that whenever something breaks, you reach for Bob instead of calling customer support or buying something new.