SandwichBot

💡 Inspiration

It all started at 2 AM.

We were deep into a hackathon, code running, brains fried—and suddenly, cravings hit. We wanted a peanut butter and jam sandwich. But no one wanted to go slice bread, fight the peanut butter jar, and make a mess in the kitchen. That’s when the lightbulb moment happened:

"What if a robot could do this for us?"

And just like that, SandwichBot was born.

🛠️ How We Built It

SandwichBot brings together mechanical design, machine learning, and multi-agent orchestration to create a fully autonomous PB&J chef.

Here's how we built it:

Mechanical Setup:

• 3D-printed jig holds the bread securely in place during the entire process.

• 3D-printed gripper extension was custom-designed to include both a spoon and a spreading knife, integrated directly into the robot's end effector.

• 3D-printed condiment holder organizes and stabilizes the jars of peanut butter and jam for easy access during the sandwich-making process.

Multi-Agent Framework:

• We used an agentic framework built around Ollama-Grok-3-Instruct, which handles high-level task scheduling and function calling for orchestrating the sandwich-making pipeline.

• Each subtask is treated as a modular capability, controlled and scheduled through this intelligent orchestration layer.

Machine Learning Layers:

The first layer features a DOT model responsible for placing the bread in the jig.

It was trained on 55 episodes for 30k steps.

The second layer is an ACT model that brings the robot arm to the jam jar.

It was trained on 56 episodes for 100k steps.

Motion Planning & Control:

• For tasks like scooping and spreading condiments, we implemented:

• PID controllers for precise force and speed control

• Inverse kinematics (IK) for accurate joint movement

A library of macro-actions such as "scoop from jar" and "spread on bread," which are triggered by the agentic framework depending on task context

This multi-layered setup allowed us to explore and evaluate different AI approaches and robotics methods for a single, cohesive goal—making the perfect sandwich. The project gave us a clear perspective on which models are best suited for specific tasks, from spatial manipulation to action planning.

🧠 What We Learned

Task Decomposition Matters

Designing a sandwich-making robot isn't just one problem—it’s a series of tightly coupled sub-problems. Breaking the process into learnable and controllable modules made development scalable and debugging manageable.

Different Models, Different Strengths

Our work clearly demonstrated the strengths of various ML approaches:

• The DOT model excelled at high-precision object placement tasks with minimal training data.

• The ACT model was more effective at dynamic movements and environment interaction.

This reinforced the importance of choosing the right model for the right task, especially in multi-step robotic pipelines.

Robotics Is an Art, Not Just Engineering

Hardware + ML is a weird dance. You can train all the models you want, but in the real world, jam is sticky, bread is soft, and robots need more than just brains—they need good hands. Agentic frameworks gave us flexibility, but performance came from trial and error: tuning PID vs. IK, testing grip strength, redesigning end-effectors. It's not just about smart systems—it’s about giving them the right tools and structure to actually do the thing.

🧱 Challenges We Faced

Training on RunPod broke with a cryptic ValueError, fixed only by a janky GitHub workaround.

A dataset version mismatch (v2.0 vs v2.1) forced us to re-run everything in lerobot.

After ~30 runs, servohorns loosened and caused drift until we locked them down with threadlocker.

Loose wiring led to erratic arm movement from mechanical stress during repeated motion.

Planned joint angle limits to protect cables, but ran out of time to implement them.

Debugging meant juggling software bugs and literal hardware misalignments.

Time was tight—lots of fixes got pushed to the "future version" list.

🚀 What’s Next?

SandwichBot is just the start. With more time, we’d love to add:

• Custom sandwich recipes

• A smart pantry detection system

• Support for other spreads and ingredients

• And, of course, a built-in toaster module!

SandwichBot isn’t just a robot—it’s a love letter to late-night cravings, creative problem solving, and the magic of a well-made sandwich. 🥪🤖

Built With

• act
• crewai
• grok-3
• huggingface
• lerobot
• ollama
• phospho
• runpod