RoboPilot

Arhan controlling the robot in live teleop.
A live scan of the dubhacks podium as a potential place where the robot could be used.

Inspiration

We were inspired by our experience in UW's robotics and computer vision research labs, where expensive sensors (often multiple) are required to get trajectories to train and use robots. These costs are prohibitive for many aspiring researchers, enthusiasts, and professionals.

What it does

RoboPilot uses hand pose estimation to allow users to remotely control robots and pick up, grasp, and move objects. This allows remote teleoperation and also trajectory capture for training - anywhere, anytime, all at once. For example, by providing safe and high fidelity teleop capture, doctors may be able to operate remotely instead of in person.

How we built it

We built RoboPilot using Python. We used libraries like MediaPipe (CV models for hand pose estimation/gestures), IssacSim/Mujoco (robot simulators), as well as physical robots to train and test on.

Challenges we ran into

Several challenges we ran into included:

Accurately triangulating 3D poses from 1-2 cameras
Streaming poses live to a simulator/robot to trajectory follow
Writing inverse kinematics equations (e.g. Jacobians) to produce movement
Finding physical hardware to test on

Accomplishments that we're proud of

We are proud that the end product was extremely close to our initial vision. It is user friendly and adaptive, in addition to working well for manual robot control and training.

What we learned

We learned a lot about T-Mobile networks, as well as the math behind 3D estimation, and how to calibrate/adjust these values to allow fine grained control of robots.

What's next for RoboPilot

Next steps for RoboPilot include expanding to more complex robots like those with multiple arms, which would require pose estimation for two hands at the same time. Also, implementing precise pose estimation for individual fingers would allow us to extend to dextrous robot manipulators.