RoboVR

• 

  Setup!

Inspiration

Human-robot interaction is a relatively new field of robotics that has emerged alongside the advances in computing, interaction, and AI. As virtual reality and computer power continues to evolve today, new frontiers have opened into how we can interact with and control robots on scale. In particular, controlling multi-robot swarms -- on the scale of 5 to 50 robots -- has long been a difficult problem for 2D interfaces, which are limited to either point-and-click or shape-select and have limited environment capabilities. In RoboVR we leverage virtual reality to explore controlling real-world multi-robot swarms through a novel dimension: 3D. Furthermore, we believe VR has great potential to unite people remotely -- RoboVR opens new frontiers for multiple people from around the world to be engaged with one point in the real world: having a direct physical influence on the physical world, virtually.

What it does

RoboVR allows multiple humans in virtual reality to control a swarm of up to 8 robots in the real world. Human operators are presented with a square arena in VR-space, where they are able to virtually and remotely relocate the robots and have the virtual robots' real-world counterparts navigate autonomously. As context for this environment we built both a physical and a virtual Capture the Flag environment. Both environments are linked -- human directions from VR-space set waypoints for robots in the physical space, and the robots in the physical world update the VR-space with their physical locations. Two teams, each led by one player in VR, compete to navigate their physical robots to capture their opponent's virtual flags in the environment.

How we built it

The project has three primary components:

1. A game server that handles inbound location updates from the robots, inbound waypoints from the VR-space, and determines the game state (where the flags are, whether robots are "blocked" due to proximity of enough enemy robots, etc) to send to both the physical and VR-space. As the game server is being constantly bombarded by HTTP requests from both the robot controller and the VR-space, we developed the server in Golang and implemented read/write locks to protect game state data. The server also contains protections to ensure the two VR players are unable to move each other's robots.

2. A robot controller that interfaces with up to 8 Cozmos wireless mobile robots to localize and navigate them to target waypoints. Each Cozmos robot emits its own WiFi network through which to interface with the robot, challenging us to control multiple simultaneously. We were able to work around this by USB-splitting Android devices from the robot controller, so the robot controller could both interface with multiple robots as well as send location updates to the game server via GT's wifi. The robot controller also successfully avoided hard-coding robot IDs to team roles by dynamically allocating robots when discrepancies existed between the number of assigned robots and the number of robot slots indicated by the game server. We wrote the robot controller in Python.

3. A Virtual Reality environment that allows users to pick-and-place robots to set their waypoints. The environment shows the current locations of each team's robots, the locations of the user's robot's waypoint markers, and the locations of both the red and blue flags. The VR environment is frequently updating the game state from the game server, redrawing accordingly, and sending user-specified waypoints back to the server. The VR environment was created in Unity3D.

Overall, RoboVR connects real-world robots to multi-human virtual reality control. While the context we created for this project was an adversarial game, Capture the Flag, this technology has a range of applications from emergency response (e.g., multi-user coordination of autonomous systems in response to fires and disasters), air traffic control and shipping logistics (which is only recently starting to gain research traction), to military applications. We believe Virtual Reality has the potential change the world, and wanted RoboVR to demonstrate that yes, it can.

Challenges we ran into

Our greatest challenges came from the Cozmos robots. They are designed for child entertainment, not development, so their SDK is not intended for fine-tuned control via third parties -- much less controlling multiple simultaneously. Furthermore, the Cozmos we were able to access (from a robotics lab on campus) had not been used in years, so most of their batteries were all but dead. However, we pushed through these physical limitations to develop RoboVR's proof-of-concept.

Accomplishments that we're proud of

We are most proud of what we were able to learn. Before starting HackGT we made a list of what we wanted to get out of it -- Glen wanted to apply multi-robot control, Jenna wanted to try making a VR environment, and Jack wanted to learn how to use Golang. We brainstormed RoboVR to combine these interests towards a broader societal impact.

What's next for RoboVR

All three of us are part of a robotics lab on campus, and we regularly have children tour groups stop by. We succeeded in meeting our brainstormed development objectives (minus some visual touch-ups), so we want to give RoboVR a chance to inspire future hackers!

Built With

• cozmo
• golang
• oculus
• python
• unity
• vr