3db: 3D visual debugging. In VR.

The inspiration

As students with a passion to learn and innovate, we sought to raise the standards for teaching by creating a debugger that allows students to visualize what is happening in their program.

Our first idea was to make a 3D debugger using Unity. This in itself seemed fairly interesting and innovative; a quick Google search yields very few results for "3D debugger." But why not take it a step further? That's why we decided to add VR support with the Oculus Rift.

How we built it

Our solution was to create a Flask server in Python that acts as a wrapper around pdb. It then makes requests to that server from Unity and displays the information dynamically in a three-dimensional environment viewed using the Oculus Rift.

The front-end in Unity was all developed using Unity 2019.2.9f1, with our scripts (including those interfacing with Python and JSON requests) in Unity all being written in C#.

After 36 hours of coding (30 of which were most likely debugging), we were able to design a program that is functional, interactive, and fun.

Our (many) challenges

Our biggest challenge came in the construction of the server, where we had to scrape what seemed like all 32 million Google results for python libraries and commands to communicate with Unity. There were a lot of options available, but it seemed like every one we tried using had some issue or dead-end that forced us to scrap everything and restart, but we were finally able to communicate using a combination of Flask and pexpect.

Correctly positioning, scaling, and rendering all of the dynamically instantiated 3D objects in Unity also seemed a little convoluted and counter-intuitive at first. Everything is positioned and scaled relative to its parent, so we had to do a little ugly fraction math to ensure that all the data structures were encapsulated correctly. In order to get the billboard-rendered text to render correctly, we had to make several empty wrapper GameObjects when needed, a custom VR-supported camera-tracking script, and even our own shader.

The most interesting problem we faced was determining how clicks were registered in Unity with VR. We ended up using raycasting from a plane in front of the camera since the VR's two cameras are not always oriented in the same direction as the scene's main camera.

What we've learned

• Unity is surprisingly complicated. Kudos to all of the talented game developers out there!
• Finding the right tools for the job is important. It's better to know all the tools at the start rather than to learn about the right tool only a few hours before the submission is due.
• Problems can often be solved with simple solutions that use indirection. I.e. using a dummy request to pexpect to ensure that our output is synchronized correctly with the process that pdb is being ran on

Built With

• c#
• flask
• json
• pdb
• pexpect
• python
• unity