Project Reality

Project Reality lets users interact with projected applications by physically throwing balls at a wall. The system uses stereoscopic vision to track the path of colored balls in 3D space.

Comment

Inspiration

As addictive as video games are (like League of Legends), we can never feel fully immersed within the game. The Wii console was a step towards immersion, but still fell short, as you're just waving your arms about without anything to connect with. VR provides a nearly perfect immersion, but can be overstimulating and abstract for the average user. Taking inspiration from YouTube videos like Mark Rober using cameras to track a dart and Michael Reeves using color masking to locate his fish, we wanted to be able to interact with the virtual world by throwing a ball, bridging the balance between interaction and stimulation.

What it does

Project Reality is a mixed reality system that lets users interact with projected applications by physically throwing or shooting objects at a wall. The system uses stereoscopic vision to track the path of colored balls in 3D space, enabling real-time interaction with virtual elements.

How we built it

  • Triangulation of the ball by using Stereostopic view
  • Raspberry Pi 5 + Dual cameras
  • 3D-printed mounts
  • Unity game
  • Custom vision detection of colored balls
  • Projecting onto a white backdrop
  • Custom calibration based on the projector and ball color

Challenges we ran into

  • Accurately locating the ball, considering that the projector projects various colors onto the ball
  • Issues with the camera being inconsistent since they were cheaper quality
  • Tracking multiple different objects at the same time

Accomplishments that we're proud of

  • The accuracy achieved with isolating and tracking an object through its color
  • The developer adjustment tools were created to help the workflow
  • Threading two camera feeds so as not to limit fps

What we learned

  • That one mask is not enough to consistently isolate a color
  • Time spent used to create developer adjustment tools ended up saving a lot of time

What's next for Project Reality

  • Implementing other applications such as:
  • Sports training – simulate and analyze throws in baseball or cricket
  • Gaming – interactive, motion-based gameplay as shown in our demo
  • Occupational therapy – build motor skills and coordination for developmental delays
  • Cognitive rehab – train mind-body coordination

Use of Artificial Intelligence

We used it to debug, research methods of communication, generate boilerplate code,

Share this project:

Updates