cAR | Devpost

Red light highlighted in AR. Highlight turns green when the light does.
Setup diagram
Your car sees capsule person chasing the ball into the street and stops.
Here is the view looking around the wind shield. Capsule person is much less visible.

Slides: https://docs.google.com/presentation/d/1F-cMiqOHJf1Sk98DXeW8XvHQPhLVM0ulRHh-ziqz-8Y/edit?usp=sharing

Video: https://drive.google.com/file/d/0B5m-WHokVgRZeVI5cEZaTzVtV0E/view?usp=sharing

Inspiration

Autonomous vehicles are the biggest change to transportation since humans transitioned from horses to the model T (1908, first mass produced car). But users are not always comfortable putting the car in control: http://i.giphy.com/oQ8HNNds30ucM.gif

Our solution

In order for users to trust their vehicle's actions, there needs to be more feedback of what the car intends to do. That is why we made cAR. It creates an augmented reality overlay which highlights the route, other cars, obstacles, traffic lights and road signs.

How we built it

cAR is proudly made with unity. A laptop displays the environment and uses the webcam to track the user's face. An iPad displays the highlighted objects which project onto the plexiglass and the face tracking aligns the overlay with where the user is looking. We used Duet for iPad to use it as a second display. The projection technique is based on pepper's ghost https://en.wikipedia.org/wiki/Pepper's_ghost

Challenges we ran into

We thought we should make a "projector" to do the overlay, so we tried using an iPhone behind a magnifying lens inside a sandwich box from the catered food. It didn't focus or get enough light. In the end an iPad right under the plexiglass worked perfectly anyway so we didn't need the projector.

Face tracking was hard because no CV libraries would integrate well with Unity for free. So our own solution takes the pixels, scales them down, checks which ones are different than the background, and checks which of those are close to our target color. It works surprisingly well! We compare by hue, saturation, value because the color space is much tighter for comparisons than red, green, blue color space.