DoodlAR

Inspiration

Inspired by Google Quick Draw.

What it does

The app has two modes. In the "doodle" mode, users can draw freely on a template (image tracking target) and the app can recognize the drawing and a relevant 3D object will appear. In the "paint" mode, users can paint freely on a colouring template (image tracking target), the app can capture the paintings and a relevant 3D object will appear painted accordingly.

How we built it

We use Zappar image tracking SDK to detect and track template images. We extend the SDK to support a "region capture" feature, in which we locate the target image region from the camera feed and apply a perspective transformation to obtain a 2D square image (texture) of the target image area.

In the "doodle" mode, we pass the extracted texture (i.e. drawing) into a pre-trained machine learning model (running in TensorflowJS) to classify the drawing. Currently, this pre-trained model can classify 10 types of items including apple, bed, cat, dog, eye, fish, grass, hand, icecream, and jacket. Once it's being detected, we will show the a corresponding 3D model in an Image tracking AR mode.

In the "paint" mode, we apply the extracted texture into a pre-built 3D model with UV wrapped correctly. Also the 3D model will appear in an Image Tracking AR mode.

Challenges we ran into

Extending the SDK to support "region capture" is definitely a challenging part. We made use of the "trackerGroup" matrix to reversely calculate the transformation matrix for extracting the camera feed. Unfortunately, even though we can locate the region, Zappar SDK doesn't allow us to obtain the webcam video input. After a bit of hacking into the source code of ZapparThree, we figured out a way to calculate another matrix to extract the image from the canvas background texture.

Accomplishments that we're proud of

Our testers found it fun to play with. Kids love seeing their drawings to live. Also combining AR and machine learning is definitely something interesting to work with.

What we learned

Deeper understanding of Zappar AR library and ThreeJS, especially on the transformation matrix and shader materials.

What's next for Doodlar

For the "doodle" mode, instead of using pre-built 3D models, we can dynamically generate 3D models so it will not be limited to only pre-assigned items. Also, even for the same type of item, it would be interesting to see variations depending on the drawings. So another interesting direction to go is to experiment with generative art machine learning networks.