OBJECT CLASSIFICATION WITH AR

Object Classification with AR

Inspiration

Augmented reality applications, running on handheld devices utilized as virtual reality headsets, can also digitalize human presence in a virtual space where they can interact and perform various actions. This is an edge technology and directly applies to the real life problem of virtually identifying missing parts of an engine, then building a compatible 3D model of the fixed part.

What it does

Being time and cost efficient, companies can take advantage of our concept of object classification with augmented reality. For instance, a technician would be able to use a pair of HoloLens to identify the damaged/missing parts of an engine in real time with the generated damaged/missing part. In order to demonstrate our concept, we implemented a simple prototype based on the marker idea given to us as an example by Honeywell.

How we built it

We extended the concept of Augmented reality by training a consistent number of related marker images as input with Tensorflow by Google, tested our model with sample test images without labels, generated and optimized model. Using the concept for Transferred Learning, we collected data from the last hidden layer of our optimized model, classified real time images, which enabled us to identify missing/damaged parts of an object. Through Android Studio, the mobile device was the platform used for identification, which can easily be migrated to a provided Hololens.
Using 3D modeling with Unity, the missing part from the classified object was built, then using Vuforia the said object was augmenting to reality through the camera, ready for replacement; thus Object Classification with Augmented Reality.

Challenges We ran into

The main challenges we had were:

Lack of computing resources such as GPU and Hololens; ideally, we had to perform about 40,000 iterations to train our model, in approximately 3 minutes per iteration, making about 120,000 minutes = 2000 hours on a CPU laptop, which would have been way faster on GPU

The biggest challenge therefore, was to develop this model in a short span of time.

Accomplishments that we are proud of

We are able to develop an Object Classification with AR, with a prototype that can be adopted to classify the state of an object (engine for instance) and propose the 3D missing part in real time.

What we learned

During the development of this model, we learned how to generated a Tensor Flow Record, use Vuforia and integrate TensorFlow into android tools, train and optimize a machine learning model

What's next for OBJECT CLASSIFICATION WITH AR

Though we have achieved a satisfactory result; however, we need to have several improvements to make this model better, for example an effort can be made to improve accuracy of this model, automatically/efficiently classify damaged parts with 3D modeling, develop a module for an accurate assessment of the possible damage underneath the object body, location and other data can be utilized to estimate the damage cost or other possible results, or even using the 3D coordinates for 3D printing of the missing/damaged parts .