Aircraft_Defect_Detection_Just_Analytics_Problem_Statement_1

Defect of scratch and dent identified from prediction.
Defect of scratch and dent identified from prediction.
Defect of scratch and dent identified from prediction.
Login Screen for my Power Apps
Upload Image for Defect Detection
Output from Power Apps
Using General[A1] domain for difficult use cases such as manufacturing. Defect of scratch and dent very small thus needing this domain.

Inspiration

I'm motivated to learn more regarding AI and this problem statement was about manufacturing defects which is an area I am interested in. I wanted to explore the possibility of using Microsoft Azure and to expose myself to AI more since I am still a student and from a non-IT background. This is my first time joining hackathon and it made me learn a lot and push myself out of my comfort zone. I believe I still have long way to go and I am trying to gain knowledge as much as I can throughout my journey.

What it does

This project solution explains how to use Microsoft Custom Vision AI to detect defects in the aircraft parts images by object detection and will be integrated with Microsoft PowerApps.

How we built it

Using Custom Vision portal to label the defects and train the tagged images and quick test could be performed to see whether the trained images have detect the defects correctly. From there, multiple iterations or trainings can be done to see which iteration of the models gave the best accuracy result. Best iterations published to predict whether it has defects or not by the images that has never been seen. Training and prediction can be done using API and Python SDK. Using Custom Vision portal, images are uploaded manually. Using training API, if there are batches of images to train, it can be be done automatically and it will start new iteration to the Custom Vision portal putting the batch images there. Terminal client app built and used for predicting the testing images. It can be seen that even the smallest dent and scratch that can't be identified with human eyes can be detected by using the prediction package of python SDK. Since the client terminal app does not have frontend UI to navigate with, another way can be done to build apps by using Power Apps. Power Apps will be connected to the API prediction resources from Custom Vision services and will be deployed to cloud. For this project, client mobile app is built by connecting to the API and deployed to cloud.

Challenges we ran into

The images for training were limited thus defect such as irregular colours that was identified could not be tagged due to Custom Vision service must have at least 15 tags for each label to be trained. When using training API and custom SDK, it was quite hard to do tagging for the region of dent and scratch. It's because after exporting to json file, the bounding box coordinates must be normalised to be able to do training for the images. I calculated in manually to get the normalised coordinates, however Custom Vision portal can adjust the label of the region and that's a great thing. It was hard to configure Microsoft Visual Object Tagging Tool ( VoTT) as well for tagging the images since I am connecting to the cloud by using Azure Blob Storage. But it's good for security of the data which is the training image I store there. I have to figure out how to connect Power Apps to Custom Vision service and in the end I try to find a way on how I could do it. I

Accomplishments that we're proud of

I am proud that I managed to detect the defects on the aircraft parts try accurately since some defects I could not even see it with my eyes. After seeing the prediction of the images, zooming on the images can be seen the detected scratch and dent defects being recognized. I felt I have accomplished something at least and for the first time building an apps and deploy to cloud. Microsoft have provided very good tools and services to be used and having experience to use them makes me happy.

What we learned

I have learned that resilience is very important in doing the hackathon and never give up even when facing some errors, there is always a way to solve it. I have learned a lot throughout all of the challenges that I faced.

What's next for Aircraft_Defect_Detection_Just_Analytics_Problem_Statement_1

It can be done for a real world scenario cases for aircraft manufacturing companies and leverage using Microsoft Custom Vision AI where a real time video of production happening and the defects could be detected during that exact time. This will really help the manufacturing companies. For my apps, it need to add more features of the bounding box region for the defects as it can only be displayed for now. This apps can be improved to be better.