Inspiration
I often find myself overwhelmed by complicated instruction manuals and unreliable repair instructions. I can't always find that one person on reddit who had the same problem ten years ago, which is frustrating when I am trying to fix my electronic devices. That's why I decided to build FixAR.
What it does
FixAR allows you to describe your technical problem, which then sends that to Gemini to determine the parts of your device that might be causing the problem. Then, using real-time object detection, you can scan your device and have the possible problem areas highlighted with Augmented Reality, allowing you to tackle each problem area individually.
How I built it
The frontend is entirely SwiftUI. After the user enters the description of their problem, the description is sent to the API backend for processing by Gemini. Gemini's prompt restricts it to outputting a list of problem areas only. After the list is sent back, the user can move to the camera interface. I created an image classification model using CreateML for electronic devices specifically which is what powers the real-time object detection. At first, I was labeling my own dataset with PixLab Annotate, but eventually I was able to find a more comprehensive dataset, and I combined my own labeled data with the existing dataset to train the model. I was able to integrate the model with my app using CoreML. I used ARKit to create the buttons that the user can interact with to learn more about the specific problem areas with their device. After the user clicks the AR buttons, another prompt is sent to Gemini, where it provides a comprehensive reasoning behind why that specific component is causing the problem.
Challenges we ran into
I faced a lot of challenges when I was trying to create the Machine Learning model. I couldn't find a good dataset, so I opted to label the data myself. This took a long time, but fortunately, I was able to find a dataset that I could use for demo purposes. However, in the future, I want to use a much more advanced dataset. I faced the most problems with the AR integration though. I had a lot of issues with the mapping for the AR buttons and labeling for the device, and even now it is fairly inconsistent, although better. There were's also a lot of problems when it came to triggering the popup and using the AR elements as buttons the user could click. I also had to figure out how to have the Gemini API execute when the AR buttons were pressed without taking to long. These were the major challenges I ran into, but there were definitely more I dealt with throughout the hackathon.
Accomplishments that we're proud of
I'm proud of being able to implement the AR functionality and also creating the ML model.
What we learned
I learned about making my own datasets and also a lot about the ARKit framework offered by Apple. I also learned how to make image classification models in CreateML. I learned more about the different capabilities of Gemini's API, even if I wasn't able to completely utilize Gemini's full extent.
What's next for FixAR
I want to improve the AR labeling of the problem areas, and also increase the dataset of the ML model so the app can assist with repairing a lot more devices. I also want to add in real-time AR instructions to guide you while you repair your device.
Log in or sign up for Devpost to join the conversation.