Snapped

Snapped is a self-checkout app, made to be used by customers when purchasing items from small businesses. It scans the items being purchased, identifies them, and produces a bill based on that data.

Comment

Inspiration

Many informal markets, such as farmer's markets and pop-up shops, lack an efficient and affordable payment system. Traditional self-checkout solutions require expensive dedicated hardware, making them inaccessible to small businesses and independent sellers. Our goal was to create a solution that eliminates the need for additional hardware, allowing customers to use their own phones to scan and pay for products seamlessly. By leveraging smartphone cameras and object detection, we envision a scalable, low-cost, and accessible self-checkout system that can empower small vendors and improve the customer shopping experience.

What it does

Snapped is an app that transforms a customer's mobile device into a self-checkout system. Instead of relying on barcode scanners or dedicated checkout stations, our solution uses the phone's camera to identify products. Customers can simply point their phone at an item, and the app will recognize it, add it to their virtual cart, and allow them to complete the purchase—all without needing assistance from the vendor. This streamlines the checkout process, reduces wait times, and makes transactions more convenient for both buyers and sellers. Our goal is to make self-checkout as effortless as taking a picture.

How we built it

We built an Android app using CameraX to handle real-time camera input and MLKit for object detection and classification. The app processes images captured by the customer's phone and identifies products using a trained model.

Challenges we ran into

We ran into quite a few challenges while working on this project. We initially wanted to make a cross-platform app, so we tried using React with Expo. However, all of the object detection packages we tried to implement were not compatible with the Expo Camera. We then also tried pivoting to a React Native app to use with Android but running this on our device also led to many problems. This is why we then pivoted to an Android using Android Studio. Another issue we had was trying to get the payment APIs to work because no matter how much exploring and testing we did, we kept facing errors with that.

Accomplishments that we're proud of

We are proud of bringing our vision to life and overcoming technical setbacks along the way. One of our biggest achievements was successfully implementing real-time object detection using the mobile device’s camera. Seeing the app accurately identify common objects was incredibly rewarding and validated our approach. We also gained valuable insights into mobile app development, payment processing, and the challenges of working with real-world machine learning models.

What we learned

  • How to access a mobile device's camera in an app
  • Object detection and identification using React and MLKit
  • React/Expo compatibility
  • The process of how to accept payments (couldn't get it to work, but we learned about it)

What's next for Snapped

Our next steps involve refining the app to fully realize our self-checkout vision. This includes improving object detection accuracy, developing a robust and intuitive checkout page, and successfully integrating payment processing to enable seamless transactions. We also aim to explore cross-platform solutions that don’t compromise performance, making Snapped accessible to both Android and iOS users. We would also like to make a seller-side app to allow sellers to add pictures of the items they sell to their store's database and develop a custom model

Share this project:

Updates