Inspiration
We want to counteract police brutality while keeping stores safe. Our project was inspired by the George Floyd incident.
What it does
Our project is an AI CCTV cam that scans the scene for lethal weapons. If no lethal weapons are present, a psychiatrist is deployed to resolve disputes.
How we built it
The backend used Python scripts to take a video and break it down into individual frames. Each frame could be analyzed with another Python script that used Google Cloud's Vision API to identify the presence of weapons, locate where they are within the frame/image, and label them (including a confidence level).
The frontend is an Android app written in Java. It connects to backend models/storage via Google Firebase and can make use of tensorflowlite models if everything needs to be run only on the frontend. Additionally, it makes use of the built in camera app to get video feed (since CCTV equipment is not accessible for our team right now) and can utilize the mail and phone apps to contact the necessary services (disabled for demo purposes).
Challenges we ran into
Google Cloud's video intelligence API had a extremely low detection rate, so we had to transition to analyzing individual images (frames), which was much more successful. Additionally, we ran out of time before we could finish the framework for linking the frontend with the backend models. We were preparing to integrate either through client side tensorflow-lite models or using Google Firebase Functions to send the video data over to Google Cloud Vision API.
Accomplishments that we're proud of
We're proud that we have a functioning store app MVP and our ML model is able to detect weapons that are present.
What we learned
Google Cloud vision is limited for more complex models and situations. Overtime this will be improved and it will also improve our product offering.
What's next for The Floyd
The Floyd is a store CCTV cam product, so it'll have other smart AI features in the future such as:
Store front foot traffic analytics, Daily Store visitor data, Lost Items tracking
Log in or sign up for Devpost to join the conversation.