SafeTracker

An AI-based security system that uses computer vision and natural language processing that aims to save human lives from potentially lethal threats.

Comment

Inspiration

The horrific Uvalde Elementary School Shooting in early May 2022 resulted in the deaths of 19 students and 2 teachers. In today’s age with the advent of self-driving cars, text to image generation, and ai-systems that can replicate humans, we found it horrifying how school shootings still existed in our community. School should be the last place where kids should have to worry about gun violence. As students ourselves, this senseless act of violence hit too close to home, so as empathetic human beings who feared for the safety of students and faculty in schools, we took it upon ourselves to create a solution that can address this pressing issue. The idea of SafeTracker, an AI-based security analysis and threat mitigation app, speaks to our vision because we believe that the safety of students and faculty in schools should be a number one priority, taking precedence over all other issues.

What it does

SafeTracker is an AI-based threat recognition and mitigation app that aims to make schools safer by recognizing potential threats faster and more efficiently. There are three main features: the live feed / webcam analyzer, Discord Bot sentiment and linguistics analyzer, and the Twitter profile feed analyzer.In the live feed / webcam analyzer, users are able to utilize our real-time object detection livestream feature to be quickly alerted via text message whenever a weapon such as a gun is detected on their preset livestream. This acts as an autonomous security camera that ensures any potential threats can be recognized and the proper authorities be notified within seconds of appearing within the camera's vision. Within the dashboard, users can check the exact date, time, and location of the recognized threat, with a picture of the potential intruder attached taken by the app's camera.While this is the main feature, users are also able to add a custom-built Discord Bot that uses natural language processing and emotional sentiment to monitor various servers for malicious threats or any messages that encourage violence. Simply by typing in "/profile" and the name of Discord user, the bot will be able to pull up all flagged messages the scanned user has sent as well as a pie chart depicting what his/her relative emotions are, ranging from friendly, anger, fear, and more. Server administrators are also sent hourly updates within their servers with any flagged messages and the general emotional sentiment of all users, to ensure the utmost safety and track down any user who is posting threatening messages.Lastly, users have the option to scan any Twitter profile and check whether or not they have posted any threatening or malicious messages intending to harm others. Similar to the Discord Bot, it uses an advanced text classification machine learning model, and users are subsequently notified via text message when an online threat is detected. Users can additionally view the exact flagged message, user tag, user name, and a pie chart of the emotional sentiment (friendly, anger, fear, etc...) in the Twitter feed dashboard and are able to contact the proper authorities if necessary.

How we built it

SafeTracker is a React.js web app that has a Tailwind CSS styled front end, uses a backend API coded in Golang and Python, as well as OpenCv and Tensorflow for Machine Learning. Firstly, the live feed analyzer uses an object detection machine learning model built off of YoloV5 using transfer learning, a highly performant pre-trained convolutional neural network fine tuned on over 3100 images. Once a potential threat on campus is recognized, users and admin are notified of the exact date, time and location of the threat via text message, using Amazon’s Simple Notification Service system. The subsequent data is stored securely in a MySQL database, which can be viewed on the user’s dashboard. Secondly, the Discord bot allows you to analyze other users for any threatening messages, while also sending hourly updates to server admin for potential suspicious activity. We used a sentiment analysis classification model built with a Recurrent Neural Network architecture including custom embedding layers and a bidirectional LSTM. Lastly, the Twitter feed tracker allows individuals to enter in the account ID of any user and analyze their past tweet history using the Twitter API and our custom sentiment analysis classification model. Users will also be alerted via text message if there are any potential threats.

Challenges we ran into

For the parts of our app that required machine learning, we ran into a problem with text classification for categorizing emotional sentiment. We initially used a one-dimensional Convolutional Neural Network with one-hot encoding; however, due to the sheer sparsity of our data set and its low accuracy rate, we had to think of a new idea instead, utilizing a Recurrent Neural Network with custom embedding layers and a bidirectional LSTM. For the live feed feature, we also had to turn to transfer learning with YoloV5 because we believed this would be the best way to improve our accuracy and performance rate while detecting weapons. We also had an issue on how to make our object detection model detect real-time video data streams, and we eventually used OpenCV to capture these data streams and then take pictures of the frame each second to determine if there is a gun in frame. Another technical issue we encountered is that during the testing of livestreams, there were stream delays up to 40-50 seconds, but we were able to counteract this lag down by optimizing our Python and Golang APIs and improving network latency. Eventually, we got the overall stream lag down to mere seconds (1-3 seconds), so our optimization of our code was quite successful. As a security analysis app, we also dealt with the problem of user authentication, where we eventually reached the conclusion using JSON web tokens would be the most secure way to do authentication and keep users' information safe. Lastly, we needed a place to host images because we were displaying pie charts indicating emotional sentiment and images of potential shooters holding guns in frame, so we used Amazon's S3 storage to upload these pictures.

Accomplishments that we're proud of

First of all, we're proud of fixing the countless bugs and errors that we faced in order to produce a working application :) . Apart from that, we're proud that we've created a highly refined application that can consistently identify threats in any given video feed -- we've used over 1000 images to train and refine our AI. Even though we knew very little about the Twitter API first, we're proud that we were able to learn how it works and bring to life a cool, novel application. To continue, we're proud that we were able to explore the basics behind text classification models and create our own (which was used in both the Discord Bot and Twitter Feed Analyzer). Finally, we're proud of ourselves for creating an application that has the potential to make a positive impact on our community, and could save hundreds of lives.

What we learned

Throughout the challenge we learned to work together as a team, whether that be to debug code or generate new ideas. In addition, we learned how to use the Twitter API and use Twitter's developer account feature. We learned how to create a Discord Bot from scratch, and, after learning how to make a text classification model, we integrated the two ideas. To continue, we learned how to create a trained, highly refined AI, and then integrate that with video footage in order to create the live feed analyzer. Finally, we refined our mySQL, AWS, tensorflow, and flask skills (among others, of course).

What's next for SafeTracker

For a 2.0 version of our app, we would like to expand the app's functionality to allow for users to embed their own videos, such as live feed directly from CCTV cameras, instead of having to use a YouTube livestream. For example, allowing schools to directly link the feed from their CCTV cameras would allow threats on campus to be identified immediately, and the proper school authorities could be notified (via a text message) before anything serious occurs. Enabling people to connect their CCTV footage would also enable banks and government offices to utilize SafeTracker for their own threat detection. In addition, we would like to develop a function which enables multiple people to be notified, such as all teachers and administrators within in a school, which could drastically reduce response times (as more people would get the message in a shorter span of time). In the next version of the app, we would also like to enable people to directly contact law enforcement when a threat is identified. This version of the app doesn't include that feature because we don't want to take a risk in wasting valuable government resources for a false alarm. That's why we sent the images just to teachers/administrators to serve as safeguard for a misscall. However if we did include the emergency services feature, it would allow school administrators to ensure safety for their students with the touch of a button when a threat is identified, rather having to take the time to call the police and then explain the situation. We would also like to develop an iOS and Android app for SafeTracker, which would allow users to have better accessibility to the app, especially if they are on-the-go. Finally, we would also like to further improve the latency between the video input and threat detection in order to detect threats at an even quicker pace, as a few seconds could potentially save thousands of lives and prevent a shooting.

Built With

Share this project:

Updates