Sentimatic

Inspiration

Over the past few years, the excessive workload for mental health workers has caused a number of them to leave their jobs. With the rise of mental health patients and a declining number of professionals, there is a lot of concern regarding the ability to treat these patients. Hence, we wanted to develop a program that is going to help them monitor patients 24/7.

What it does

We created a web application called Sentimatic. Sentimatic utilizes a webcam and machine learning to analyze the patient's emotions at all times and log them onto our web app. Based on the analysis of the emotions, it will output suggestions on activities they will do.

How we built it

We leveraged Flask, Opencv, and Deep Face for the Backend Development. Additionally, we used Firebase to store the data we collected from the Backend. With the help of OpenCV, we are able to process the real time images with computer vision. We utilized Next.JS and Tailwind CSS to create an interface that displays the patients, and when we click on each patient, it is directed to a page with their time log and emotions automatically. Finally, we implemented the use of Gemini AI to generate non medical ways of improving their mental state, catered to each patients. We also featured an message output that notifies the health care workers immediately if the patients are in poor mental state.

Challenges we ran into

We had trouble finding a dataset to analyze the emotions because many of them could not detect the proper emotions. We went through multiple datasets to find an accurate dataset with our project. We also had trouble finetuning the idea that we had during the start of the Hackathon, and we had meetings to discuss about our team's vision.

Accomplishments that we're proud of

We were able to launch a real time webcam emotion analysis. We were also able to get our main emotions and also accurately detect them. Furthermore, we were able to learn more about Machine Learning, Full Stack Development, and Databases. We feel exceptionally proud that we were able to launch a product as this is our first hackathon all together.

What we learned

Since none of us had any experience in machine learning prior to this project, it was a fun experience to get to experiment with it. We were also able to enhance our knowledge in computer vision and implement new APIs.

What's next for Sentimatic

In a long term project, we will be able to use multiple webcams to detect patients in a greater scale. We will be able to implement more details and improve the speed of the program especially with better resources and hardware. Also, we would love to implement the idea of text to speech automator AI for every patients so that we will be able to assist them as soon as possible.