MarvinGPT

Inspiration

In order to be truly effective, a personal assistant needs to understand you emotions. Nonverbal communication makes up the majority of communication, so by using nonverbal information, we can bring ChatGPT closer to being a true Personal Assistant.

What it does

We enable ChatGPT to see how you're feeling so that it can tailor it responses to your current emotional state. Our plugins analyzes your face in real time and relays your emotional state to ChatGPT.

How we built it

We created an app that runs real-time facial detection and emotional analysis and created an API to allow our plugin to receive this information. Our text stack includes Flask, Node, and Express.

Challenges we ran into

Getting facial detection and emotional analysis to work reliably and learning the nuances of ChatGPT plugin implementation.

Accomplishments that we're proud of

Our plugins accurately assess the emotional state and relays it to ChatGPT and ChatGPT actually differentiates its response in a meaningful way. We are proud that our thesis (giving ChatGPT emotional context is valuable to quality of response) was proven.

What we learned

What's next for MarvinGPT

Increasing the number of modalities by which MarvinGPT assesses your emotional state (i.e. text analysis, vocal tone analysis) and increasing the variety of emotions that MarvinGPT can detect.