Sergogram

Sergogram is the all-in-one solution for making your electronic messages more expressive and understandable!

Comment

Inspiration

One of the greatest benefits of texting in the current age is the ability to utilize emojis and emoticons. However, it isn’t a farfetched claim that most individuals don’t utilize emojis to efficiently express themselves through this form of communication. Segrogram aims to be this solution for these individuals at the immediate level and has the potential to be useful in enhancing interpersonal connections and understanding the importance of effective communication. The main inspiration for Segrogram was clear, providing a simple yet effective way to bring to life a conversation with minimal effort from the user. Essentially, a chatbot assistant that makes your everyday conversation more emotive and inviting for others.

What it does

Segrogram is an Optical Character Recognition Sentiment Analysis Expressive Assistant. Using only visual analysis, Segrogram is able to intelligently and efficiently determine the messaging program of use, but also the location and text being typed within the program’s respective chat bar. Segrogram is also able to visually and quickly decipher any text on the screen and convert it to type for analysis. As long as Segrogram is active, before any message is sent, you’ll have the option to use our AI trained algorithm to identify the connotative value of your message and enhance it with an emoji. All of this comes automatically and with just a single combination keypress, to allow a seamless transition between your texting/messaging experience and assisted chatting.

How we built it

We created a machine learning python script using OpenCV to detect the emotions: joy, fear, anger, sadness, disgust, shame, and guilt. Then using the library PyTesseract, were able to scan a range of platforms including: Discord, Facebook Messenger, and Whatsapp. Lastly we connected our application to a GUI using the python module Eel.

Challenges we ran into

Bridging GUI with Script: Our GUI and script are ideally supposed to continuously run simultaneously with one another, but are not able to in. So, we decided to run one after another with the python scripts taking the majority of time. Problems with Text Blot: Originally instead of OpenCV we were relying on Text Blot for sentiment analysis and foreign language translation. However, we decided to pivot, because Text Blot sentiment analysis wasn’t indepth to distinguish emotion and the Text Blot translator is no longer functional.

Accomplishments that we're proud of

The ability to attach connotative value to a stream of text through algorithmic means is the cornerstone of this project. It allowed us to take a dip into machine learning while employing our previous experiences in other software design languages and Python. The connection between a Web based GUI and a local program that operates in real time was also an accomplishment that was worth acknowledging, being one of the most difficult parts of our implementation. The optimization of our screen capture method is also a notable accomplishment that has allowed our program to complete the analysis on an average of just 0.8 seconds following the initial button press.

What we learned

We learned a lot about the specific libraries used to perform this type of visual text analysis and about the necessary tools to provide quality sentiment analysis on text through this project. Being able to quickly pick up new languages or modules in the online world is a crucial skill that was also developed during this project within the given time constraints. This was also both of our first times working with visual analysis software rather than your traditional typed text input and has given us insight on the route that is usually taken to provide accurate and precise information from such an abstract source. Overall, there were a plethora of smaller skills that we picked up along the way to drive this project to completion, all of which are sure to be reused as we progress toward our future careers.

What's next for Sergogram

To deliver a functioning project on time, we had to simplify our application's machine learning model and scanning capabilities. Moving forward, we'd like to not only expand on the number of emotions we can detect but also the number of messaging platforms our program can work on.

Built With

Share this project:

Updates