Talk Tuah

Inspiration

This idea came from one of our team members (who will remain unnamed) who struggles with talking to women. We wanted to build this product to help people who are introverts or socially awkward be able to find easy and positive responses to any conversation I person might face.

What it does

The AR glasses are built with a camera and microphone to capture the facial expressions and voice of anybody speaking to the user. Using an advanced algorithm our AI chatbot is able to determine the mood of the speaker based on audio and visual cues. This response is then transmitted back through an OLED display in the glasses and provides a response to which the user can say. All these interactions, including name, hobbies, likes, dislikes etc. are also stored in a server to train the AI in future conversations. Transcripts are stored in a separate mobile app which users can access to look back on previous conversations

How we built it

We used an openAI API to build and train our chatbot to determine the most socially responsible and positive response based on an input. We also used the Deepgram API to convert speech to text and determine the mood behind a secondary person's input. The mobile app was built using React Native and built the glasses using self bought parts.

Challenges we ran into

One major challenge we ran into was setting up the server to store personal data, of both the user and their chatters. We struggled to develop a secure inter-device connection between the data storage and the mobile app as speech data needed to be uploaded. We found a solution by using ngrok to allow the mobile device to connect.

Accomplishments that we're proud of

Our team is most proud of our implementation of the mobile app. We truly believe that the app was the perfect way to tie our product in and make it more accessible for people by allowing customization and including custom prompts.

What we learned

Many of us learned how to code in a language we didn't feel confident in or knew at all. This was a great learning experience that we all believe will carry us through our desired career paths and positions.

What's next for Talk Tuah

Creating a customizable chatbot brings a lot of creativity and opportunities for growth. We believe there are ample opportunities to apply this outside of simple day to day conversations, whether that is teachers determining what units their students are struggling on based vocal and facial expressions, to how to craft a speech on the fly based on your audience's response to certain topics/statements.

Built With

Submitted to

Cal Hacks 11.0
- Winner Cal Hacks: Best Beginner Hack
- Winner Deepgram: Deepgram Voice Agent Quest

Created by

Hi, I’m Steven. I worked on the hardware and the corresponding firmware (C). Mostly worked with getting audio and video capture + the scripts everyone else wrote to run concurrently to achieve next to zero latency, from speech to response to display on the AR glasses, on a custom OS. Also designed the circuitry and physical hardware of the glasses

Steven Mu
Hi I’m Alex! I worked on training and prompting our AI chatbot (GPT-4) with conversational intelligence and various mood settings, which is determined using our advanced algorithmic reasoning around what we call the “sensibility score”. I also ensured our chatbot would have access to our custom built server stored with general information about the user and people in contact to create realistic and meaningful responses.

Alex Wang
he/him
Hey I'm Rick! I was the dev for the mobile configuration app that takes in user info including personal details, a live voice recording, and conversation emotions. I also helped build the Flask backend with Karthik to conduct config file transfers in order to be used in post-processing for RAG. Check out the figma I made for the mobile app designs in our devpost!

Rick Liu
Hey I'm Karthik, I worked on integrating Deepgram to transcribe real-time conversations, Hume AI to detect emotions from facial expressions, and SpeechBrain with PyTorch to recognize user voices from sample audio. I also collaborated with Rick to develop our Flask backend for receiving user configurations from the mobile app

Karthik Unnikrishnan