Persona11ized

UI
Logo

Inspiration

Lukasz, one of our team members, has been working on representation engineering and had a way of creating LLMs that could produce a much realistic representation of characters, especially regards to how they speak, compared to just prompting and fine-tuning.

We first had the idea of using this technology to create famous experts (e.g. Einstein for Physics, Zizek for philosophy) to teach people about topics, allowing them people to have conversations with them. We eventually settled on teaching sales with Michael Scott and Dwight Schrute, since we're all fans of the TV show The Office, and they were great salesmen.

Dwight and Jim from The Office practising a Sales Call

What it does

We let users choose the characters they like and simulate a conversation between these characters, allowing the user to specify the topic so they can learn about the topic in a fun way. Users will also get a chance to interrupt these characters when they are talking.

How we built it

We used Whisper's Speech To Text Model hosted on Groq to convert the user's speech into text to be processed. Then we pass the user's response into either of our character LLMs to generate a response from a character, which is turned into a speech response by ElevenLabs' Text To Speech API

Character LLMs

We first generated positive and negative example pairs of what each characters would typically say, with OpenAI's text completion API. Then we used these pairs as training data to create vectors with Lukasz's representation engineering API. With these vectors, we then created character LLMs.

Character Voices

We recorded the voices of Michael Scott and Dwight Schrute and created custom voices with the Eleven Labs Voice Cloning API.

Challenges we ran into

Trying to incorporate multiple streaming APIs into a backend can be tricky, especially when dealing with two-way communication with web sockets. We eventually decided to give the frontend the responsibility of calling the Speech To Text and Text To Speech APIs instead and that simplified the orchestration on the backend. Otherwise, we we had to deal with complete multithreading logic.

What's next for Persona11ized

We may look to clone other characters who are experts in other areas to teach users other skills. Think of learning investing from Warren Buffett and Ray Dalio, physics from Einstein and Oppenheimer.