Vitre ᯅ

A charming VR experience that helps you recall your French skills by stepping into your memories.

Comment

Relive and Relearn

Step foot into a living photo album – a window into your memories of your time spent in Paris.

Inspiration

Did you know that 70% of people worldwide are interested in learning a foreign language? However, the most effective learning method, immersion and practice, is often challenging for those hesitant to speak with locals or unable to find the right environment. We sought out to try and solve this problem by – even for experiences you yourself may not have lived; While practicing your language skills and getting personalized feedback, enjoy the ability to interact and immerse yourself in a new world!

What it does

Vitre allows you to interact with a photo album containing someone else’s memories of their life! We allow you to communicate and interact with characters around you in those memories as if they were your own. At the end, we provide tailored feedback and an AI backed DELF (Diplôme d'Études en Langue Française) assessment to quantify your French capabilities. Finally, it allows for the user to make learning languages fun and effective; where users are encouraged to learn through nostalgia.

How we built it

We built all of it on Unity, using C#. We leveraged external API’s to make the project happen. When the user starts speaking, we used ChatGPT’s Whisper API to transform speech into text. Then, we fed that text into co:here, with custom prompts so that it could role play and respond in character. Meanwhile, we are checking the responses by using co:here rerank to check on the progress of the conversation, so we knew when to move on from the memory. We store all of the conversation so that we can later use co:here classify to give the player feedback on their grammar, and give them a level on their french. Then, using Eleven Labs, we converted co:here’s text to speech and played it for the player to simulate a real conversation.

Challenges we ran into

VR IS TOUGH – but incredibly rewarding! None of our team knew how to use Unity VR and the learning curve sure was steep. C# was also a tricky language to get our heads around but we pulled through! Given that our game is multilingual, we ran into challenges when it came to using LLMs but we were able to use and prompt engineering to generate suitable responses in our target language.

Accomplishments that we're proud of

Figuring out how to build and deploy on Oculus Quest 2 from Unity Getting over that steep VR learning curve – our first time ever developing in three dimensions Designing a pipeline between several APIs to achieve desired functionality Developing functional environments and UI for VR

What we learned

  • 👾 An unfathomable amount of Unity & C# game development fundamentals – from nothing!
  • 🧠 Implementing and working with Cohere models – rerank, chat & classify
  • ☎️ C# HTTP requests in a Unity VR environment
  • 🗣️ OpenAI Whisper for multilingual speech-to-text, and ElevenLabs for text-to-speech
  • 🇫🇷🇨🇦 A lot of French. Our accents got noticeably better over the hours of testing.

What's next for Vitre

  • More language support
  • More scenes for the existing language
  • Real time grammar correction
  • Pronunciation ranking and rating
  • Change memories to different voices

Credits

We took inspiration from the indie game “Before Your Eyes”, we are big fans!

Built With

Share this project:

Updates