Versee

GIF
Versee hifi prototype gif

Inspiration

Traveling to a foreign country can be an exhilarating and eye-opening experience, full of new cultures, languages, and customs. However, the same differences that make travel so exciting can also present challenges. The language barrier can make it difficult to connect with locals, understand cultural nuances, and navigate everyday situations such as understanding unfamiliar transportation or deciphering foreign menus. One of our team members faced this challenge when visiting Thailand. While Google Translate or other popular translation apps helped with essential translations, they weren't enough to facilitate the casual, conversational dialogue that would truly allow for a genuine cultural exchange.

Despite these challenges, many travelers find that the rewards of travel far outweigh the difficulties. Breathtaking scenery, mouth-watering cuisine, and warm hospitality are just a few of the experiences that make traveling to a foreign country a deeply enriching experience. In order to unlock the full potential of the travel experiences, our team was highly determined to come up with a solution that would make meaningful connections between people of different cultures easier than ever before.

What it does

We decided to create a mobile-friendly web app that would use the phone camera to scan foreign language text, translate it, and provide suggested phrases or questions based on the context. But we didn't stop there — we wanted to make sure our app was intuitive and easy to use. We know how frustrating it can be to sift through countless Google search results, trying to find the right words or phrases to communicate effectively. Our app provides a curated list of suggested phrases and questions, taking the guesswork out of communicating in a foreign language. Whether you're trying to order food at a local restaurant, haggle for souvenirs at a market, or strike up a conversation with a new friend, our app has got you covered.

This app is more than just a translation tool — it's a bridge between languages and cultures. It allows people from different backgrounds to communicate more easily and meaningfully, to share their stories and experiences, and to build connections that transcend language barriers. You'll be able to travel with confidence, knowing that you can communicate with locals and experience their culture firsthand.

How we built it

Using Next.js as our main framework, we seamlessly integrated Google Cloud Vision and Translation APIs to read in foreign text from images (taken through the react-camera-pro feature of our app) and translate it to English. Using this text input, we utilized OpenAI API to generate conversational phrases and questions for users to have enriching conversations with locals. To ensure that our users can recite these suggestions in the correct language, we translated those phrases back to the foreign language and included romanization and pronunciations.

Challenges we ran into

Most of us came with experience using React.js, Firebase, and CSS experience; in other words, we were "full-fledged" frontend web developers. However, this app has a ton of backend logic and interactions, which proved to be quite the challenge as we had to create our own Express server, work with APIs like Google Cloud Vision, Translation, and OpenAI. Also, we had never made an application that was extremely mobile-friendly, so it was an adjustment to learn how to use CSS to cater for this. Additionally, most of us had never used Next.js nor yarn to create web apps or run commands, so that in itself was a readjustment. Also, since our idea involves integrating multiple APIs (that we'd never used before) together, we needed to read a lot, and I mean a lot, of documentation and work through even more bugs. We also tried to use the Cohere API to do several things: classify scanned texts into categories to generate more relevant results, and generate the results based on the context of the word and the related category. However, it proved to be a bit more difficult to do in the given timeframe of the hackathon, than expected. We also ran out of time and couldn't completely implement our CSS designs.

Accomplishments that we're proud of

On a technical level, we are so incredibly proud of figuring out how to integrate react-camera-pro with Google Cloud Vision and Google Cloud Translation to make computer vision, text detection, and language translation work seamlessly. Also, integrating the OpenAI API into our web app to generate conversational phrases and questions was extremely satisfying, and it was truly an "aha!" moment when we got it to work.

On a personal level, we are so proud of ourselves for staying resilient and challenging ourselves to not settle for "something easier." We spent countless hours (until 4 am on Saturday morning) ideating the perfect idea, designing an aesthetic and seamless UI/UX (shoutout to our wonderful designer, Sarah!), and making this incredible prototype come to life functionally (shoutout to our awesome devs Ryan, Andrew, and Erin!). We all came into this hackathon to learn something, and we are all extremely proud to say that we walked out with so much more than we could have ever hoped.

What we learned

We learned SO MUCH about how to style pages in a mobile-friendly manner, integrate API calls and manipulate the data to extract the exact details needed, file and code organization, approving PRs, prompt engineering, debugging thousands of bugs, reading documentation, translating Figma designs to responsive CSS, and much more!

What's next for Versee

Given more time, we hope to add the functionality that we had in our hi-fi prototype, which includes: adding a user account system (login/signup/account), allowing users to generate even more suggested conversational phrases/questions, save important translations to their account, and be compatible to even more languages!