SquirrelAI

Inspiration

As university students, we know that taking notes can be difficult especially during fast-paced lectures. Even worse, the fine details of what happens during a class can easily be forgotten. What if your friend asked you a question while the professor said something important? What do you do? Use SquirrelAI!

What it does

It is a cross-platform mobile app that allows students to audio-record lectures and other important events. It automatically transcribes the contents of the audio file into text, separating speakers and focusing on the most prominent one. Additionally, we auto-generate a set of flashcards per audio file, identifying and summarizing key ideas to varying degrees of specificity. We also computed the corresponding key ideas that are the most semantically similar from the audio file, and map the flashcards to one another. This way, you can swipe left and right to access different but similar flashcards!

How we built it

We have a backend API written in Flask and hosted on Google Cloud Run. Our frontend is built on React Native and Typescript. Our database and storage is hosted serverlessly on Firebase.

In order to manage the audio data, we utilized machine learning. Using APIs such as Cohere, AssemblyAI, and SpaCy, we were able to identify numerous speakers, transcribe text, segmented the text based on content at different levels, extracted named entities, and more! We then organized this data and performed text summarization based on large language models in order to create useful and concise flashcards.

In order to connect the different key ideas on each flashcard, we took a deep dive into graph theory. Using sentence embeddings and approximate nearest neighbour search, we developed a heuristic to rank the other key ideas based on which were the most semantically similar. We also ensured that the directed graph of flashcards would be connected while each flashcard only connected to its top 2 most similar flashcards; this was so that you could effectively study all the necessary topics by simply going through the flashcards but still have the app's controls be relatively simple.

Challenges we ran into

Our main challenge was effectively creating the graph of interconnected flashcards. It was difficult because we wanted to ensure that users could swipe through all the flashcards from a given lecture or audio file, which we found took a lot more initial planning and consideration than most other processes.

Accomplishments that we're proud of

We formed our group because we all had similar intentions to create a product using speech-to-text technology to improve the note taking experience for students, and we're proud of accomplishing our goals! Additionally, we're proud of being able to incorporate new APIs sponsoring Hack the North into our product, as we hadn't used them before.

What we learned

As members of our team came from different technological backgrounds and worked on different parts of the project, so we learned a lot when putting the different parts of code together. For example, there was one challenge we encountered where we couldn't pass the data we wanted to the frontend; we learned of this kind of restriction and how to go around it.

What's next for SquirrelAI

We had gathered more data than we properly used, such as named entities from the text. One next step would be to make use of this data, for example through a keywords page.