QuickCards

• 

  Test 2 of notes if the vision algorithm can differentiate between the block, paragraph and the word.

• 

  Test 1 of notes if the vision algorithm can differentiate between the block, paragraph and the word.

• 

  Test 3 of notes if the vision algorithm can differentiate between the block, paragraph and the word.

• 

  ex. Key output 1 displays the key (i.e. the word which will be defined). We have this similar outputs for test 2 and test 3.

Inspiration

Being passionate about academics, it frustrated us to see many students struggle academically due to inefficient study techniques. We observed many of our peers spend their limited study time before their exams, on tasks such as flashcard creation. Flashcards have proven to be one of the most effective ways to study, utilizing many metacognitive faculties and engaging active recall, which is crucial for memorization and studying. However, because of the laborious nature of re-writing one's entire course material on flashcards just to be able to review from them, many find it not worthwhile to spend time creating them. We came to UB Hacking to solve this issue for students by utilizing computer vision on students' notes to automate the process of flashcard creation.

What it does

Quick Cards allows a student to scan their notes, and uses Google's Cloud Vision API for handwriting detection and OCR to process their notes, and generate flashcards based on them. It currently features a web-based interface which allows its users to upload a picture of their notes, typically taken by their phone. It then makes an API call to Google Cloud Vision service, which captures the location of the elements on the notes and parses the handwriting into text.

How we built it

Challenges we ran into

One of the challenges we ran into was first with the detection of the boxes of text. The boxes formed around areas of words with high word density were overlapping with other paragraphs which made it hard to distinguish between the block and this caused the algorithm to ineffectively distinguish between the key (i.e the title) and value (i.e the explantation). Due to this, it would be hard to insert the word/question on one side of the flashcard and the answer/definition on the other side.

Accomplishments that we're proud of

We managed to detect text from images and converted that into flashcards. We managed to stay awake for the whole night and worked on our project. On a serious note, we managed to understand the Google Cloud vision API, implement our own techniques to optimize the detection. We configured the bounds on the boxes made by making an algorithm which figures out the corner coordinates of the rectangles and situates the correct boxes made around the world and the definition. On the front end side using a serverless cloud function API with a website allowing the user to upload the notes as a .jpg and outputs the generated flash cards.

What we learned

We learnt about various Google Cloud Vision APIs. From one of the "newbies" on our team, it was his first time using GitHub and after this project, he was much more confident with GitHub and is able to use it in his future applications.

What's next for Quick Cards

Our goal is to generalize this for all types of notes with unique customizable features such as if a person prefers text then the flashcards will be generated with text. We'll improve on our algorithm and front-end UI. Along with the current Vision APIs and the auto-ML, we'll try to develop our own Machine Learning models to make it independent of the format, enabling it to read all types of notes. Further, seek to implement a cross-platform mobile app, allowing users to use this service more easily.

Built With

• computer-vision
• google-cloud
• node.js
• python
• typescript