LegiTech

App Walkthrough
Example of Using OpenCV to Isolate Word
Example of Using OpenCV to Isolate Letter

Inspiration

College has a problem with handwriting.

Professors constantly have to deal with illegible homework submissions. And, students have no way to know if their handwriting is legible enough for these professors to read.
There is no good way to convert the scientific handwritten notes of professors during lectures (equations, etc.) to digital text.

So, we aimed to build a product that solves both of these problems.

What it does

LegiTech is an optical character recognition scanner application with two main features:

Scan students’ work and grade their handwriting
Automatically convert scientific handwritten notes into LaTeX

Feature 1: Scanning and Grading Handwriting

Often students have an issue discerning if their handwriting is REALLY legible. Just because you can read your own handwriting doesn’t mean others can. Our technology helps to quantify this distinction!

Our application employs an algorithm to be able to consistently, impartially determine legibility. It works like this:

Using OpenCV, each character is read sequentially, and then passed into our neural network to determines how confident we are in predicting each individual letter.
By averaging all of these per-character predictions, we determine an accurate “score” for how confidently the entirety of your handwriting can be read.

Feature 2: Converting Handwriting to LaTeX

Countless students such as ourselves have taken classes where the professors write out long, complex equations, and we as students have no easy way to write these in our digital notes.
“LaTeX”, a document preparation software, makes it possible to write these equations in a digital format, but actually doing so it a tedious, confusing process.
Our software makes this process effortless.
Simply use the LegiTech app to scan any equation, and our efficient, A.I. driven model accurately converts the handwritten equation into LaTeX form for you automatically.

How we built it

Implementation of our Android App:

Our app is build using Java and XML in the Android Studio integrated developer environment High level overview:
The app runs “Activities” (different screens) that hold various components such as Image Views, Text Views, Button Views, etc.
On these Activities, we have “fragments” that allow for the dynamic capabilities of our app - displaying new images, updating the listings of completed scans, etc.
Java controls the logic, the functionality of dynamic components like buttons, and making the HTTP calls through our custom API to the backend.
XML controls the layout of each screen, as well as the navigation graph for how the app knows how to move between screens. A walkthrough of the app can be found in the project media.

Implementation of the backend and database:

Our Data came from a Kaggle deep learning model for handwritten letters.
We implemented a Convolutional Neural Network in Python to perform the analysis on the images- this meaning our accuracy is constantly improving.
All of the data for user is stored in an AWS Database which we query in order to populate the fields in our app.
We used a custom API to interface the backend with the frontend.

Challenges we ran into

The largest challenge we ran into was definitely the use of Android Studio. None of our team members had any experience using the IDE, and we found out very quickly that the learning curve was QUITE steep. We persisted, ultimately spending the majority of our time on this Android app implementation, and being successful in the end.

Accomplishments that we're proud of

Being able to persist and get the Android app functionality up and running is what the team is most proud of. It took a lot of determination to stay up all night and work through all of the countless errors, and our team is proud to say that we came out successful in the end.

What we learned

Above all, we learned that persistence is key. There were multiple times throughout the long night that our team felt like quitting, but we persisted and ended up creating a really cool project together as a result. In terms of technical knowledge, we learned Android Studio, how to make HTTP (POST, GET) requests, and how to integrate a backend with an android application.

What's next for LegiTech

In the future, we believe that our product could further be expanded to implement more OCR technology, as well as features such as sentiment analysis or a handwriting coach for the scanned writing.
In the further future, we believe our app could be integrated with current services such as Adobe Scan or GradeScope to allow for one feature-packed, streamlined product.