Elevate

Generated vs. Real Notes
Graph Generated Via Embedded Vector Similarity between notes

Inspiration

Frustrated by the steep learning curve of LaTeX—a formatting language used by over 90% of students in scientific fields like math or physics—we, as engineering students, found ourselves spending more time converting handwritten notes into LaTeX than solving the actual problems.

What it does

Upload notes or problem sets as images, and Elevate converts them into LaTeX documents while annotating errors with explanations. This process not only simplifies formatting but also enhances learning by helping you identify and correct mistakes for improved academic performance.

How we built it

Convex helped us link our React frontend to the backend smoothly, making it quicker to change things and catch mistakes thanks to its type-safe coding. We organized our data using Convex's NoSQL database, adding structure and making searches faster with indexing. For security, we hooked up authentication easily with Convex's Clerk Adaptor. Since we needed to store images and LaTeX files, Convex was handy there too. Plus, we used Convex’s vector database to smarten up our chatbot, making everything work together nicely and more efficiently.

OCR Preprocessing (Image2Latex): We converted images to grayscale via the following formula:

Grayscale Conversion Formula

We applied Gaussian Blur, which smooths the image by averaging the pixels based on their spatial closeness and intensity similarity, helping to reduce high-frequency noise for next steps. We utilized the OpenCV Library to compute a (5, 5) kernel with standard deviation 0, via the following formula for a two-dimensional Gaussian:

Finally, we employed a custom sharpening kernel,

which enhances edges by increasing the contrast between adjacent pixels. The convolution operation of the image with the kernel, I * K, is given by

After the preprocessing stage, when then feed our resulting image into OpenAI’s Computer Vision API, to generate text describing what is in the image. We then feed this text into together.AI’s Mixtral-8x7B LLM, which is able to generate appropriate latex code for the text. Finally, once again using together.AI’s LLM and extensive prompt engineering and fine-tuning, we were able to highlight errors and generate corrections.

Original Image

Image After Gaussian Blur (Reduced Noise)

Image After Sharpening Kernel Convolution

Latex Code Generated by Together.AI

Latex Code Corrected by Together.AI

Database Vector Search:

Using Intersystem’s Langchain-IRIS vector search technology, we were able to:

Generate embedded vectors for latex notes
Use cosine similarity to query, for a certain problem, the most relevant notes that contain information to answer this problem
By representing each note as an edge and drawing an edge between nodes with high similarity (low vector degree distance), create a connection graph and mind map of all notes in the database
For example, querying the question “How can I find a stable matching?” in a database of one of our team member's discrete math course notes results in the following responses from Intersystem’s API:

Leveraging Together.AI’s Mixtral LLM, we also created an interactive ChatBot that is able to summarize ideas in the notes and answer questions from users about the notes.

Challenges we ran into

It took a while to figure out how to render LaTeX text in a browser, as no existing packages met our needs for clarity and precision in displaying complex formulas.

Creating LLM prompts for annotating student solutions with feedback proved difficult. We had to not only ensure that the feedback was accurate but also that it was formatted and presented properly.

The technical complexity of implementing the vector graph for note comparison presented a hard challenge. Perfecting an algorithm that seemlessly integrate Intersystem's Langchain vector embeddings into a graph structure, and generating appropriate adjacency lists required deep dives into ML, optimization, and Data Structures. Additionally, rendering the graph on our live website presented an additional challenge.

Accomplishments that we're proud of

We successfully built a tool that we will use for the rest of our lives and are optimistic enough about the quality/market potential to go to market after the Hackathon.

What we learned

We learned the crucial importance of having a diverse team with varied skill sets, enabling us to effectively divide the workload across design, backend development, and machine learning tasks.