Inspiration
As EECS Students who go to large public schools, we have noticed two problems that remain significant throughout our education journeys. Firstly, in the many math-heavy courses that we take, professors prefer and even sometimes require work to be submitted in LaTex format. Often, this means doing the work by hand, (on an ipad or similar note taking device) then manually transcribing the notes into LaTex, which is a time consuming, and rather inefficient process. Secondly, when we find ourselves stumped by particularly difficult problems, Office hours becomes a drag. Queues to talk to a TA can take literal hours, and by the time it becomes our turn to ask questions, we have already moved on. We wanted to solve both of these problems in unison by creating a smart note taking app that could act as an intelligent study partner that enables students to be as productive as possible.
What it does
Through our user interface, students can upload their pdfs which they want to markup. After writing their work by hand, they can use our LaTex select tool to select over the area of work that they want to transcribe into rendered LaTeX. After a quick call to the Gemini API, rendered LaTex appears in place of the handwritten work.
By pressing the AI Tutor hint button, a text field opens for students to enter the problem that they specifically are working on, so that the AI knows where to focus. Utilizing the local context as well as the transcribed work on the page, a quick call to Gemini generates informative hints based on current progress through the problem, possible mistakes made throughout the work, and overall problem strategy.
How we built it
Using google ai studio we created a frontend to house our pdf and markups, then utilized API calls to groq, and gemini to handle the LaTex decoding as well as the instructional hints. This was all displayed through a React App frontend and hosted locally as well as on Google Cloud.
Challenges we ran into
Frequently we ran into issues with displaying the LaTex on the pdf. In particular, the library we were using KaTex, would display our Latex rendered text as well as raw text side by side, which was not desired. We found that through proper installations and imports of the KaTex CSS, we resolved this issue.
Additionally, Our early version of the AI tutor was unoptimized. At first, it did not utilize local contexts, which was an issue in the amount of information that we were sending it (not sending all annotations made to the document). After this, the tutor became way to revealing, often giving the full answer if prompted. Through iterative prompt engineering, we finalized a tutor that highlights mistakes, and gives hints that only inch the user closer to the answer, ensuring that proper learning can be done.
Accomplishments that we're proud of
Sleek UI that allows for page indexing of PDF, multicolored/size variable markups. Accurate and efficient LaTex Translations, as well as appropriately tuned helpfulness of AI Hints
What we learned
We learned how to implement using Google AI studio for creating seamless frontend environments. We also learned how to use LaTex in React using the Katex Library. Finally, we learned how to utilize groq along with our LLMs to generate realtime responses for our queries.
What's next for Tex Tutor
Next, we want to create an iOS app that would have all of this functionality locally (through an open source model such as gemma) so that this can be done end to end on an ipad. Additionally, we want to add voice integration so that the student can talk to the AI tutor in order to have a converstational experience as if they were at Office Hours. Finally, we would want to implement cloud based databased so that students can have access to their notes anywhere, regardless of if they have access to the machine that they made the original notes on.
Log in or sign up for Devpost to join the conversation.