ScribbleTex

ScribbleTex simplifies converting handwritten text to LaTeX, seamlessly blending React, Flask, & a Google Cloud-hosted ML model. Effortlessly transcribing your notes into precise equations.

Comment 1

Inspiration

In school, one of the largest contributors to academic success is the ability to efficiently take notes. Some people prefer paper and pencil due to its rustic and simplistic design, others prefer the iPad with new and modern note taking features. Finally, there are those who perhaps have put down the pencil entirely... and picked up a computer whether it's due to the bulkiness and burden of too many notebooks or its relative cost savings compared to the iPad (not everyone needs one). However, when typing notes, it's extremely difficult to input equations. Needing to quickly switch to latex and memorize complex latex formulas in order to efficiently and neatly type equations, we have created a tool that has completely alleviated the pains of typing out notes in math.

What it does

Our unrealized goal: Our project is a two step solution that gives instructors a whiteboard to write their notes and once an equation is written, in real time, our machine learning algorithm would make an API call that translates the handwriting completely into latex formatted equations. Unlike the current norm whether that be Notability, Goodnotes, or Microsoft Paint, our website, hosted on a domain, would be able to replace every current notes tool, and for students who can access the website through a room key, be able to provide latex code to students in real time hitting multiple birds with one stone.

Our MVP: We were able to create a machine learning algorithm hosted on Google cloud and given a functioning website that works locally, we have proved that the idea is possible due to a functioning whiteboard and handwritten equations that can be translated in real time. Given more time, we would have undoubtedly been able to do so much more which will be covered below.

How we built it

The entire project essentially consists of three important translation layers that work in harmony.

1) The frontend React project 2) Algorithms to parse images and deliver correctly formatted data to the machine learning model 3) The machine learning model, hosted on the back end, which then communicate the corresponding latex symbol(s) back to the frontend

Our base layer was crafted through React where basic whiteboard features were to be implemented. We then wrote python scripts that were to essentially take in any sized whiteboard images, extract meaning from the images through computer vision where each character was separated, pre-processed, resized, and then plugged to the machine learning model.

Challenges we ran into

Mastering API endpoints presented a formidable challenge. Navigating the intricacies of different endpoints, understanding their functionalities, and effectively integrating them into ScribbleTex was a learning curve. We faced complexities in ensuring seamless communication between our frontend and backend, requiring meticulous attention to data formats, authentication methods, and error handling.

One significant hurdle revolved around comprehending the documentation and nuances of diverse APIs. Each API came with its unique set of specifications, which demanded a deep dive into their documentation. Deciphering their intricacies, understanding parameters, and deciphering response formats posed initial roadblocks that required meticulous trial and error.

Handling asynchronous requests and managing the flow of data between the frontend and backend via these endpoints was another challenge. Ensuring that the right data reached the right endpoints at the right time demanded a keen understanding of asynchronous programming and efficient handling of promises.

Moreover, troubleshooting errors and debugging issues that arose from miscommunication or incorrect data transmission added to the complexity. The process of identifying and rectifying issues across different API endpoints while maintaining the overall functionality of ScribbleTex was a persistent challenge.

However, despite these challenges, perseverance and collaborative efforts were our guiding lights. Through diligent research, continuous testing, and collaborative problem-solving, we steadily navigated through these hurdles. Learning from each setback, we fine-tuned our approach, incrementally improving our understanding and implementation of API endpoints.

Overcoming these challenges not only expanded our technical expertise but also reinforced our teamwork and problem-solving skills. Each obstacle became an opportunity to grow, enhancing our ability to navigate complex API integrations and reinforcing our commitment to delivering a robust and seamless ScribbleTex experience to our users.

Accomplishments that we're proud of

Let's dive into the accomplishments that we're thrilled about with ScribbleTex—a journey filled with victories that have propelled us forward in our quest for excellence.

Firstly, mastering Google Cloud for hosting our machine learning model stands tall as a significant achievement. The seamless deployment and scalability we achieved with Google's infrastructure empowered ScribbleTex to handle complex tasks of handwriting recognition, converting scribbles into LaTeX with precision.

Our successful integration of various APIs marks another triumph. Bridging the gap between frontend and backend through these APIs unlocked a world of functionalities. This accomplishment enabled ScribbleTex to understand and translate diverse handwriting styles into accurate LaTeX equations—a milestone that reflects our dedication to seamless user experiences.

The evolution of ScribbleTex's frontend with React has been a source of immense pride. Crafting an engaging and intuitive interface using React's modular components has elevated ScribbleTex's usability. This accomplishment signifies our commitment to creating an environment where users feel empowered and at ease while interacting with the application.

On the backend front, leveraging Flask's capabilities has been a remarkable achievement. Harnessing Flask's simplicity and robustness enabled us to orchestrate smooth communication between ScribbleTex and Google Cloud resources. This accomplishment streamlined backend operations, ensuring responsiveness and reliability for our users.

Moreover, fine-tuning the machine learning model represents a significant milestone. Enhancing its ability to decipher intricate handwriting styles and diverse inputs has elevated ScribbleTex's accuracy to a commendable level. This accomplishment reflects our relentless pursuit of perfection in transforming handwritten content into precise LaTeX.

Looking back at these accomplishments fills us with pride, but it's just the beginning. We're excited to build upon these achievements, gearing up to enhance ScribbleTex's speed, reliability, and user experience. The journey ahead holds the promise of even greater milestones, cementing ScribbleTex as a transformative tool in the realm of handwritten text conversion.

What we learned

Firstly, we dove headfirst into the world of Google Cloud, where the magic truly began. We found that integrating ScribbleTex with Google's Cloud platform provided an incredible backbone for our machine learning aspirations. The ease of deploying and scaling machine learning models using Google Cloud resources was a game-changer. We learned how to host our models and manage resources effectively, providing a robust and scalable backend for ScribbleTex.

But our aspirations didn't stop there. APIs became our bridge to connect ScribbleTex with the vast world of possibilities. We discovered how APIs acted as conduits, enabling seamless communication between our frontend and backend. This empowered us to incorporate cutting-edge functionalities like handwriting recognition, transforming ScribbleTex into a wizard that understands and translates written text into LaTeX.

React emerged as our beacon of creativity on the frontend. Its modular, component-based structure allowed us to craft an engaging user interface, fostering an environment where users could interact effortlessly with ScribbleTex. We marveled at React's flexibility and efficiency, enabling us to iterate rapidly and refine the user experience iteratively.

Flask, on the backend, emerged as our steadfast companion, orchestrating the flow of data and requests. Its simplicity, coupled with its robustness, streamlined our backend operations. We learned to leverage Flask's prowess in handling HTTP requests, routing, and interfacing with Google Cloud resources, ensuring a smooth and responsive experience for ScribbleTex users.

Our journey involved fine-tuning ScribbleTex's brain—the machine learning model—immensely. We strived to enhance its ability to recognize various handwriting styles, intricate inputs, and even different languages. The goal was pinpoint accuracy, ensuring ScribbleTex seamlessly translated even the most diverse and artistic handwriting into LaTeX with precision.

Looking ahead, we're eager to amplify ScribbleTex's speed and reliability. The focus is on optimizations—making it faster and more robust to cater to a growing user base. Enhancing the user experience is on the horizon too—personalizing settings, enabling real-time collaboration, and integrating seamlessly with other tools to augment user workflows.

Our quest with ScribbleTex isn't just about technology; it's about crafting a tool that empowers users, simplifies workflows, and transcends language barriers. The lessons learned from Google Cloud, APIs, React, and Flask have laid the foundation for ScribbleTex's evolution into a dynamic, user-friendly, and powerful application. Brace yourself for the next chapter in ScribbleTex's journey—it's set to be a revolutionary experience!

What's next for ScribbleTex

Firstly, we're diving into expanding its language skills. Right now, it can handle one letter at a time, but we're gearing up to teach it to understand full words and sentences in multiple languages. Imagine being able to write in French, Spanish, or Mandarin and have ScribbleTex effortlessly convert it into LaTeX!

But we're not stopping there. ScribbleTex is set to become a wizard at handling whole documents, not just single equations. Picture uploading an entire handwritten page and having it instantly transformed into perfectly formatted LaTeX. That's the dream we're working towards!

Of course, we're also fine-tuning its brain—our machine learning model. We want it to get even better at recognizing different handwriting styles and complex inputs. The goal is super accurate conversions, even if your handwriting is more artistic than textbook-perfect.

On the tech side, we're looking to make ScribbleTex faster and more reliable, so it can handle more users without breaking a sweat. Think of it as getting an engine upgrade to make sure it runs smoothly even when lots of people are using it at once.

User experience is a big deal too. We're planning to make ScribbleTex more personal, letting users tweak settings to match their style. And imagine being able to collaborate in real-time on converted documents with friends or colleagues—ScribbleTex is gearing up to make that happen.

Lastly, we're aiming to connect ScribbleTex with other tools you love using. Whether it's seamlessly integrating it with note-taking apps or educational platforms, the goal is to make your workflow smoother by making ScribbleTex a part of it.

The journey ahead is all about turning ScribbleTex into a powerful, user-friendly tool that makes converting handwriting to LaTeX a breeze, no matter the language or complexity. So, get ready for an awesome ride with ScribbleTex—it's shaping up to be a game-changer!

Share this project:

Updates