OnePen

We’re all heavy note-takers. After trying OneNote, GoodNotes, Notability — we found the same problem:

Switching tools breaks the flow.

To make a title, highlight, or draw a box, we stop writing, reach for a toolbar, then return — again and again. It’s a small step that, repeated hundreds of times, disrupts thought and creativity.

At HackUMass, we realized we shared the same frustration — and the same vision:

Note-taking should feel as natural as paper — only smarter.

🖋️ What it does

OnePen is a handwriting-based note-taking web app that uses AI stroke recognition to let you write, format, and organize without ever touching a toolbar.
You can change colors, thickness, visibility, or even trigger advanced actions — all directly through how you write.

🧠 Key Features

Feature	Description
✍️ Modifier Recognition	Detects underlines, boxes, curly brackets, or strike-throughs to automatically highlight, delete, or group content.
🎨 Auto Styling & Coloring	Instantly switch between up to 20+ pen styles — color, opacity, and stroke type — all through handwriting context.
⚡ Quick Tool Selection	No toolbars needed — simply circle, hold, or gesture to open a smart popup and change tools or colors instantly.
🗒️ Sticky Notes Behind Text	Attach a hidden note behind any text. Click the text later to reveal your sticky note — perfect for side thoughts or reminders.
🔗 Embedded Links on Handwriting	Turn any handwritten stroke into a clickable link to open files, websites, or folders.
🧮 Math Recognition	Integrates with Pix2Text for real-time parsing and solving of handwritten formulas and expressions.
📖 Handwritten Table of Contents	Automatically detects titles and subtitles from handwriting and builds a clickable TOC to navigate your notes.
🧾 Smart Summarize Tool	Collects all titles, boxes, or formulas from multiple notes into one “Total Note” — ideal for cheat sheets or study summaries.
☁️ Seamless Sync & Backup	Automatically saves notes locally in IndexedDB and syncs with Google Drive for cross-device access.

🛠️ How we built it

We used:

Frontend: Canvas-based custom rendering with zoom, pan, and HiDPI stylus support.
AI Engine: TensorFlow.js hybrid model combining image and geometric stroke features.
Backend: Flask server with Pix2Text integration and Google Drive API for sync.
Storage: IndexedDB for local autosave and persistent settings, synced via JSON to Drive.

🚧 Challenges we ran into

Designing an interface that feels “paper-natural” but “AI-smart.”
Training accurate gesture recognition models with limited time and data.
Handling performance — ensuring real-time inference without latency.
Keeping everything simple while adding powerful features.

🏆 Accomplishments we’re proud of

It works — exactly as we imagined. We hand-wrote over 4,000 training samples, expanded them to 36,000+, and trained the model late into Friday night — surviving on just 4 hours of sleep and pure determination until sunrise.

We’re especially proud of the UI — it seamlessly intertwines with the AI model to render strokes, apply styles, and trigger actions exactly as intended.

The result? A note-taking experience that’s smooth, intuitive, and effortless — just like real writing, with OnePen.

🧩 What we learned

Simplicity is hard. Making something feel natural takes longer than adding more features.
Data is everything. Teaching a model to understand handwriting means thousands of examples even for “simple” gestures.
Trust the process (and your teammates). Especially at 3 AM when nothing works — and you barely do either.

🚀 What’s next for OnePen

Fix all the bugs
Implement Gemini AI for more advanced feature such as summarize the whole notes or search for note and circle to search
Implement more advanced Math Solver, I have experienced working on much better math model but this time I failed to do that in here.
Real-time collaboration with multiple pens on one canvas
Gesture-based file management and tagging