Lenz

Imagine an AI side-kick that watches what you’re reading or typing—slides, code, search queries—and instantly spots where you’re stuck and surfaces insights when necessary!

Comment

Inspiration

Scrolling through dense docs or long articles, we realised we were tab-bing out to Google every few minutes just to clarify a term. That breaks flow and rarely gets recorded as a learning signal. We wanted an unobtrusive AI that:

  1. Notices the moment you hit friction.
  2. Explains only the confusing bit, right in place.
  3. Remembers every struggle so the next hint is smarter.

What it does

• Captures live OCR + window titles to understand what’s on your screen.
• Detects questions you type (Google, ChatGPT, YouTube) and logs them as “confusion events.”
• Cross-references those phrases with your personal mastery graph.
• Rewrites just the sentence you’re reading—adds a brown-highlighted, grade-9-level explanation.
• Surfaces a 25-word tooltip if it spots a weak concept you haven’t mastered yet.
All of this happens locally; nothing is sent to external servers except an anonymised phrase list to OpenAI for rewriting.

How we built it

Backend – FastAPI + SQLite + Sentence-Transformer + FAISS for mastery scoring.
MCP Server – Python FastMCP bridges ScreenPipe OCR with CrewAI agents.
Browser Extension – Content script tags paragraphs, background worker calls /rewrite, swaps in sentence-level spans.
Agents –

  1. Observer extracts questions & logs confusion via log_confusion.
  2. Intervention captures OCR, classifies mastery, and shows targeted tooltips.

Challenges we ran into

  1. Keeping JSON-RPC pipes alive between Node, Python, and CrewAI without blocking.
  2. Making the LLM rewrite only parts of a paragraph—solved with a strict span-wrapping prompt.
  3. Avoiding infinite rewrite loops when users manually rephrased text.
  4. Getting cross-directory imports (mastery.py) to resolve inside the standalone MCP server.

Accomplishments that we're proud of

• End-to-end demo where highlighting “posterior distribution” instantly pops a tooltip with a friendly definition.
• Zero-click onboarding—open any article and the AI quietly improves it without menus or settings.
• Logged mastery events sync across sessions, so help becomes progressively rarer and more precise.

What we learned

• Good UX in AI isn’t more answers; it’s fewer, perfectly-timed answers.
• Sentence-level rewrites feel natural; full-paragraph replacements feel invasive.
• Structured protocols (MCP) make multi-language components play nicely—worth the upfront complexity.

What's next for Lenz

  1. Adaptive difficulty scaling – long-term plan-of-study based on your confusion trends.
  2. Mobile capture – use ScreenPipe on iOS/Android to overlay hints on PDFs or Kindle.
  3. General code refactoring - so that agents across the browser and mac can communicate seamlessly.

WandD can be found here: https://wandb.ai/tanishwasp-lenz/Lenz/weave

Share this project:

Updates