Nova Clerk

Nova Clerk is a voice-driven robot computer that triages inboxes, drafts replies, schedules meetings, and outputs audit-ready receipts—powered by Amazon Nova (Sonic, Lite) + Nova Act.

Comment

Inspiration

Email + calendar is the “death by a thousand clicks” loop: triage, reply, schedule, follow up—repeat forever. We wanted a robot computer that doesn’t just chat about work, but finishes the work end-to-end with enterprise-grade safety and receipts. Amazon Nova’s stack made that possible: real-time voice, strong reasoning, and reliable UI automation.

What it does

Nova Clerk is a voice-driven auto-worker that turns natural language into verified outcomes:

  • Triage inbox: classifies unread email into Reply / Schedule / Follow-up / FYI.
  • Draft (or send) replies with the right tone, correct context, and clear next steps.
  • Propose times and update calendar: checks conflicts, suggests slots, creates holds/invites when allowed.
  • Robot-computer fallback: when APIs aren’t enough, Nova Act can operate web UIs to complete or verify steps.
  • Receipts bundle for every run: a structured log of what happened, what changed, what was verified, and what still needs approval.

Modes: READ_ONLY, DRAFT, EXECUTE—so it can be safe by default and still go full auto when permitted.

How we built it

We built Nova Clerk as a modular agent system on AWS:

  • Amazon Nova 2 Sonic for real-time voice interaction (hands-free “operator” experience).
  • Amazon Nova 2 Lite as the orchestrator: planning, tool selection, policy gating, and structured outputs (tool/function calling).
  • Amazon Nova Act for browser automation (“robot computer” moments) and visual verification when needed.
  • FastAPI backend exposing a simple /api/run workflow endpoint plus run history / receipts retrieval.
  • Avant-garde operator console UI (web app) designed around a Command Bar, live Timeline, and a Receipts Viewer so users can trust what the agent did.
  • Mock mode fixtures so the demo runs reliably without credentials, plus “real mode” connectors for actual inbox/calendar integrations.

Challenges we ran into

  • Reliability vs. wow-factor: it’s easy to make agents look magical; harder to make them consistently correct. We prioritized deterministic flows and verification.
  • Safety boundaries: sending email or creating invites can be high-impact. We implemented modes + approvals so the agent never over-commits.
  • Tooling realism: UI automation is powerful but brittle; API integrations are robust but not always available. We designed a hybrid routing approach.
  • Auditability: enterprise-grade automation needs “receipts,” not vibes. We had to standardize logs, artifacts, and verification checks into a clean bundle per run.

Accomplishments that we're proud of

  • A full end-to-end auto-worker demo: voice/text command → triage → drafts → scheduling → receipts.
  • A clean operator-grade UI that makes agent behavior legible: timeline, artifacts, verification, approvals.
  • A pragmatic hybrid execution engine: API-first for reliability, Nova Act for browser automation and proof.
  • A Receipts Bundle format that makes the system auditable and enterprise-ready.

What we learned

  • The killer feature isn’t “automation”—it’s automation you can trust.
  • Agents need explicit operating modes, guardrails, and verification loops to be viable in real organizations.
  • UI matters: a strong console turns an opaque agent into a controllable system (and makes demos dramatically better).
  • Nova Act shines as a robot-computer layer when you keep tasks atomic and add verification.

What's next for Nova Clerk

  • Multi-user / org mode: team inboxes, delegated roles, approval chains, and workspace policies.
  • Agent fleets: parallel workers handling multiple workflows (inbox, CRM, support queues) with a unified receipts ledger.
  • Richer approvals: granular “two-person rule,” policy templates, and risk scoring per action.
  • Streaming timeline + replay: step-by-step playback of decisions and UI actions for debugging and compliance.
  • Broader connectors: Slack, Jira, Salesforce, Zendesk, and internal tools via MCP—while keeping Nova Act as universal fallback.

Built With

  • amazonbedrock
  • amazoncloudwatch
  • amazonnova2lite
  • amazonnova2sonic
  • amazonnovaact
  • amazons3
  • awsiam
  • awssecretsmanager
  • docker
  • fastapi
  • googlecalendarapi
  • googlegmailapi
  • next.js
  • oauth2
  • openapi
  • playwright
  • python
  • react
  • tailwindcss
  • typescript
Share this project:

Updates