Skip to content

olathedev/wise-spend

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

53 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

πŸ’° WiseSpend

Transform Financial Fog Into Actionable Resilience Through AI-Powered Receipt Intelligence

Built for Comet Hackathon Powered by Opik Gemini 3

For judges: This README is structured so you can quickly see the problem, solution, architecture, all features (including the Grow page), and a suggested evaluation path. Use Product at a Glance and Demo Scenarios to know what to try in the app; use Evaluation Summary for scoring alignment.


🎯 The Problem

Most people don't realize what they're spending until it's too late.

  • "I'll just track it manually" β†’ Gets forgotten after 3 days
  • "I'll use a budgeting app" β†’ Too generic, doesn't understand context
  • "I already spent the money, why track it?" β†’ Missing the point entirely

The real issue: People don't see the trade-offs of their spending decisions. That $6.50 latte isn't just coffeeβ€”it's 15 minutes of your future retirement, or 2 weeks further from your house deposit.


πŸ’‘ Our Solution

WiseSpend is an agentic, AI-first financial coach that transforms "financial fog" into actionable resilience.

Instead of just tracking expenses, we:

  1. Scan receipts with Gemini 3 β†’ Multimodal AI understands what you bought and why it matters
  2. Calculate real goal impact β†’ Shows you: "That $100 purchase = 2 weeks further from your house deposit"
  3. Socratic coaching β†’ Asks thought-provoking questions, not generic advice
  4. Full observability β†’ Every reasoning step traced in Opik for continuous improvement

The receipt scan isn't about that ONE purchase β€” it's about building awareness so your NEXT 100 purchases are different.


πŸ—οΈ Architecture

System Overview

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                           WISESPEND                                    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚                                                                         β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚   Next.js   │────▢│   Express    │────▢│   Google GenAI Service  β”‚  β”‚
β”‚  β”‚   Frontend  │◀────│   Backend    │◀────│   (Gemini 3)            β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚        β”‚                    β”‚                         β”‚                 β”‚
β”‚        β”‚                    β–Ό                         β–Ό                 β”‚
β”‚        β”‚             β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”          β”‚
β”‚        β”‚             β”‚   MongoDB    β”‚         β”‚   Cloudinaryβ”‚          β”‚
β”‚        β”‚             β”‚   Database   β”‚         β”‚  (Receipts) β”‚          β”‚
β”‚        β”‚             β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜         β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜          β”‚
β”‚        β”‚                                             β”‚                  β”‚
β”‚        β”‚                    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                  β”‚
β”‚        β”‚                                 β–Ό                              β”‚
β”‚        β”‚                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                 β”‚
β”‚        └───────────────────▢│         OPIK            β”‚                 β”‚
β”‚           (View Traces)     β”‚  (Full Observability)   β”‚                 β”‚
β”‚                             β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                 β”‚
β”‚                                                                         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Receipt Analysis Reasoning Chain (Fully Traced in Opik)

    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚   RECEIPT SCAN   │───▢│  IMAGE ANALYSIS  │───▢│  ITEM EXTRACTION β”‚
    β”‚                  β”‚    β”‚                  β”‚    β”‚                  β”‚
    β”‚ User uploads     β”‚    β”‚ Gemini 3          β”‚    β”‚ Parse line items β”‚
    β”‚ receipt photo    β”‚    β”‚ Vision processes  β”‚    β”‚ Categorize items β”‚
    β”‚                  β”‚    β”‚                  β”‚    β”‚                  β”‚
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                                              β”‚
                                                              β–Ό
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚  GOAL IMPACT     │◀───│  SPENDING PATTERN│◀───│  CONTEXT ANALYSISβ”‚
    β”‚  CALCULATION     β”‚    β”‚  ANALYSIS        β”‚    β”‚                  β”‚
    β”‚                  β”‚    β”‚                  β”‚    β”‚ User history     β”‚
    β”‚ "This $100       β”‚    β”‚ "You usually     β”‚    β”‚ Financial goals  β”‚
    β”‚  purchase = 2    β”‚    β”‚  spend $X on     β”‚    β”‚ Monthly income   β”‚
    β”‚  weeks further   β”‚    β”‚  Fridays"        β”‚    β”‚                  β”‚
    β”‚  from goal"      β”‚    β”‚                  β”‚    β”‚                  β”‚
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                                              β”‚
                                                              β–Ό
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚  SOCRATIC COACH  │───▢│  INSIGHT          │───▢│  USER ACTION     β”‚
    β”‚  QUESTION        β”‚    β”‚  GENERATION       β”‚    β”‚                  β”‚
    β”‚                  β”‚    β”‚                  β”‚    β”‚ User sees        β”‚
    β”‚ "Is this worth   β”‚    β”‚ "15% spike in     β”‚    β”‚ trade-off, makes β”‚
    β”‚  delaying your   β”‚    β”‚  coffee spending  β”‚    β”‚ informed choice  β”‚
    β”‚  house deposit?" β”‚    β”‚  this week"       β”‚    β”‚                  β”‚
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
    
    ⬆️ Every step traced in Opik with nested parent-child spans

Grow Page Flow (Knowledge + Investments + Chat)

    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚   USER OPENS      │───▢│  GROW PAGE       │───▢│  TWO TABS        β”‚
    β”‚   GROW            β”‚    β”‚                  β”‚    β”‚                  β”‚
    β”‚   (sidebar)       β”‚    β”‚  Learning &      β”‚    β”‚  Knowledge       β”‚
    β”‚                   β”‚    β”‚  Investing hub   β”‚    β”‚  | Investments   β”‚
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                                              β”‚
           β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
           β”‚                                                  β”‚                                                  β”‚
           β–Ό                                                  β–Ό                                                  β–Ό
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚  KNOWLEDGE TAB   │───▢│  LITERACY CARDS  │───▢│  GENERATE QUIZ   │───▢│  GEMINI CURATES  │───▢│  USER TAKES QUIZ β”‚
    β”‚                  β”‚    β”‚  12 topics       β”‚    β”‚  (5 questions)   β”‚    β”‚  5 MC questions  β”‚    β”‚  Score + explain β”‚
    β”‚  50/30/20,       β”‚    β”‚  Click topic     β”‚    β”‚  On demand       β”‚    β”‚  Traced in Opik  β”‚    β”‚  "New quiz"      β”‚
    β”‚  compound, etc.  β”‚    β”‚                  β”‚    β”‚                  β”‚    β”‚                  β”‚    β”‚  anytime         β”‚
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
           β”‚
           β”‚
           β–Ό
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚  INVESTMENT TAB  │───▢│  ALPHA VANTAGE   │───▢│  CURATED LIST    β”‚
    β”‚                  β”‚    β”‚  Real-time data  β”‚    β”‚  ETFs, stocks    β”‚
    β”‚  Filters: type   β”‚    β”‚  Prices, % changeβ”‚    β”‚  Risk, min $     β”‚
    β”‚  Risk, sort      β”‚    β”‚                  β”‚    β”‚  Descriptions    β”‚
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                                              β”‚
                                                              β–Ό
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚  AI CHAT PANEL   │◀───│  CONTEXTUAL Q&A  │───▢│  USER ACTION     β”‚
    β”‚  (side of Grow)  β”‚    β”‚                  β”‚    β”‚                  β”‚
    β”‚                  β”‚    β”‚  "What is VTI?"  β”‚    β”‚  Informed about  β”‚
    β”‚  Same as AI      β”‚    β”‚  "Explain 50/30" β”‚    β”‚  concepts &      β”‚
    β”‚  Coach; Opik     β”‚    β”‚  Traced in Opik  β”‚    β”‚  opportunities   β”‚
    β”‚  traced          β”‚    β”‚                  β”‚    β”‚                  β”‚
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

    ⬆️ Quiz generation and Grow chat traced in Opik (chat-gemini-3, quiz generation spans)

Opik Tracing Architecture

OPIK INTEGRATION
β”œβ”€β”€ Trace Creation ──────────── Every AI call creates a trace
β”‚   β”œβ”€β”€ chat-gemini-3 ─────────── Chat (AI Coach + Grow panel)
β”‚   β”œβ”€β”€ analyze-receipt-image ─── Receipt analysis (main feature)
β”‚   └── generate-quizzes ──────── Grow: AI-generated literacy quizzes
β”‚
β”œβ”€β”€ Nested Spans ─────────────── Detailed reasoning chain
β”‚   β”œβ”€β”€ preprocess-receipt-image ─ Image preprocessing
β”‚   β”œβ”€β”€ gemini-multimodal-analysis ─ LLM vision analysis
β”‚   β”œβ”€β”€ calculate-goal-impact ─── Goal impact calculation
β”‚   └── (Grow) quiz curation ───── Quiz generation for Knowledge tab
β”‚
β”œβ”€β”€ Error Handling ───────────── All errors logged to traces
β”‚   └── Error metadata captured ─ Error type, message, stack
β”‚
β”œβ”€β”€ Token Usage Tracking ─────── Full cost observability
β”‚   └── promptTokens, completionTokens, totalTokens
β”‚
└── Graceful Shutdown ─────────── Auto-flush on SIGTERM/SIGINT
    └── Ensures all traces sent before exit

✨ Core Features

πŸ“Έ Multimodal Receipt Scanning

Snap a photo, understand the impact.

  • Gemini 3 Vision: Uses Gemini 3's multimodal capabilities to parse unstructured receipt images
  • Intent Recognition: Distinguishes "necessity" from "luxury" (not just OCR)
  • Automatic Categorization: Smart categorization of line items
  • Full Opik Tracing: Every step from image β†’ analysis β†’ insight is traced

πŸ€” Socratic Financial Coaching

Ask questions, not commands.

Instead of: "You spent too much on coffee"

We ask: "Is that $6.50 latte worth 15 minutes of your future retirement?"

  • Thought-Provoking Questions: Helps users discover their own financial insights
  • Goal-Aware: Every question references your actual financial goals
  • Pattern Recognition: Identifies spending patterns over time
  • Supportive but Firm: Tone tuned via Opik Agent Optimizer

🎯 Goal Impact Calculation

See the real cost of every purchase.

  • Real-Time Calculations: Shows how each purchase affects your goals
  • Visual Trade-offs: "That $100 purchase = 2 weeks further from your house deposit"
  • Historical Context: Compares current spending to past patterns
  • Actionable Insights: Not just data, but meaning

πŸ›‘οΈ Safety & Moderation (Opik Evaluation)

Ensures our coach never goes rogue.

  • ModerationMetric: Prevents suggesting high-risk investments
  • HallucinationMetric: Ensures accurate balance data
  • LLM-as-a-Judge: Continuous evaluation of AI responses
  • Regression Testing: 50+ edge-case financial scenarios in Opik

πŸ“Š Financial Dashboard

  • Wise Score: Your financial resilience score (0-100)
  • Monthly Spending: Real-time spending tracking
  • Emergency Fund: Safety net visualization
  • Transaction History: All scanned receipts with insights
  • Analytics: Spending patterns, trends, and predictions

πŸŽ“ Gamified Financial Learning (AI-Curated)

Learn financial concepts through interactive quizzes.

  • 12 Financial Literacy Topics: Covering budgeting, investing, debt management, and more

    • The 50/30/20 Rule
    • Compound Growth
    • Inflation vs. Cash
    • Emergency Fund Basics
    • Debt Snowball Method
    • Diversification
    • Automated Savings
    • Tax-Advantaged Accounts
    • Credit Score Basics
    • ROI Calculation
    • Budgeting Apps
    • Financial Goals (SMART framework)
  • Interactive Quiz System:

    • Click any literacy card to start a quiz
    • Multiple-choice questions with explanations
    • Progress tracking and completion rewards
    • AI-curated content tailored to your financial profile
  • Gamification Elements:

    • Visual progress indicators
    • Completion celebrations
    • Topic-based learning paths
    • Knowledge reinforcement through repetition

πŸ’Ό Investment Opportunity Curation

AI-powered investment suggestions based on real market data.

  • Real-Time Market Data: Powered by Alpha Vantage API

    • Live stock prices and ETF quotes
    • Daily change percentages
    • Market trends and top gainers
  • Curated Investment Opportunities:

    • ETFs: SPY (S&P 500), VTI (Total Stock Market), VEA (International), BND (Bonds), VNQ (Real Estate)
    • Major Stocks: Apple, Microsoft, Alphabet, and more
    • Risk Assessment: Low, Medium, Medium-High risk classifications
    • Minimum Investment: Shows accessible entry points ($1+)
  • Smart Filtering:

    • Filters by investment type (ETF, Stock, Bond, REIT)
    • Sorted by performance and risk level
    • Educational descriptions for each opportunity
    • Sector information and market insights
  • Safety Features:

    • ModerationMetric ensures no high-risk suggestions
    • Educational disclaimers
    • Encourages consultation with financial advisors
    • Past performance disclaimers

πŸ”„ Agent Optimization (Opik Agent Optimizer)

Continuously improving prompts.

  • Prompt Tuning: System prompts optimized for "supportive but firm" tone
  • A/B Testing: Different prompt variations tested
  • Performance Tracking: Completion rates, user satisfaction
  • Auto-Improvement: Prompts refined based on real user interactions

πŸ” Opik Integration (Deep)

WiseSpend showcases production-grade Opik integration with comprehensive tracing, evaluation, and optimization.

Opik integration β€” files for reviewers (traverse here):

File Role
backend/src/infrastructure/services/OpikService.ts Core Opik client: init, createTrace(), flush(), env config (OPIK_API_KEY, OPIK_PROJECT_NAME, OPIK_WORKSPACE)
backend/src/infrastructure/services/GoogleGenAIService.ts Traces for chat and receipt analysis reasoning chain; creates spans (e.g. analyze-receipt-image, chat trace), logs token usage and errors
backend/src/application/use-cases/FinancialAssistantChatUseCase.ts Trace financial-assistant-chat for AI Coach (and Grow panel) conversations
backend/src/application/use-cases/GenerateQuizzesUseCase.ts Trace generate-personalized-quizzes for Grow Knowledge tab quiz generation
backend/src/infrastructure/services/QuizCuratorAgent.ts Nested trace generate-quiz-for-concept; flat input/output for Opik LLM-as-a-Judge
backend/src/infrastructure/services/QuizEvaluator.ts Trace quiz-llm-judge-evaluation; LLM-as-a-Judge evaluation of quiz content (when EVALUATE_QUIZZES=true)
backend/src/index.ts Graceful shutdown: calls opikService.flush() on SIGTERM/SIGINT so traces are sent before exit

πŸ“Š Full Tracing Coverage

Every AI operation is traced:

Operation Trace Name Spans
Chat Conversation chat-gemini-3 LLM call, response processing
Receipt Analysis analyze-receipt-image Preprocessing, Vision analysis, Goal impact calculation

Trace Structure:

trace: analyze-receipt-image
β”œβ”€β”€ span: preprocess-receipt-image (type: general)
β”œβ”€β”€ span: gemini-multimodal-analysis (type: llm)
β”‚   β”œβ”€β”€ input: { model, prompt, image, temperature }
β”‚   β”œβ”€β”€ output: { analysis, fullAnalysis }
β”‚   └── metadata: { usage: { tokens }, provider: "google-genai" }
└── span: calculate-goal-impact (type: general)
    └── input: { analysis }

🎯 Evaluation Metrics

Built-in Opik Evaluators:

  • ModerationMetric: Ensures no high-risk investment suggestions
  • HallucinationMetric: Validates financial data accuracy
  • Custom Evaluators: Strategy alignment, personalization quality

Evaluation Workflow:

AI Response Generated
        β”‚
        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Opik Evaluation      β”‚
β”‚  β€’ ModerationMetric    β”‚
β”‚  β€’ HallucinationMetricβ”‚
β”‚  β€’ Custom Evaluators  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
        β”‚
        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Response Approved    β”‚
β”‚  or Flagged           β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸ”§ Agent Optimization

Opik Agent Optimizer Integration:

  • System Prompt Tuning: Optimized for "supportive but firm" coaching tone
  • Performance Tracking: Measures user engagement and goal completion rates
  • Iterative Improvement: Prompts refined based on real-world performance
  • Regression Testing: 50+ edge-case scenarios ensure stability

πŸ“ˆ Observability Features

  • Token Usage Tracking: Full cost observability for every LLM call
  • Error Tracking: All errors captured with full context
  • Performance Metrics: Response times, success rates
  • User Journey Mapping: Complete trace of user interactions

πŸš€ Quick Start

Prerequisites

  • Node.js 18+ and npm
  • MongoDB (local or Atlas)
  • Google Cloud account (for Gemini API)
  • Comet account (for Opik)

1. Backend Setup

cd backend

# Install dependencies
npm install

# Copy environment variables
cp .env.example .env

# Edit .env with your keys:
# - GEMINI_API_KEY (from Google Cloud)
# - OPIK_API_KEY (from Comet)
# - OPIK_PROJECT_NAME (e.g., "WiseSpend-Evals")
# - OPIK_WORKSPACE (your Comet workspace)
# - MONGODB_URI (your MongoDB connection string)
# - CLOUDINARY_* (for receipt image storage)

# Start development server
npm run dev

Backend runs on http://localhost:8000

2. Frontend Setup

cd frontend

# Install dependencies
npm install

# Copy environment variables
cp .env.example .env

# Edit .env with your keys:
# - NEXT_PUBLIC_API_URL=http://localhost:8000/api/v1
# - GOOGLE_CLIENT_ID (for OAuth)
# - GOOGLE_CLIENT_SECRET (for OAuth)
# - NEXTAUTH_SECRET (generate with: openssl rand -base64 32)
# - NEXTAUTH_URL=http://localhost:3000

# Start development server
npm run dev

Frontend runs on http://localhost:3000

3. Access the Application


🎬 Demo Scenarios

Scenario 1: First Receipt Scan

  1. User uploads coffee receipt β†’ Gemini analyzes image
  2. System extracts: "Coffee - $6.50"
  3. Goal impact calculated: "That's 15 minutes of your retirement fund"
  4. Socratic question: "Is this worth delaying your house deposit by 2 weeks?"
  5. Full trace in Opik: See the complete reasoning chain

Scenario 2: Pattern Recognition

  1. User scans 5 receipts β†’ System identifies pattern
  2. Insight generated: "You spend 15% more on coffee when stressed"
  3. Coaching question: "Noticed a spike in convenience food spending. What's driving this?"
  4. User reflects β†’ Makes informed choice next time

Scenario 3: Goal Achievement

  1. User sets goal: "Save $10,000 for house deposit"
  2. Scans receipts β†’ Each purchase shows impact
  3. Visual progress: "You're 2 weeks closer to your goal!"
  4. Motivation: Seeing real progress encourages continued saving

Scenario 4: Grow Page β€” Knowledge + Quiz

  1. User opens Grow (sidebar) β†’ Defaults to Knowledge tab
  2. Sees financial literacy cards β†’ 12 topics (50/30/20, compound growth, emergency fund, etc.)
  3. Clicks a topic β†’ Generate quiz β†’ AI returns 5 multiple-choice questions
  4. Completes quiz β†’ Sees score and explanations; can generate a new quiz anytime

Scenario 5: Grow Page β€” Investment Discovery

  1. User opens Grow β†’ Switches to Investment Suggestions tab
  2. Sees curated opportunities β†’ ETFs (SPY, VTI, VEA, BND, VNQ), stocks; real-time prices (Alpha Vantage)
  3. Filters by type β†’ ETF, Stock, Bond, REIT; reviews risk levels and descriptions
  4. Uses side AI panel β†’ Asks follow-up questions about a ticker or concept
  5. Makes informed decision β†’ With full context and disclaimers

πŸ“‹ Evaluation Summary

Criteria Implementation
βœ… True Agent Autonomous reasoning chain: Receipt β†’ Analysis β†’ Goal Impact β†’ Coaching
⭐ Deep Opik Integration Every AI call traced with nested spans, full observability
βœ… Evaluation-Driven ModerationMetric, HallucinationMetric, custom evaluators
βœ… Agent Optimization Opik Agent Optimizer tunes prompts for optimal coaching tone
βœ… Production Ready Error handling, graceful shutdown, comprehensive logging
βœ… Novel Use Case Multimodal receipt analysis with goal impact calculation
βœ… User Value Transforms "financial fog" into actionable insights
βœ… Gamified Learning πŸŽ“ Interactive financial literacy quizzes (12 topics)
βœ… Investment Curation πŸ’Ό Real-time investment opportunities with market data
βœ… Full Stack Next.js frontend, Express backend, MongoDB, Cloudinary
βœ… Authentication Google OAuth via NextAuth, protected routes

πŸ“ Key Files

backend/
β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ infrastructure/
β”‚   β”‚   └── services/
β”‚   β”‚       β”œβ”€β”€ GoogleGenAIService.ts    # πŸ€– Gemini integration with Opik tracing
β”‚   β”‚       └── OpikService.ts           # πŸ” Opik client wrapper
β”‚   β”œβ”€β”€ application/
β”‚   β”‚   └── use-cases/
β”‚   β”‚       β”œβ”€β”€ AnalyzeReceiptUseCase.ts  # πŸ“Έ Receipt analysis logic
β”‚   β”‚       β”œβ”€β”€ ChatUseCase.ts            # πŸ’¬ Socratic coaching
β”‚   β”‚       └── ComputeWiseScoreUseCase.ts # πŸ“Š Financial resilience score
β”‚   └── presentation/
β”‚       └── controllers/
β”‚           β”œβ”€β”€ ReceiptController.ts      # Receipt upload & analysis
β”‚           └── AIController.ts           # AI chat endpoints
β”‚
frontend/
β”œβ”€β”€ app/
β”‚   β”œβ”€β”€ dashboard/
β”‚   β”‚   β”œβ”€β”€ page.tsx                     # πŸ“Š Main dashboard
β”‚   β”‚   β”œβ”€β”€ transactions/page.tsx        # πŸ“ Transaction history
β”‚   β”‚   β”œβ”€β”€ analytics/page.tsx           # πŸ“ˆ Spending analytics
β”‚   β”‚   β”œβ”€β”€ goals/page.tsx               # 🎯 Financial goals
β”‚   β”‚   β”œβ”€β”€ grow/page.tsx                # 🌱 Grow page β€” Knowledge tab (literacy + AI quizzes) + Investment Suggestions tab
β”‚   β”‚   └── ai-coach/page.tsx            # πŸ€” Socratic coach chat
β”‚   └── scan/page.tsx                    # πŸ“Έ Receipt scanning
β”œβ”€β”€ components/
β”‚   β”œβ”€β”€ dashboard/
β”‚   β”‚   β”œβ”€β”€ FinancialSummaryCards.tsx   # Summary cards
β”‚   β”‚   β”œβ”€β”€ SocraticCoach.tsx            # Coach insights
β”‚   β”‚   β”œβ”€β”€ TransactionList.tsx          # Recent transactions
β”‚   β”‚   β”œβ”€β”€ FinancialLiteracyCards.tsx   # πŸŽ“ Gamified learning cards
β”‚   β”‚   β”œβ”€β”€ QuizModal.tsx                # πŸ“ Interactive quiz component
β”‚   β”‚   β”œβ”€β”€ InvestmentSuggestions.tsx    # πŸ’Ό Investment opportunities
β”‚   β”‚   └── FinancialInsights.tsx        # πŸ’‘ AI-generated insights
β”‚   └── receipt/
β”‚       └── AnalysisResultModal.tsx      # Receipt analysis results
└── services/
    β”œβ”€β”€ receiptService.ts                 # Receipt API calls
    β”œβ”€β”€ aiService.ts                      # AI chat API calls
    └── investmentService.ts              # πŸ’Ό Investment data (Alpha Vantage)

πŸ”§ Technology Stack

Frontend

  • Next.js 14 - React framework with App Router
  • TypeScript - Type safety
  • Tailwind CSS - Styling
  • NextAuth - Authentication
  • Framer Motion - Animations

Backend

  • Express.js - REST API server
  • TypeScript - Type safety
  • MongoDB - Database (via Mongoose)
  • Cloudinary - Receipt image storage

AI & Observability

  • Google Gemini 3 - Multimodal AI (vision + text)
  • Opik - Full observability and evaluation
  • Opik Agent Optimizer - Prompt optimization

πŸ“Š Opik Dashboard

View your traces:

  1. Go to https://www.comet.com/opik
  2. Navigate to your workspace
  3. Open project: WiseSpend-Evals (or your configured project)
  4. See traces:
    • chat-gemini-3 - Chat conversations
    • analyze-receipt-image - Receipt analysis (with nested spans)

Trace Details:

  • Click any trace to see:
    • Full input/output
    • Nested spans (preprocessing β†’ analysis β†’ goal impact)
    • Token usage
    • Timing information
    • Error logs (if any)

πŸ§ͺ Testing Opik Integration

Test Receipt Analysis

# Start backend
cd backend && npm run dev

# In another terminal, test receipt analysis
curl -X POST http://localhost:8000/api/v1/ai/receipt/analyze \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -F "image=@/path/to/receipt.jpg"

Test Chat Endpoint

curl -X POST http://localhost:8000/api/v1/ai/chat \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "temperature": 0.7,
    "maxTokens": 100
  }'

View Traces in Opik

After making API calls:

  1. Wait 10-30 seconds (traces are batched)
  2. Go to Opik dashboard
  3. Refresh and see your traces!

πŸ’‘ Philosophy

"The receipt scan isn't about that ONE purchase β€” it's about building awareness so your NEXT 100 purchases are different."

WiseSpend transforms hindsight into foresight. By showing you the real cost of every purchase, you start making informed choices before you spend, not after.


πŸ“ License

Built for the Comet "Commit to Change" AI Agents Hackathon


πŸ™ Acknowledgments

  • Google Gemini - Multimodal AI capabilities
  • Comet Opik - Observability and evaluation platform
  • Comet Team - Hackathon organizers and support

Built with πŸ’° for the Comet AI Agents Hackathon

About

WiseSpend is an AI-first, multimodal financial agent that turns receipt images into real-time insights, proactive coaching, and goal-aligned spending decisions.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages