Transform Financial Fog Into Actionable Resilience Through AI-Powered Receipt Intelligence
For judges: This README is structured so you can quickly see the problem, solution, architecture, all features (including the Grow page), and a suggested evaluation path. Use Product at a Glance and Demo Scenarios to know what to try in the app; use Evaluation Summary for scoring alignment.
Most people don't realize what they're spending until it's too late.
- "I'll just track it manually" β Gets forgotten after 3 days
- "I'll use a budgeting app" β Too generic, doesn't understand context
- "I already spent the money, why track it?" β Missing the point entirely
The real issue: People don't see the trade-offs of their spending decisions. That $6.50 latte isn't just coffeeβit's 15 minutes of your future retirement, or 2 weeks further from your house deposit.
WiseSpend is an agentic, AI-first financial coach that transforms "financial fog" into actionable resilience.
Instead of just tracking expenses, we:
- Scan receipts with Gemini 3 β Multimodal AI understands what you bought and why it matters
- Calculate real goal impact β Shows you: "That $100 purchase = 2 weeks further from your house deposit"
- Socratic coaching β Asks thought-provoking questions, not generic advice
- Full observability β Every reasoning step traced in Opik for continuous improvement
The receipt scan isn't about that ONE purchase β it's about building awareness so your NEXT 100 purchases are different.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β WISESPEND β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β βββββββββββββββ ββββββββββββββββ βββββββββββββββββββββββββββ β
β β Next.js ββββββΆβ Express ββββββΆβ Google GenAI Service β β
β β Frontend βββββββ Backend βββββββ (Gemini 3) β β
β βββββββββββββββ ββββββββββββββββ βββββββββββββββββββββββββββ β
β β β β β
β β βΌ βΌ β
β β ββββββββββββββββ βββββββββββββββ β
β β β MongoDB β β Cloudinaryβ β
β β β Database β β (Receipts) β β
β β ββββββββββββββββ βββββββββββββββ β
β β β β
β β ββββββββββββββ¬ββββββββββββ β
β β βΌ β
β β βββββββββββββββββββββββββββ β
β βββββββββββββββββββββΆβ OPIK β β
β (View Traces) β (Full Observability) β β
β βββββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β RECEIPT SCAN βββββΆβ IMAGE ANALYSIS βββββΆβ ITEM EXTRACTION β
β β β β β β
β User uploads β β Gemini 3 β β Parse line items β
β receipt photo β β Vision processes β β Categorize items β
β β β β β β
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β GOAL IMPACT ββββββ SPENDING PATTERNββββββ CONTEXT ANALYSISβ
β CALCULATION β β ANALYSIS β β β
β β β β β User history β
β "This $100 β β "You usually β β Financial goals β
β purchase = 2 β β spend $X on β β Monthly income β
β weeks further β β Fridays" β β β
β from goal" β β β β β
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β SOCRATIC COACH βββββΆβ INSIGHT βββββΆβ USER ACTION β
β QUESTION β β GENERATION β β β
β β β β β User sees β
β "Is this worth β β "15% spike in β β trade-off, makes β
β delaying your β β coffee spending β β informed choice β
β house deposit?" β β this week" β β β
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β¬οΈ Every step traced in Opik with nested parent-child spans
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β USER OPENS βββββΆβ GROW PAGE βββββΆβ TWO TABS β
β GROW β β β β β
β (sidebar) β β Learning & β β Knowledge β
β β β Investing hub β β | Investments β
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββ¬ββββββββββ
β
ββββββββββββββββββββββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββ
β β β
βΌ βΌ βΌ
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β KNOWLEDGE TAB βββββΆβ LITERACY CARDS βββββΆβ GENERATE QUIZ βββββΆβ GEMINI CURATES βββββΆβ USER TAKES QUIZ β
β β β 12 topics β β (5 questions) β β 5 MC questions β β Score + explain β
β 50/30/20, β β Click topic β β On demand β β Traced in Opik β β "New quiz" β
β compound, etc. β β β β β β β β anytime β
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β
β
βΌ
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β INVESTMENT TAB βββββΆβ ALPHA VANTAGE βββββΆβ CURATED LIST β
β β β Real-time data β β ETFs, stocks β
β Filters: type β β Prices, % changeβ β Risk, min $ β
β Risk, sort β β β β Descriptions β
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββ¬ββββββββββ
β
βΌ
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β AI CHAT PANEL ββββββ CONTEXTUAL Q&A βββββΆβ USER ACTION β
β (side of Grow) β β β β β
β β β "What is VTI?" β β Informed about β
β Same as AI β β "Explain 50/30" β β concepts & β
β Coach; Opik β β Traced in Opik β β opportunities β
β traced β β β β β
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β¬οΈ Quiz generation and Grow chat traced in Opik (chat-gemini-3, quiz generation spans)
OPIK INTEGRATION
βββ Trace Creation ββββββββββββ Every AI call creates a trace
β βββ chat-gemini-3 βββββββββββ Chat (AI Coach + Grow panel)
β βββ analyze-receipt-image βββ Receipt analysis (main feature)
β βββ generate-quizzes ββββββββ Grow: AI-generated literacy quizzes
β
βββ Nested Spans βββββββββββββββ Detailed reasoning chain
β βββ preprocess-receipt-image β Image preprocessing
β βββ gemini-multimodal-analysis β LLM vision analysis
β βββ calculate-goal-impact βββ Goal impact calculation
β βββ (Grow) quiz curation βββββ Quiz generation for Knowledge tab
β
βββ Error Handling βββββββββββββ All errors logged to traces
β βββ Error metadata captured β Error type, message, stack
β
βββ Token Usage Tracking βββββββ Full cost observability
β βββ promptTokens, completionTokens, totalTokens
β
βββ Graceful Shutdown βββββββββββ Auto-flush on SIGTERM/SIGINT
βββ Ensures all traces sent before exit
Snap a photo, understand the impact.
- Gemini 3 Vision: Uses Gemini 3's multimodal capabilities to parse unstructured receipt images
- Intent Recognition: Distinguishes "necessity" from "luxury" (not just OCR)
- Automatic Categorization: Smart categorization of line items
- Full Opik Tracing: Every step from image β analysis β insight is traced
Ask questions, not commands.
Instead of: "You spent too much on coffee"
We ask: "Is that $6.50 latte worth 15 minutes of your future retirement?"
- Thought-Provoking Questions: Helps users discover their own financial insights
- Goal-Aware: Every question references your actual financial goals
- Pattern Recognition: Identifies spending patterns over time
- Supportive but Firm: Tone tuned via Opik Agent Optimizer
See the real cost of every purchase.
- Real-Time Calculations: Shows how each purchase affects your goals
- Visual Trade-offs: "That $100 purchase = 2 weeks further from your house deposit"
- Historical Context: Compares current spending to past patterns
- Actionable Insights: Not just data, but meaning
Ensures our coach never goes rogue.
- ModerationMetric: Prevents suggesting high-risk investments
- HallucinationMetric: Ensures accurate balance data
- LLM-as-a-Judge: Continuous evaluation of AI responses
- Regression Testing: 50+ edge-case financial scenarios in Opik
- Wise Score: Your financial resilience score (0-100)
- Monthly Spending: Real-time spending tracking
- Emergency Fund: Safety net visualization
- Transaction History: All scanned receipts with insights
- Analytics: Spending patterns, trends, and predictions
Learn financial concepts through interactive quizzes.
-
12 Financial Literacy Topics: Covering budgeting, investing, debt management, and more
- The 50/30/20 Rule
- Compound Growth
- Inflation vs. Cash
- Emergency Fund Basics
- Debt Snowball Method
- Diversification
- Automated Savings
- Tax-Advantaged Accounts
- Credit Score Basics
- ROI Calculation
- Budgeting Apps
- Financial Goals (SMART framework)
-
Interactive Quiz System:
- Click any literacy card to start a quiz
- Multiple-choice questions with explanations
- Progress tracking and completion rewards
- AI-curated content tailored to your financial profile
-
Gamification Elements:
- Visual progress indicators
- Completion celebrations
- Topic-based learning paths
- Knowledge reinforcement through repetition
AI-powered investment suggestions based on real market data.
-
Real-Time Market Data: Powered by Alpha Vantage API
- Live stock prices and ETF quotes
- Daily change percentages
- Market trends and top gainers
-
Curated Investment Opportunities:
- ETFs: SPY (S&P 500), VTI (Total Stock Market), VEA (International), BND (Bonds), VNQ (Real Estate)
- Major Stocks: Apple, Microsoft, Alphabet, and more
- Risk Assessment: Low, Medium, Medium-High risk classifications
- Minimum Investment: Shows accessible entry points ($1+)
-
Smart Filtering:
- Filters by investment type (ETF, Stock, Bond, REIT)
- Sorted by performance and risk level
- Educational descriptions for each opportunity
- Sector information and market insights
-
Safety Features:
- ModerationMetric ensures no high-risk suggestions
- Educational disclaimers
- Encourages consultation with financial advisors
- Past performance disclaimers
Continuously improving prompts.
- Prompt Tuning: System prompts optimized for "supportive but firm" tone
- A/B Testing: Different prompt variations tested
- Performance Tracking: Completion rates, user satisfaction
- Auto-Improvement: Prompts refined based on real user interactions
WiseSpend showcases production-grade Opik integration with comprehensive tracing, evaluation, and optimization.
Opik integration β files for reviewers (traverse here):
| File | Role |
|---|---|
backend/src/infrastructure/services/OpikService.ts |
Core Opik client: init, createTrace(), flush(), env config (OPIK_API_KEY, OPIK_PROJECT_NAME, OPIK_WORKSPACE) |
backend/src/infrastructure/services/GoogleGenAIService.ts |
Traces for chat and receipt analysis reasoning chain; creates spans (e.g. analyze-receipt-image, chat trace), logs token usage and errors |
backend/src/application/use-cases/FinancialAssistantChatUseCase.ts |
Trace financial-assistant-chat for AI Coach (and Grow panel) conversations |
backend/src/application/use-cases/GenerateQuizzesUseCase.ts |
Trace generate-personalized-quizzes for Grow Knowledge tab quiz generation |
backend/src/infrastructure/services/QuizCuratorAgent.ts |
Nested trace generate-quiz-for-concept; flat input/output for Opik LLM-as-a-Judge |
backend/src/infrastructure/services/QuizEvaluator.ts |
Trace quiz-llm-judge-evaluation; LLM-as-a-Judge evaluation of quiz content (when EVALUATE_QUIZZES=true) |
backend/src/index.ts |
Graceful shutdown: calls opikService.flush() on SIGTERM/SIGINT so traces are sent before exit |
Every AI operation is traced:
| Operation | Trace Name | Spans |
|---|---|---|
| Chat Conversation | chat-gemini-3 |
LLM call, response processing |
| Receipt Analysis | analyze-receipt-image |
Preprocessing, Vision analysis, Goal impact calculation |
Trace Structure:
trace: analyze-receipt-image
βββ span: preprocess-receipt-image (type: general)
βββ span: gemini-multimodal-analysis (type: llm)
β βββ input: { model, prompt, image, temperature }
β βββ output: { analysis, fullAnalysis }
β βββ metadata: { usage: { tokens }, provider: "google-genai" }
βββ span: calculate-goal-impact (type: general)
βββ input: { analysis }Built-in Opik Evaluators:
- ModerationMetric: Ensures no high-risk investment suggestions
- HallucinationMetric: Validates financial data accuracy
- Custom Evaluators: Strategy alignment, personalization quality
Evaluation Workflow:
AI Response Generated
β
βΌ
βββββββββββββββββββββββββ
β Opik Evaluation β
β β’ ModerationMetric β
β β’ HallucinationMetricβ
β β’ Custom Evaluators β
βββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββ
β Response Approved β
β or Flagged β
βββββββββββββββββββββββββ
Opik Agent Optimizer Integration:
- System Prompt Tuning: Optimized for "supportive but firm" coaching tone
- Performance Tracking: Measures user engagement and goal completion rates
- Iterative Improvement: Prompts refined based on real-world performance
- Regression Testing: 50+ edge-case scenarios ensure stability
- Token Usage Tracking: Full cost observability for every LLM call
- Error Tracking: All errors captured with full context
- Performance Metrics: Response times, success rates
- User Journey Mapping: Complete trace of user interactions
- Node.js 18+ and npm
- MongoDB (local or Atlas)
- Google Cloud account (for Gemini API)
- Comet account (for Opik)
cd backend
# Install dependencies
npm install
# Copy environment variables
cp .env.example .env
# Edit .env with your keys:
# - GEMINI_API_KEY (from Google Cloud)
# - OPIK_API_KEY (from Comet)
# - OPIK_PROJECT_NAME (e.g., "WiseSpend-Evals")
# - OPIK_WORKSPACE (your Comet workspace)
# - MONGODB_URI (your MongoDB connection string)
# - CLOUDINARY_* (for receipt image storage)
# Start development server
npm run devBackend runs on http://localhost:8000
cd frontend
# Install dependencies
npm install
# Copy environment variables
cp .env.example .env
# Edit .env with your keys:
# - NEXT_PUBLIC_API_URL=http://localhost:8000/api/v1
# - GOOGLE_CLIENT_ID (for OAuth)
# - GOOGLE_CLIENT_SECRET (for OAuth)
# - NEXTAUTH_SECRET (generate with: openssl rand -base64 32)
# - NEXTAUTH_URL=http://localhost:3000
# Start development server
npm run devFrontend runs on http://localhost:3000
- Frontend: http://localhost:3000
- Backend API: http://localhost:8000/api/v1
- Opik Dashboard: https://www.comet.com/opik
- User uploads coffee receipt β Gemini analyzes image
- System extracts: "Coffee - $6.50"
- Goal impact calculated: "That's 15 minutes of your retirement fund"
- Socratic question: "Is this worth delaying your house deposit by 2 weeks?"
- Full trace in Opik: See the complete reasoning chain
- User scans 5 receipts β System identifies pattern
- Insight generated: "You spend 15% more on coffee when stressed"
- Coaching question: "Noticed a spike in convenience food spending. What's driving this?"
- User reflects β Makes informed choice next time
- User sets goal: "Save $10,000 for house deposit"
- Scans receipts β Each purchase shows impact
- Visual progress: "You're 2 weeks closer to your goal!"
- Motivation: Seeing real progress encourages continued saving
- User opens Grow (sidebar) β Defaults to Knowledge tab
- Sees financial literacy cards β 12 topics (50/30/20, compound growth, emergency fund, etc.)
- Clicks a topic β Generate quiz β AI returns 5 multiple-choice questions
- Completes quiz β Sees score and explanations; can generate a new quiz anytime
- User opens Grow β Switches to Investment Suggestions tab
- Sees curated opportunities β ETFs (SPY, VTI, VEA, BND, VNQ), stocks; real-time prices (Alpha Vantage)
- Filters by type β ETF, Stock, Bond, REIT; reviews risk levels and descriptions
- Uses side AI panel β Asks follow-up questions about a ticker or concept
- Makes informed decision β With full context and disclaimers
| Criteria | Implementation |
|---|---|
| β True Agent | Autonomous reasoning chain: Receipt β Analysis β Goal Impact β Coaching |
| β Deep Opik Integration | Every AI call traced with nested spans, full observability |
| β Evaluation-Driven | ModerationMetric, HallucinationMetric, custom evaluators |
| β Agent Optimization | Opik Agent Optimizer tunes prompts for optimal coaching tone |
| β Production Ready | Error handling, graceful shutdown, comprehensive logging |
| β Novel Use Case | Multimodal receipt analysis with goal impact calculation |
| β User Value | Transforms "financial fog" into actionable insights |
| β Gamified Learning | π Interactive financial literacy quizzes (12 topics) |
| β Investment Curation | πΌ Real-time investment opportunities with market data |
| β Full Stack | Next.js frontend, Express backend, MongoDB, Cloudinary |
| β Authentication | Google OAuth via NextAuth, protected routes |
backend/
βββ src/
β βββ infrastructure/
β β βββ services/
β β βββ GoogleGenAIService.ts # π€ Gemini integration with Opik tracing
β β βββ OpikService.ts # π Opik client wrapper
β βββ application/
β β βββ use-cases/
β β βββ AnalyzeReceiptUseCase.ts # πΈ Receipt analysis logic
β β βββ ChatUseCase.ts # π¬ Socratic coaching
β β βββ ComputeWiseScoreUseCase.ts # π Financial resilience score
β βββ presentation/
β βββ controllers/
β βββ ReceiptController.ts # Receipt upload & analysis
β βββ AIController.ts # AI chat endpoints
β
frontend/
βββ app/
β βββ dashboard/
β β βββ page.tsx # π Main dashboard
β β βββ transactions/page.tsx # π Transaction history
β β βββ analytics/page.tsx # π Spending analytics
β β βββ goals/page.tsx # π― Financial goals
β β βββ grow/page.tsx # π± Grow page β Knowledge tab (literacy + AI quizzes) + Investment Suggestions tab
β β βββ ai-coach/page.tsx # π€ Socratic coach chat
β βββ scan/page.tsx # πΈ Receipt scanning
βββ components/
β βββ dashboard/
β β βββ FinancialSummaryCards.tsx # Summary cards
β β βββ SocraticCoach.tsx # Coach insights
β β βββ TransactionList.tsx # Recent transactions
β β βββ FinancialLiteracyCards.tsx # π Gamified learning cards
β β βββ QuizModal.tsx # π Interactive quiz component
β β βββ InvestmentSuggestions.tsx # πΌ Investment opportunities
β β βββ FinancialInsights.tsx # π‘ AI-generated insights
β βββ receipt/
β βββ AnalysisResultModal.tsx # Receipt analysis results
βββ services/
βββ receiptService.ts # Receipt API calls
βββ aiService.ts # AI chat API calls
βββ investmentService.ts # πΌ Investment data (Alpha Vantage)
- Next.js 14 - React framework with App Router
- TypeScript - Type safety
- Tailwind CSS - Styling
- NextAuth - Authentication
- Framer Motion - Animations
- Express.js - REST API server
- TypeScript - Type safety
- MongoDB - Database (via Mongoose)
- Cloudinary - Receipt image storage
- Google Gemini 3 - Multimodal AI (vision + text)
- Opik - Full observability and evaluation
- Opik Agent Optimizer - Prompt optimization
View your traces:
- Go to https://www.comet.com/opik
- Navigate to your workspace
- Open project:
WiseSpend-Evals(or your configured project) - See traces:
chat-gemini-3- Chat conversationsanalyze-receipt-image- Receipt analysis (with nested spans)
Trace Details:
- Click any trace to see:
- Full input/output
- Nested spans (preprocessing β analysis β goal impact)
- Token usage
- Timing information
- Error logs (if any)
# Start backend
cd backend && npm run dev
# In another terminal, test receipt analysis
curl -X POST http://localhost:8000/api/v1/ai/receipt/analyze \
-H "Authorization: Bearer YOUR_JWT_TOKEN" \
-F "image=@/path/to/receipt.jpg"curl -X POST http://localhost:8000/api/v1/ai/chat \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "user", "content": "What is the capital of France?"}
],
"temperature": 0.7,
"maxTokens": 100
}'After making API calls:
- Wait 10-30 seconds (traces are batched)
- Go to Opik dashboard
- Refresh and see your traces!
"The receipt scan isn't about that ONE purchase β it's about building awareness so your NEXT 100 purchases are different."
WiseSpend transforms hindsight into foresight. By showing you the real cost of every purchase, you start making informed choices before you spend, not after.
Built for the Comet "Commit to Change" AI Agents Hackathon
- Google Gemini - Multimodal AI capabilities
- Comet Opik - Observability and evaluation platform
- Comet Team - Hackathon organizers and support
Built with π° for the Comet AI Agents Hackathon