Senior-friendly daily check-in app with:

• webcam capture and facial symmetry analysis
• optional video heart-rate estimation (VHR / open-rppg)
• live voice screening (Gemini Live) + transcript parsing
• triage classification (Green, Yellow, Red)
• doctor dashboard and AI-generated senior summary

This repository contains:

• main product app (frontend/ + backend/)
• a separate VHR spike prototype (VHR/) kept for reference

1. Project Overview
2. Repository Layout
3. Tech Stack
4. How the App Works
5. Quick Start
6. Configuration
7. API Reference
8. Database and Data Model
9. Frontend Notes
10. Backend Notes
11. Testing and Validation
12. Troubleshooting
13. Additional Docs

The main app provides two role-based experiences:

• senior: daily check-in flow with camera + voice screening
• doctor: dashboard analytics, senior list, and AI summary generation

Primary flow (senior):

1. user starts check-in (POST /checkins/start)
2. frontend records ~10s camera clip and uploads it (POST /checkins/{id}/upload)
3. backend runs facial symmetry + VHR analysis
4. live voice screening captures responses
5. frontend posts responses/transcript (POST /screenings, then POST /checkins/{id}/complete)
6. backend computes triage and persists result

.
├── frontend/                     # Vite + React client (main app)
│   ├── src/
│   │   ├── components/           # Login, SeniorCheckin, DoctorDashboard
│   │   ├── hooks/                # useAuth, useCheckin, useDoctorDashboard
│   │   └── lib/                  # API helpers, audio helpers, screening constants
│   ├── public/senicarelogo.png
│   └── .env.example
├── backend/                      # FastAPI API (main app)
│   └── app/
│       ├── routes/               # auth, checkins, dashboard, seniors, reports...
│       ├── services/             # triage, AI summary, facial symmetry
│       ├── models/               # Pydantic request/response schemas
│       ├── vhr/                  # open-rppg video HR pipeline
│       ├── auth.py               # JWT + password auth logic
│       ├── db.py                 # Mongo connection helpers
│       └── main.py               # app setup + route registration
├── vision/                       # standalone facial symmetry analyzer module
├── run_facial_symmetry_checkin.py# local CLI diagnostic capture script
├── mongo/init-mongo.js           # Mongo init helper script
├── ARCHITECTURE.md               # high-level architecture notes
├── MONGODB_SCHEMA_SIMPLE.md      # schema and query examples
└── VHR/                          # separate spike demo (Next.js + FastAPI)

• React 18 + Vite 5
• Tailwind CSS
• @google/genai for Gemini Live session from browser
• browser APIs: MediaRecorder, getUserMedia, SpeechRecognition

• FastAPI + Uvicorn
• MongoDB + PyMongo
• JWT auth (python-jose) + password hashing (passlib)
• Gemini server SDK (google-genai) for ephemeral token + report summaries
• MediaPipe + OpenCV for facial symmetry
• open-rppg + ffmpeg/ffprobe for video heart-rate estimation

• Register/Login endpoints issue JWT access tokens.
• Frontend stores JWT in localStorage (guardian_checkin.jwt).
• /me returns role/profile used to route UI:
  • doctor -> Doctor Dashboard
  • others -> Senior Check-in

• startVoice() in useCheckin orchestrates:
  • create check-in
  • request ephemeral Gemini token from backend
  • connect Gemini Live audio session
  • record/upload 10s camera clip
  • collect speech responses
  • submit screening + complete check-in
• Triage logic combines symptom answers and facial symmetry status.

• Dashboard pulls:
  • /dashboard/analytics
  • /dashboard/seniors
• Doctor can request AI summary:
  • POST /reports/senior-summary

• Node.js 18+ (20 recommended)
• Python 3.10+
• MongoDB instance (local or Atlas)
• ffmpeg + ffprobe installed (recommended for VHR reliability)

macOS:

brew install ffmpeg

cd backend
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000

Backend runs at http://localhost:8000.

cd frontend
npm install
npm run dev

Frontend runs at http://localhost:5173.

• GET http://localhost:8000/health
• open http://localhost:5173

Registration creates senior users by default. To access doctor routes/UI, update a user role in Mongo:

use guardian
db.users.updateOne(
  { email: "[email protected]" },
  { $set: { role: "doctor" } }
)

File: frontend/.env.example

VITE_API_BASE=http://localhost:8000

No checked-in backend/.env.example currently. Create backend/.env (or set shell env vars) with:

# Mongo
MONGODB_URI=mongodb://localhost:27017/guardian
# or use split fields below instead of MONGODB_URI:
MONGO_HOST=localhost
MONGO_PORT=27017
MONGO_DB=guardian
MONGO_USER=
MONGO_PASSWORD=
MONGO_AUTH_SOURCE=guardian
MONGO_SERVER_SELECTION_TIMEOUT_MS=5000
MONGO_CONNECT_TIMEOUT_MS=5000

# Auth
JWT_SECRET=replace-with-a-strong-secret
JWT_ALG=HS256
JWT_EXPIRES_MIN=60
REQUIRE_AUTH=false

# Gemini
GEMINI_API_KEY=replace-with-real-key

# VHR/open-rppg (optional tuning)
OPEN_RPPG_MODEL=FacePhys.rlap
OPEN_RPPG_MAX_WIDTH=640

Notes:

• JWT_SECRET is required for token issuance/verification.
• GEMINI_API_KEY is required for:
  • /auth/ephemeral
  • /reports/senior-summary AI summaries

Base URL: http://localhost:8000

• GET /health
  • returns app health, mongo status, auth flags, runtime metadata

• POST /auth/register
  • body: { firstName, lastName, email, password }
• POST /auth/login
  • body: { email, password }
  • returns { access_token, token_type }
• GET /me
  • requires bearer token
• GET /auth/ephemeral
• POST /auth/ephemeral
  • returns short-lived Gemini token for frontend live session

• POST /checkins/start (auth required)
  • creates in-progress check-in
• POST /checkins/{checkin_id}/upload
  • multipart upload
  • accepts video, optional audio, optional frames[], optional metadata
  • runs facial symmetry + VHR analysis
• POST /checkins/{checkin_id}/complete
  • body: { answers, transcript }
  • validates completeness and computes triage
• GET /checkins/{checkin_id}
  • returns detailed check-in model
• GET /checkins/{checkin_id}/validate
  • pre-completion data completeness check
• POST /checkins/{checkin_id}/auto-complete
  • auto-complete abandoned in-progress session with existing data
• POST /checkins/cleanup-abandoned?max_age_hours=24
  • bulk auto-complete stale sessions

• POST /screenings
  • create screening session linked to check-in
• GET /screenings/{session_id}
  • reads in-memory screening cache (not persistent lookup)

• GET /seniors/{senior_id}/checkins
  • query params: from_date, to_date, include_incomplete
• GET /seniors/{senior_id}/baseline
• POST /seniors/{senior_id}/summaries/weekly
• GET /seniors/{senior_id}/summaries/weekly?week_start=...
• GET /seniors/{senior_id}/alerts

• GET /dashboard/analytics
• GET /dashboard/seniors

• POST /reports/senior-summary
  • AI-generated summary from provided overview/checkins payload

• POST /alerts/test
  • creates test alert record in memory

Main collections used by backend:

• users
• checkin_history
• screenings

Key points:

• Check-ins are persisted in Mongo (checkin_history).
• Some app data is intentionally in-memory in current implementation:
  • alerts store (ALERTS)
  • weekly summaries (WEEKLY_SUMMARIES)
  • baseline cache (BASELINES)
  • screening GET cache (SCREENINGS)
• In-memory data resets on backend restart.

See:

• MONGODB_SCHEMA_SIMPLE.md for detailed sample documents and indexes.

• API base comes from VITE_API_BASE.
• Login/registration UI is in frontend/src/components/login/*.
• Registration currently enforces @gmail.com client-side in useAuth.
• Senior check-in experience is implemented in:
  • frontend/src/components/SeniorCheckin.jsx
  • frontend/src/hooks/useCheckin.js
• Doctor dashboard/report UI is in:
  • frontend/src/components/DoctorDashboard.jsx
  • frontend/src/hooks/useDoctorDashboard.js
• Favicon and brand asset:
  • frontend/public/senicarelogo.png

• Entry point: backend/app/main.py
• Routes registered from backend/app/routes/__init__.py
• CORS allows:
  • http://localhost:5173
  • http://localhost:5174
  • http://localhost:4173
• User index creation (users.email unique) is attempted on startup.

Current repo state:

• No top-level automated test command is configured in frontend/package.json or root.
• Manual validation recommended:
  1. start backend/frontend
  2. register/login
  3. run check-in with camera + mic permissions
  4. verify /checkins/* writes in Mongo
  5. test doctor dashboard with a doctor role user

• Set JWT_SECRET in backend env before using /auth/login and protected routes.

• Verify MONGODB_URI or MONGO_* values.
• Ensure network access/credentials are correct.

• Set a valid GEMINI_API_KEY.

• Use longer, stable, well-lit clips.
• Install ffmpeg/ffprobe.
• Reduce motion and keep face centered.

• Use HTTPS or localhost.
• Grant browser permissions for camera and microphone.
• Prefer latest Chrome for most stable MediaRecorder and speech APIs.

• ARCHITECTURE.md: high-level product and architecture writeup
• MONGODB_SCHEMA_SIMPLE.md: practical schema + indexes + query examples
• backend/README.md: backend-specific setup details
• VHR/README.md: standalone VHR spike walkthrough

For local terminal diagnostics outside the web app:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python run_facial_symmetry_checkin.py --duration 20 --show-video

This uses vision/facial_symmetry.py and the MediaPipe task model at models/face_landmarker.task (auto-download fallback is implemented).

• Daniel Zhong
• Raihan Carder
• Suhiyini Kasim
• Long Vo