Employees were increasingly using public tools (ChatGPT, Google Translate) for document digestion and translation. This posed two core risks:

1. Data leakage to external AI platforms
2. Inconsistent document analysis quality

The client required a fully local AI assistant, scalable, multilingual, and compliant with internal data governance.

I implemented a secure hybrid architecture with full separation of concerns. A React-based frontend with SSO login managed user sessions and chat interactions, while a dedicated user database stored chat history. A vector database (PostgreSQL + pgvector) held embedded company knowledge, optimized for RAG use. Distributed Ollama LLM endpoints were deployed across a local 9-GPU HPC cluster, and event-driven pipelines handled ingestion of documents, embedding generation, translation, and news updates. Each component was isolated and containerized, enabling scalable, fault-tolerant operations while enforcing data boundaries between chat storage and knowledge retrieval.

A secure local AI assistant architecture using a React-based UI, user DB, Ollama endpoints on 9x Nvidia GPUs HPC, and pgvector-based RAG pipelines.

• Infrastructure: On-prem HPC cluster (9x Nvidia GPUs)
• Frontend: React-based UI with SSO
• LLM Runtime: Ollama endpoints (local, GPU distributed)
• Vector DB: PostgreSQL + pgvector
• User DB: PostgreSQL (chat metadata, settings)
• Ingestion Pipelines: Python-based, event-triggered
• Containerization & Scaling: Docker, optionally Kubernetes (bare-metal)

• Data Sovereignty: Complete local deployment eliminated external dependencies and data leakage risks.
• Scalability: Distributed inference on GPU clusters maintained low latency (<2s) even with 300+ concurrent users.
• Data Hygiene: Separating user metadata from vector knowledge prevented knowledge base pollution.
• User Adoption: Integrating with existing SSO and providing a familiar UI (React-based) smoothed the transition from public tools.

• Achieved complete local deployment, removing all dependency on external AI services
• Improved onboarding efficiency by ~70% through instant document summarization
• Prevented knowledge base "pollution" by separating user metadata from vector knowledge
• Maintained <2s latency for 300+ concurrent users via GPU-distributed inference endpoints
• Enabled fast, private multilingual translation across departments