Enterprise-grade orchestration and management layer for AllSource event store

The AllSource Control Plane has been upgraded from v0.1.0 to v1.0 with enterprise features:

• ✅ JWT Authentication - Validates tokens from Rust core
• ✅ Role-Based Access Control (RBAC) - 4 roles, 7 permissions
• ✅ Audit Logging - Complete audit trail of all operations
• ✅ OpenTelemetry Tracing - Distributed tracing via OTLP (Jaeger, Tempo, etc.)
• ✅ Policy Enforcement - 5 default policies, custom policy engine
• ✅ Permission-Based Routes - Fine-grained access control
• ✅ Authenticated Proxying - Secure forwarding to Rust core

• ⬆️ From: Basic health checks and metrics
• ⬆️ To: Full enterprise orchestration layer
• ⬆️ Added: 5 new modules, 2,000 lines of code
• ⬆️ Security: Now requires authentication for all endpoints

┌─────────────────────────────────────────────────────┐
│            Operators / Dashboards                   │
│         (Authenticated via JWT/API Key)             │
└─────────────────────────────────────────────────────┘
                         │ HTTPS
                         ▼
┌─────────────────────────────────────────────────────┐
│       Go Control Plane v1.0 (Port 3901)             │
│  ┌───────────────────────────────────────────────┐  │
│  │  🔐 JWT Authentication Middleware             │  │
│  │  🛡️  RBAC & Permission Enforcement            │  │
│  │  📋 Audit Logging (All Operations)            │  │
│  │  📊 OpenTelemetry Distributed Tracing         │  │
│  │  📜 Policy Engine (5 Default Policies)        │  │
│  │  🎯 Authenticated Proxy to Core               │  │
│  └───────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────┘
                         │ HTTP + Auth Token
                         ▼
┌─────────────────────────────────────────────────────┐
│      Rust Event Store Core v1.0 (Port 8080)        │
│  • Event Ingestion (469K events/sec)                │
│  • Multi-Tenancy with Quotas                        │
│  • Rate Limiting (Token Bucket)                     │
│  • Authentication & RBAC                            │
│  • Backup & Restore                                 │
└─────────────────────────────────────────────────────┘

• JWT Token Validation - Validates tokens issued by Rust core
• API Key Support - Accepts API keys for service accounts
• Auth Middleware - Automatic token extraction and validation
• Permission Checking - Fine-grained permission enforcement
• Role-Based Access - 4 roles with distinct capabilities

• Complete Audit Trail - Logs all API requests and operations
• Structured Logging - JSON format for easy parsing
• Event Types - API requests, auth events, tenant events, operations
• Rich Metadata - User ID, tenant ID, IP address, user agent, duration
• File-Based Storage - Appends to audit.log file
• Configurable - Enable/disable via AUDIT_LOG_PATH env var

• OpenTelemetry SDK - Industry-standard tracing
• OTLP Exporter - Export traces via OTLP HTTP (works with Jaeger, Tempo, etc.)
• Span Propagation - Distributed context across services
• Rich Attributes - HTTP method, route, status, user info
• Error Tracking - Automatic error recording in spans
• Custom Events - Add custom events for important actions

• Policy Engine - Evaluate policies against requests
• Condition-Based Rules - Support for eq, ne, gt, lt, contains, in
• Priority System - Higher priority policies evaluated first
• Actions - Allow, Deny, Warn
• Default Policies - 5 security policies out-of-the-box
• Custom Policies - Add your own policies dynamically

• Admin - Full access to all features
• Developer - Read, write, manage schemas & pipelines
• ReadOnly - Read-only access
• ServiceAccount - Read and write (no admin)

• Read - View resources
• Write - Create and update resources
• Admin - Administrative operations
• Metrics - View metrics and monitoring data
• ManageSchemas - Manage event schemas
• ManagePipelines - Manage data pipelines
• ManageTenants - Manage tenant configuration

GET  /health              Health check
GET  /metrics             Prometheus metrics

POST /api/v1/auth/register  Register new user (public)
POST /api/v1/auth/login     User login (public)
GET  /api/v1/auth/me        Current user info (requires auth)

GET  /api/v1/cluster/status       Cluster status
GET  /api/v1/metrics/json         Aggregated metrics
GET  /api/v1/health/core          Core service health

POST /api/v1/operations/snapshot  Create snapshot
POST /api/v1/operations/replay    Trigger event replay
POST /api/v1/operations/backup    Create backup

GET    /api/v1/tenants         List all tenants
GET    /api/v1/tenants/:id     Get tenant details
POST   /api/v1/tenants         Create new tenant
PUT    /api/v1/tenants/:id     Update tenant
DELETE /api/v1/tenants/:id     Delete tenant

GET    /api/v1/users           List all users
DELETE /api/v1/users/:id       Delete user

• Go 1.22 or higher
• AllSource Rust Core v1.0 running on localhost:3900
• (Optional) OTLP-compatible tracing backend (Jaeger, Tempo, etc.)

# Clone repository
cd services/control-plane

# Install dependencies (if Go is installed)
go mod download

# Set environment variables
export JWT_SECRET="your-jwt-secret-key"  # Must match Rust core
export AUDIT_LOG_PATH="audit.log"
export OTLP_ENDPOINT="localhost:4318"  # Optional, OTLP HTTP endpoint
export GIN_MODE="release"  # For production

go run main_v1.go

# Build optimized binary
CGO_ENABLED=0 go build -ldflags="-s -w" -o control-plane main_v1.go

# Run
./control-plane

FROM golang:1.26-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY *.go ./
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o control-plane main_v1.go

FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=builder /app/control-plane .
EXPOSE 8081
CMD ["./control-plane"]

# Production configuration
export PORT=8081
export JWT_SECRET="$(openssl rand -base64 32)"  # Generate secure secret
export AUDIT_LOG_PATH="/var/log/allsource/audit.log"
export OTLP_ENDPOINT="jaeger:4318"  # OTLP HTTP endpoint
export ENVIRONMENT="production"
export GIN_MODE="release"

The control-plane ships with 5 default security policies:

• Resource: tenant
• Action: deny
• Conditions: tenant_id == "default" AND operation == "delete"
• Priority: 100

• Resource: tenant
• Action: deny
• Conditions: operation == "create" AND role != "Admin"
• Priority: 90

• Resource: user
• Action: deny
• Conditions: operation == "delete" AND target_user_id == user_id
• Priority: 95

• Resource: operation
• Action: warn
• Conditions: record_count > 10000
• Priority: 50

• Resource: operation
• Action: deny
• Conditions: operation_type IN ["snapshot", "backup", "restore"] AND recent_operations > 5
• Priority: 80

# Register a new admin user
curl -X POST http://localhost:3901/api/v1/auth/register \
  -H "Content-Type: application/json" \
  -d '{
    "username": "admin",
    "email": "[email protected]",
    "password": "secure_password_123",
    "role": "Admin"
  }'

# Login to get JWT token
TOKEN=$(curl -s -X POST http://localhost:3901/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "username": "admin",
    "password": "secure_password_123"
  }' | jq -r '.token')

echo "Token: $TOKEN"

# Get current user info
curl -X GET http://localhost:3901/api/v1/auth/me \
  -H "Authorization: Bearer $TOKEN"

# Get cluster status
curl -X GET http://localhost:3901/api/v1/cluster/status \
  -H "Authorization: Bearer $TOKEN"

# Create a new tenant
curl -X POST http://localhost:3901/api/v1/tenants \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "acme",
    "name": "Acme Corporation",
    "tier": "professional"
  }'

# List all tenants
curl -X GET http://localhost:3901/api/v1/tenants \
  -H "Authorization: Bearer $TOKEN"

# Get tenant details
curl -X GET http://localhost:3901/api/v1/tenants/acme \
  -H "Authorization: Bearer $TOKEN"

# Create snapshot
curl -X POST http://localhost:3901/api/v1/operations/snapshot \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'

# Trigger event replay
curl -X POST http://localhost:3901/api/v1/operations/replay \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "entity_id": "user-123",
    "as_of": "2025-01-15T10:00:00Z"
  }'

# Tail audit log
tail -f audit.log

# View recent audit events
tail -20 audit.log | jq .

# Filter auth events
jq 'select(.event_type == "api_request" and .path | contains("auth"))' audit.log

Run the comprehensive integration test suite:

# Make sure both services are running:
# Terminal 1: cd services/core && cargo run
# Terminal 2: cd services/control-plane && go run main_v1.go

# Terminal 3: Run integration tests
cd services
chmod +x integration_test.sh
./integration_test.sh

The test suite includes:

• ✅ Pre-flight checks (services running)
• ✅ Authentication flow tests
• ✅ Multi-tenancy tests
• ✅ RBAC & permission tests
• ✅ Core service integration
• ✅ Audit & observability tests
• ✅ Policy enforcement tests
• ✅ Operation tests

The control-plane exposes Prometheus metrics at /metrics:

# HTTP request metrics
control_plane_http_requests_total
control_plane_http_request_duration_seconds
control_plane_http_requests_in_flight

# Core health check metrics
control_plane_core_health_checks_total
control_plane_core_health_check_duration_seconds

# Operation metrics
control_plane_snapshot_operations_total
control_plane_replay_operations_total

# System metrics
control_plane_uptime_seconds

View distributed traces in Jaeger UI (or any OTLP-compatible backend like Grafana Tempo):

# Start Jaeger with OTLP support (Docker)
docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

# Access Jaeger UI
open http://localhost:16686

# Configure control-plane with OTLP HTTP endpoint
export OTLP_ENDPOINT="localhost:4318"

# View all audit events
cat audit.log | jq .

# Filter by user
cat audit.log | jq 'select(.user_id == "user-123")'

# Filter by event type
cat audit.log | jq 'select(.event_type == "policy_denial")'

# Count requests by path
cat audit.log | jq -r '.path' | sort | uniq -c

1. Change JWT Secret

   export JWT_SECRET="$(openssl rand -base64 32)"

2. Enable HTTPS (via reverse proxy like Nginx/Traefik)

3. Rotate Secrets Regularly

4. Monitor Audit Logs

   • Set up log aggregation (ELK, Splunk, etc.)
   • Alert on suspicious patterns

5. Enable Rate Limiting in Rust core

6. Use Strong Passwords

   • Minimum 12 characters
   • Argon2 hashing in Rust core

7. Restrict Network Access

   • Firewall rules
   • VPC/security groups

control-plane/
├── main.go              # Original v0.1.0 (deprecated)
├── main_v1.go           # New v1.0 application entry
├── auth.go              # JWT authentication client
├── audit.go             # Audit logging system
├── tracing.go           # OpenTelemetry tracing
├── policy.go            # Policy enforcement engine
├── metrics.go           # Prometheus metrics (v0.1.0)
├── middleware.go        # Middleware (v0.1.0)
├── go.mod               # Dependencies
├── go.sum               # Checksums
├── README.md            # Original README (v0.1.0)
├── README_V1.md         # This file (v1.0)
└── package.json         # Metadata

# Format code
go fmt ./...

# Lint (requires golangci-lint)
golangci-lint run

# Vet code
go vet ./...

1. Authentication Required - All endpoints (except /health and /metrics) now require authentication
2. New Main File - Use main_v1.go instead of main.go
3. Environment Variables - New env vars for JWT secret, audit log, OTLP tracing

1. Update Dependencies

   go mod tidy

2. Set Environment Variables

   export JWT_SECRET="your-secret"
   export AUDIT_LOG_PATH="audit.log"

3. Use New Main File

   go run main_v1.go  # Instead of main.go

4. Update Clients

   • Add Authorization: Bearer <token> header
   • Obtain token via /api/v1/auth/login

5. Test Integration

   ./integration_test.sh

• Gin Web Framework
• JWT Authentication
• OpenTelemetry Go
• Jaeger Tracing
• Prometheus Monitoring
• AllSource Core v1.0

Contributions welcome! Areas of interest:

• Unit tests for Go code (currently 0% coverage)
• Additional default policies
• Enhanced tracing context
• Grafana dashboards
• Kubernetes operator

MIT License - see LICENSE file for details

• ✅ JWT authentication client
• ✅ RBAC enforcement
• ✅ Audit logging
• ✅ OpenTelemetry tracing
• ✅ Policy enforcement
• ✅ Permission-based routes
• ✅ 12 API endpoints
• ✅ Integration test suite

• ✅ Basic health checks
• ✅ Prometheus metrics
• ✅ Cluster status endpoint
• ✅ Core health proxy
• ✅ Snapshot/replay operations (demo mode)