Correlation Engine v3

The Correlation Engine transforms raw security events into actionable incidents by detecting multi-stage attacks, cross-sensor patterns, and temporal anomalies.

Overview

Metric	Value
Built-in Rules	15
Correlation Window	5 minutes (configurable)
Alert Clustering	DBSCAN + MiniLM embeddings
Noise Reduction	100+ alerts/hour → 5-10 incidents
Lattice Integration	TSA, GPS, MIRE primitives

How Correlation Works

Raw Events (100+/hour)
    │
    ├── 1. Time Window Grouping ──── Group by 5-minute windows
    │
    ├── 2. Rule Matching ──────────── 15 rules checked in parallel
    │
    ├── 3. Kill Chain Mapping ─────── Map to MITRE ATT&CK stages
    │
    ├── 4. Alert Clustering ───────── DBSCAN reduces duplicates
    │
    ├── 5. Severity Calculation ───── Max(event severities) + escalation
    │
    └── 6. Incident Creation ──────── Unique incident with linked events
    
Correlated Incidents (5-10/hour)

Built-in Rules

Multi-Stage Attack Rules

#	Rule ID	Trigger Events	Window	Severity
1	`MULTI_STAGE_JAILBREAK`	jailbreak → tool_abuse	5m	HIGH
2	`EXFILTRATION_CHAIN`	pii_detected → exfiltration → network_alert	5m	CRITICAL
3	`INJECTION_TO_EXEC`	injection → tool_abuse (file/exec)	3m	CRITICAL
4	`CRESCENDO_ATTACK`	5+ low-severity from same source	10m	HIGH
5	`DORMANT_PAYLOAD_ACTIVATION`	dormant_payload → any_high_severity	30m	CRITICAL

Cross-Sensor Rules

#	Rule ID	Trigger	Window	Severity
6	`BYPASS_SHIELD_THEN_INJECT`	shield_block → core_detect	30s	HIGH
7	`CROSS_SENSOR_ESCALATION`	shield_bypass + core_detect	1m	CRITICAL
8	`IMMUNE_KERNEL_PLUS_CORE`	immune_alert + core_alert	2m	HIGH

Temporal Pattern Rules

#	Rule ID	Trigger	Window	Severity
9	`REPEATED_TOOL_ABUSE`	3+ tool_abuse from same IP	10m	HIGH
10	`ALERT_FLOOD`	100+ events from one sensor	60s	MEDIUM
11	`BRUTE_FORCE_PATTERN`	10+ auth_bypass attempts	5m	HIGH

Lattice Rules

#	Rule ID	Trigger	Primitive	Severity
12	`TSA_VIOLATION`	Temporal Safety Automata violation	TSA	CRITICAL
13	`GPS_HIGH_DANGER`	Goal Predictability Score > 0.7	GPS	HIGH
14	`MIRE_CONTAINMENT`	Model containment triggered	MIRE	CRITICAL
15	`IRM_HIDDEN_INTENT`	Intent Revelation divergence > 0.5	IRM	HIGH

Rule Definition Format

Rules are defined in YAML:

YAML

rules:
  - id: MULTI_STAGE_JAILBREAK
    name: "Multi-Stage Jailbreak Attack"
    description: "Jailbreak followed by tool abuse within 5 minutes"
    enabled: true
    
    conditions:
      - sequence:
          - category: jailbreak
            min_confidence: 0.7
          - category: tool_abuse
            min_confidence: 0.5
        within: "5m"
        same_field: "source_ip"
    
    action:
      create_incident: true
      severity: HIGH
      kill_chain_stage: "exploitation"
      playbook: "auto_block_jailbreak"
      
    metadata:
      mitre_atlas: ["AML.T0054", "AML.T0040"]
      owasp_llm: ["LLM01", "LLM07"]

Condition Types

Type	Description	Example
`sequence`	Events in order	jailbreak → tool_abuse
`count`	N events of type	3+ tool_abuse
`absence`	Expected event missing	No auth within 5m of access
`threshold`	Metric exceeds value	GPS > 0.7

Matching Options

Option	Description
`within`	Time window for correlation
`same_field`	Events must share this field value
`different_source`	Events must come from different sensors
`min_confidence`	Minimum confidence score per event

Alert Clustering

Problem

66 engines × incoming requests = hundreds of raw alerts per hour. Most are related to the same attack.

Solution: DBSCAN + MiniLM

100+ raw alerts
    │
    ├── 1. Embed alert descriptions (MiniLM-L6)
    │       → 384-dimensional vectors
    │
    ├── 2. DBSCAN clustering (ε=0.3, min_samples=3)
    │       → Group similar alerts
    │
    ├── 3. Representative selection
    │       → Pick highest-confidence alert per cluster
    │
    └── 4. Noise filtering
            → Isolated alerts kept as standalone events

5-10 incident-worthy clusters

Configuration

YAML

correlation:
  clustering:
    enabled: true
    algorithm: "dbscan"
    min_cluster_size: 3       # Minimum events per cluster
    similarity_threshold: 0.7  # Cosine similarity threshold
    embedding_model: "all-MiniLM-L6-v2"

Kill Chain Mapping

Correlated incidents are mapped to attack stages:

┌────────────┐  ┌────────────┐  ┌────────────┐  ┌────────────┐
│   Recon    │→ │  Delivery  │→ │Exploitation│→ │   Action   │
│            │  │            │  │            │  │            │
│ injection  │  │ shield_blk │  │ jailbreak  │  │exfiltration│
│ probe      │  │ evasion    │  │ tool_abuse │  │ pii_leak   │
└────────────┘  └────────────┘  └────────────┘  └────────────┘

Each incident displays:

Kill Chain Stage: Current attack phase
Progression: Visual timeline of stages
Prediction: GPS-based forecast of likely next stage

Incident Lifecycle

┌────────────────────────────────────────────────────┐
│                                                    │
│   ┌──────┐     ┌───────────────┐     ┌──────────┐│
│   │ OPEN │────▶│ INVESTIGATING │────▶│ RESOLVED ││
│   └──┬───┘     └───────┬───────┘     └──────────┘│
│      │                 │                          │
│      │         ┌───────▼───────┐                  │
│      └────────▶│  ESCALATED   │                  │
│                │ (Zero-G mode) │                  │
│                └───────────────┘                  │
│                                                    │
└────────────────────────────────────────────────────┘

Status	Description	Actions Available
OPEN	New, unacknowledged incident	Acknowledge, escalate
INVESTIGATING	Analyst working on it	Run playbook, update, resolve
ESCALATED	Requires senior approval	Zero-G mode actions
RESOLVED	Handled and documented	Reopen (if needed)

Performance

Metric	Value
Rule evaluation	<5ms for all 15 rules
Clustering	<50ms for 100 alerts
End-to-end correlation	<100ms
Memory usage	~50MB for rule engine
Max concurrent correlations	1000

Custom Rules

See the Custom Correlation Rules Tutorial for step-by-step guide on creating your own rules.

Next Steps

Lattice Primitives — Mathematical safety guarantees
Custom Rule Tutorial — Build custom rules
Alert Clustering Config — Fine-tune clustering