Skip to content

Commit 96fba76

Browse files
stacknilstacknil
authored
feat: add rule-evaluation-and-dedup-demo (#22)
1 parent 9a0b521 commit 96fba76

13 files changed

Lines changed: 1532 additions & 20 deletions

File tree

‎README.md‎

Lines changed: 23 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,17 @@ Latest milestone: [v0.4.0 — second demo and portfolio integration](https://git
1010

1111
- [telemetry-window-demo](#telemetry-window-demo)
1212
- [ai-assisted-detection-demo](demos/ai-assisted-detection-demo/README.md)
13+
- [rule-evaluation-and-dedup-demo](demos/rule-evaluation-and-dedup-demo/README.md)
1314

1415
| Demo | Input | Deterministic core | LLM role | Main artifacts | Guardrails / non-goals |
1516
| --- | --- | --- | --- | --- | --- |
1617
| [telemetry-window-demo](#telemetry-window-demo) | JSONL / CSV events | Windows<br>Features<br>Alert thresholds | None | `features.csv`<br>`alerts.csv`<br>`summary.json`<br>3 PNG plots | MVP only<br>No realtime<br>No case management |
1718
| [ai-assisted-detection-demo](demos/ai-assisted-detection-demo/README.md) | JSONL auth / web / process | Normalize<br>Rules<br>Grouping<br>ATT&CK mapping | JSON-only case drafting | `rule_hits.json`<br>`case_bundles.json`<br>`case_summaries.json`<br>`case_report.md`<br>`audit_traces.jsonl` | Human verification required<br>No autonomous response<br>No final verdict |
19+
| [rule-evaluation-and-dedup-demo](demos/rule-evaluation-and-dedup-demo/README.md) | JSON raw rule hits | Scope resolution<br>Cooldown grouping<br>Suppression reasoning | None | `rule_hits_before_dedup.json`<br>`rule_hits_after_dedup.json`<br>`dedup_explanations.json`<br>`dedup_report.md` | No realtime<br>No dashboard<br>No AI stage |
1820

1921
## What This Repo Is
2022

21-
`telemetry-lab` is a small portfolio repository for telemetry analytics and constrained detection-oriented workflows. It is organized as two local, file-based demos that are reproducible from committed sample data and intentionally scoped for public review rather than production use.
23+
`telemetry-lab` is a small portfolio repository for telemetry analytics and constrained detection-oriented workflows. It is organized as three local, file-based demos that are reproducible from committed sample data and intentionally scoped for public review rather than production use.
2224

2325
### telemetry-window-demo
2426

@@ -27,13 +29,22 @@ Latest milestone: [v0.4.0 — second demo and portfolio integration](https://git
2729
### ai-assisted-detection-demo
2830

2931
`ai-assisted-detection-demo` uses deterministic normalization, detection, case grouping, and ATT&CK mapping, then limits the LLM to JSON-only case summarization. Human verification is required, there is no autonomous response, and the demo does not produce a final incident verdict.
32+
33+
### rule-evaluation-and-dedup-demo
34+
35+
`rule-evaluation-and-dedup-demo` starts from raw rule hits and makes cooldown behavior legible. It shows which hits were kept, which were suppressed, how scope was resolved, and why repeated hits collapsed into fewer retained alerts.
3036

3137
## Quick Run
3238

33-
```bash
34-
python -m pip install -e .
35-
python -m telemetry_window_demo.cli run --config configs/default.yaml
36-
```
39+
```bash
40+
python -m pip install -e .
41+
python -m telemetry_window_demo.cli run --config configs/default.yaml
42+
```
43+
44+
Other demo entrypoints:
45+
46+
- `python -m telemetry_window_demo.cli run-ai-demo`
47+
- `python -m telemetry_window_demo.cli run-rule-dedup-demo`
3748

3849
That command reads `data/raw/sample_events.jsonl` and regenerates:
3950

@@ -95,12 +106,13 @@ Cooldown behavior:
95106
- scope prefers the first available entity-like field in this order: `entity`, `source`, `target`, `host`
96107
- when no entity-like field is present, cooldown falls back to per-`rule_name` behavior
97108

98-
## Repo Guide
99-
100-
- [`docs/sample-output.md`](docs/sample-output.md) summarizes the committed sample artifacts
101-
- [`docs/roadmap.md`](docs/roadmap.md) sketches the next demo directions
102-
- [`data/processed/summary.json`](data/processed/summary.json) captures the default run in machine-readable form
103-
- [`data/processed/richer_sample/summary.json`](data/processed/richer_sample/summary.json) captures the richer scenario pack
109+
## Repo Guide
110+
111+
- [`demos/rule-evaluation-and-dedup-demo/README.md`](demos/rule-evaluation-and-dedup-demo/README.md) explains the third demo and links its committed before/after dedup artifacts
112+
- [`docs/sample-output.md`](docs/sample-output.md) summarizes the committed sample artifacts
113+
- [`docs/roadmap.md`](docs/roadmap.md) sketches the next demo directions
114+
- [`data/processed/summary.json`](data/processed/summary.json) captures the default run in machine-readable form
115+
- [`data/processed/richer_sample/summary.json`](data/processed/richer_sample/summary.json) captures the richer scenario pack
104116
- [`tests/`](tests/) keeps regression coverage close to the CLI behavior and windowing logic
105117

106118
## Next Demo Directions

‎demos/rule-evaluation-and-dedup-demo/README.md‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Rule Evaluation And Dedup Demo
2+
3+
This demo is part of `telemetry-lab` and is intentionally small, local, and reviewer-friendly.
4+
5+
It focuses on alert semantics, not new AI behavior. The sample input is a committed set of raw rule hits that already fired before deduplication. The demo then applies deterministic cooldown handling and writes artifacts that explain which hits were kept, which were suppressed, and why.
6+
7+
## Purpose
8+
9+
The goal is to make repeated alert behavior legible.
10+
11+
Instead of only showing the final retained alerts, this demo shows:
12+
13+
- raw rule hits before deduplication
14+
- retained alerts after cooldown handling
15+
- per-hit suppression reasons
16+
- a short report that explains how repeated hits collapsed into fewer alerts
17+
18+
## Quick Start
19+
20+
From the repository root:
21+
22+
```bash
23+
python -m pip install -e .
24+
python -m telemetry_window_demo.cli run-rule-dedup-demo
25+
```
26+
27+
Generated artifacts are written to `demos/rule-evaluation-and-dedup-demo/artifacts/`.
28+
29+
## Demo Inputs
30+
31+
- sample data: `data/raw/sample_rule_hits.json`
32+
- cooldown config: `config/dedup.yaml`
33+
34+
The bundled sample intentionally includes:
35+
36+
- repeated hits for the same rule and `entity`
37+
- repeated hits for the same rule and `source`
38+
- repeated hits with no scope fields, which fall back to rule-only dedup
39+
- different scopes for the same rule, which stay independent
40+
41+
## Cooldown Semantics
42+
43+
Cooldown keys use `(rule_name, scope)`.
44+
45+
Scope resolution follows the same precedence described in the main demo:
46+
47+
1. `entity`
48+
2. `source`
49+
3. `target`
50+
4. `host`
51+
5. unscoped rule-only fallback
52+
53+
That means repeated hits for the same rule can still be kept separately when their scopes differ.
54+
55+
## Expected Artifacts
56+
57+
- `artifacts/rule_hits_before_dedup.json`
58+
- `artifacts/rule_hits_after_dedup.json`
59+
- `artifacts/dedup_explanations.json`
60+
- `artifacts/dedup_report.md`
61+
62+
## Artifact Semantics
63+
64+
- `rule_hits_before_dedup.json`: normalized raw hits with resolved cooldown scope and cooldown key
65+
- `rule_hits_after_dedup.json`: only the retained alerts, including which suppressed hits each retained alert now represents
66+
- `dedup_explanations.json`: one explanation record per raw hit, marked as either `retained` or `suppressed`
67+
- `dedup_report.md`: a short reviewer-facing report with run counts, per-group summary, retained alert explanations, and suppressed hit reasons
68+
69+
## Reviewer Walkthrough
70+
71+
1. Open `rule_hits_before_dedup.json` and note that several hits share the same `cooldown_key`.
72+
2. Open `dedup_explanations.json` and verify that each raw hit is labeled `retained` or `suppressed` with a concrete reason.
73+
3. Open `rule_hits_after_dedup.json` and confirm that retained alerts carry the suppressed hit ids they now represent.
74+
4. Open `dedup_report.md` and confirm that the before/after counts and per-group behavior are readable without looking at code.
75+
76+
## Limitations
77+
78+
- input is precomputed rule hits, not live telemetry
79+
- cooldown logic is intentionally simple and deterministic
80+
- there is no streaming state, dashboard, or service deployment
81+
- artifacts are designed for review, not production alert routing

‎demos/rule-evaluation-and-dedup-demo/artifacts/dedup_explanations.json‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
[
2+
{
3+
"hit_id": "RH-001",
4+
"rule_name": "login_fail_burst",
5+
"severity": "high",
6+
"alert_time": "2026-03-18T10:01:00Z",
7+
"cooldown_scope": "entity=account:alice",
8+
"scope_source": "entity",
9+
"cooldown_key": "login_fail_burst|entity=account:alice",
10+
"status": "retained",
11+
"reason": "kept as the first hit for `login_fail_burst / entity=account:alice`.",
12+
"group_position": 1,
13+
"group_size": 3,
14+
"cooldown_seconds": 180,
15+
"suppressed_by_hit_id": null,
16+
"seconds_since_last_retained": null,
17+
"message": "login_fail_count reached 8, threshold is 8"
18+
},
19+
{
20+
"hit_id": "RH-004",
21+
"rule_name": "login_fail_burst",
22+
"severity": "high",
23+
"alert_time": "2026-03-18T10:01:20Z",
24+
"cooldown_scope": "entity=account:bob",
25+
"scope_source": "entity",
26+
"cooldown_key": "login_fail_burst|entity=account:bob",
27+
"status": "retained",
28+
"reason": "kept as the first hit for `login_fail_burst / entity=account:bob`.",
29+
"group_position": 1,
30+
"group_size": 2,
31+
"cooldown_seconds": 180,
32+
"suppressed_by_hit_id": null,
33+
"seconds_since_last_retained": null,
34+
"message": "login_fail_count reached 8, threshold is 8"
35+
},
36+
{
37+
"hit_id": "RH-002",
38+
"rule_name": "login_fail_burst",
39+
"severity": "high",
40+
"alert_time": "2026-03-18T10:01:40Z",
41+
"cooldown_scope": "entity=account:alice",
42+
"scope_source": "entity",
43+
"cooldown_key": "login_fail_burst|entity=account:alice",
44+
"status": "suppressed",
45+
"reason": "suppressed because it matched the same cooldown key as retained hit `RH-001` only 40 seconds later, inside the 180 second cooldown.",
46+
"group_position": 2,
47+
"group_size": 3,
48+
"cooldown_seconds": 180,
49+
"suppressed_by_hit_id": "RH-001",
50+
"seconds_since_last_retained": 40,
51+
"message": "login_fail_count reached 9, threshold is 8"
52+
},
53+
{
54+
"hit_id": "RH-005",
55+
"rule_name": "login_fail_burst",
56+
"severity": "high",
57+
"alert_time": "2026-03-18T10:02:00Z",
58+
"cooldown_scope": "entity=account:bob",
59+
"scope_source": "entity",
60+
"cooldown_key": "login_fail_burst|entity=account:bob",
61+
"status": "suppressed",
62+
"reason": "suppressed because it matched the same cooldown key as retained hit `RH-004` only 40 seconds later, inside the 180 second cooldown.",
63+
"group_position": 2,
64+
"group_size": 2,
65+
"cooldown_seconds": 180,
66+
"suppressed_by_hit_id": "RH-004",
67+
"seconds_since_last_retained": 40,
68+
"message": "login_fail_count reached 10, threshold is 8"
69+
},
70+
{
71+
"hit_id": "RH-003",
72+
"rule_name": "login_fail_burst",
73+
"severity": "high",
74+
"alert_time": "2026-03-18T10:04:30Z",
75+
"cooldown_scope": "entity=account:alice",
76+
"scope_source": "entity",
77+
"cooldown_key": "login_fail_burst|entity=account:alice",
78+
"status": "retained",
79+
"reason": "kept because 210 seconds elapsed since retained hit `RH-001`, which meets the 180 second cooldown.",
80+
"group_position": 3,
81+
"group_size": 3,
82+
"cooldown_seconds": 180,
83+
"suppressed_by_hit_id": null,
84+
"seconds_since_last_retained": 210,
85+
"message": "login_fail_count reached 8, threshold is 8"
86+
},
87+
{
88+
"hit_id": "RH-006",
89+
"rule_name": "high_error_rate",
90+
"severity": "medium",
91+
"alert_time": "2026-03-18T10:05:00Z",
92+
"cooldown_scope": "source=api-01",
93+
"scope_source": "source",
94+
"cooldown_key": "high_error_rate|source=api-01",
95+
"status": "retained",
96+
"reason": "kept as the first hit for `high_error_rate / source=api-01`.",
97+
"group_position": 1,
98+
"group_size": 3,
99+
"cooldown_seconds": 180,
100+
"suppressed_by_hit_id": null,
101+
"seconds_since_last_retained": null,
102+
"message": "error_rate 0.46 exceeded 0.30"
103+
},
104+
{
105+
"hit_id": "RH-007",
106+
"rule_name": "high_error_rate",
107+
"severity": "medium",
108+
"alert_time": "2026-03-18T10:06:10Z",
109+
"cooldown_scope": "source=api-01",
110+
"scope_source": "source",
111+
"cooldown_key": "high_error_rate|source=api-01",
112+
"status": "suppressed",
113+
"reason": "suppressed because it matched the same cooldown key as retained hit `RH-006` only 70 seconds later, inside the 180 second cooldown.",
114+
"group_position": 2,
115+
"group_size": 3,
116+
"cooldown_seconds": 180,
117+
"suppressed_by_hit_id": "RH-006",
118+
"seconds_since_last_retained": 70,
119+
"message": "error_rate 0.41 exceeded 0.30"
120+
},
121+
{
122+
"hit_id": "RH-009",
123+
"rule_name": "rare_event_repeat_malware_alert",
124+
"severity": "high",
125+
"alert_time": "2026-03-18T10:07:00Z",
126+
"cooldown_scope": null,
127+
"scope_source": "unscoped",
128+
"cooldown_key": "rare_event_repeat_malware_alert|unscoped",
129+
"status": "retained",
130+
"reason": "kept as the first hit for `rare_event_repeat_malware_alert / unscoped`.",
131+
"group_position": 1,
132+
"group_size": 2,
133+
"cooldown_seconds": 180,
134+
"suppressed_by_hit_id": null,
135+
"seconds_since_last_retained": null,
136+
"message": "malware_alert repeated 2 times in one window"
137+
},
138+
{
139+
"hit_id": "RH-010",
140+
"rule_name": "rare_event_repeat_malware_alert",
141+
"severity": "high",
142+
"alert_time": "2026-03-18T10:08:00Z",
143+
"cooldown_scope": null,
144+
"scope_source": "unscoped",
145+
"cooldown_key": "rare_event_repeat_malware_alert|unscoped",
146+
"status": "suppressed",
147+
"reason": "suppressed because it matched the same cooldown key as retained hit `RH-009` only 60 seconds later, inside the 180 second cooldown.",
148+
"group_position": 2,
149+
"group_size": 2,
150+
"cooldown_seconds": 180,
151+
"suppressed_by_hit_id": "RH-009",
152+
"seconds_since_last_retained": 60,
153+
"message": "malware_alert repeated 3 times in one window"
154+
},
155+
{
156+
"hit_id": "RH-008",
157+
"rule_name": "high_error_rate",
158+
"severity": "medium",
159+
"alert_time": "2026-03-18T10:08:30Z",
160+
"cooldown_scope": "source=api-01",
161+
"scope_source": "source",
162+
"cooldown_key": "high_error_rate|source=api-01",
163+
"status": "retained",
164+
"reason": "kept because 210 seconds elapsed since retained hit `RH-006`, which meets the 180 second cooldown.",
165+
"group_position": 3,
166+
"group_size": 3,
167+
"cooldown_seconds": 180,
168+
"suppressed_by_hit_id": null,
169+
"seconds_since_last_retained": 210,
170+
"message": "error_rate 0.44 exceeded 0.30"
171+
}
172+
]

‎demos/rule-evaluation-and-dedup-demo/artifacts/dedup_report.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Rule Evaluation And Dedup Demo Report
2+
3+
This deterministic demo shows how repeated raw rule hits turn into fewer retained alerts after cooldown handling.
4+
Cooldown keys are built from `(rule_name, scope)`, where scope prefers `entity`, then `source`, then `target`, then `host`, and falls back to rule-only dedup when none are present.
5+
6+
## Run Summary
7+
8+
- raw_rule_hits: 10
9+
- retained_alerts: 6
10+
- suppressed_hits: 4
11+
- cooldown_seconds: 180
12+
13+
## Group Summary
14+
15+
| Rule / scope | Raw hits | Retained | Suppressed | First seen | Last seen |
16+
| --- | ---: | ---: | ---: | --- | --- |
17+
| login_fail_burst / entity=account:alice | 3 | 2 | 1 | 2026-03-18T10:01:00Z | 2026-03-18T10:04:30Z |
18+
| login_fail_burst / entity=account:bob | 2 | 1 | 1 | 2026-03-18T10:01:20Z | 2026-03-18T10:02:00Z |
19+
| high_error_rate / source=api-01 | 3 | 2 | 1 | 2026-03-18T10:05:00Z | 2026-03-18T10:08:30Z |
20+
| rare_event_repeat_malware_alert / unscoped | 2 | 1 | 1 | 2026-03-18T10:07:00Z | 2026-03-18T10:08:00Z |
21+
22+
## Retained Alerts
23+
24+
- RH-001 kept for `login_fail_burst / entity=account:alice`; kept as the first hit for `login_fail_burst / entity=account:alice`.
25+
Represents suppressed duplicates: RH-002.
26+
- RH-004 kept for `login_fail_burst / entity=account:bob`; kept as the first hit for `login_fail_burst / entity=account:bob`.
27+
Represents suppressed duplicates: RH-005.
28+
- RH-003 kept for `login_fail_burst / entity=account:alice`; kept because 210 seconds elapsed since retained hit `RH-001`, which meets the 180 second cooldown.
29+
- RH-006 kept for `high_error_rate / source=api-01`; kept as the first hit for `high_error_rate / source=api-01`.
30+
Represents suppressed duplicates: RH-007.
31+
- RH-009 kept for `rare_event_repeat_malware_alert / unscoped`; kept as the first hit for `rare_event_repeat_malware_alert / unscoped`.
32+
Represents suppressed duplicates: RH-010.
33+
- RH-008 kept for `high_error_rate / source=api-01`; kept because 210 seconds elapsed since retained hit `RH-006`, which meets the 180 second cooldown.
34+
35+
## Suppressed Hits
36+
37+
- RH-002 suppressed by RH-001 for `login_fail_burst / entity=account:alice`; suppressed because it matched the same cooldown key as retained hit `RH-001` only 40 seconds later, inside the 180 second cooldown.
38+
- RH-005 suppressed by RH-004 for `login_fail_burst / entity=account:bob`; suppressed because it matched the same cooldown key as retained hit `RH-004` only 40 seconds later, inside the 180 second cooldown.
39+
- RH-007 suppressed by RH-006 for `high_error_rate / source=api-01`; suppressed because it matched the same cooldown key as retained hit `RH-006` only 70 seconds later, inside the 180 second cooldown.
40+
- RH-010 suppressed by RH-009 for `rare_event_repeat_malware_alert / unscoped`; suppressed because it matched the same cooldown key as retained hit `RH-009` only 60 seconds later, inside the 180 second cooldown.

0 commit comments

Comments
 (0)