Michael Usachenko (50ba1344) at 18 Mar 23:56

What does this MR do and why?

The query pipeline was a single module inside gkg-server, tightly coupled to gRPC streaming, ClickHouse, and JWT authentication. This made it impossible to reuse for local query execution (e.g. MR 590 which runs queries against DuckDB from the CLI).

This MR breaks the pipeline into layered crates so that the core framework and shared logic can be composed differently depending on the context — full server pipeline with auth/redaction/hydration, or a minimal local pipeline that skips all of that.

How it works

The pipeline is built around a PipelineStage trait with a .then() chain pattern. Each stage reads the previous stage's output from a shared context, does its work, and returns its own output. The runner enforces stage ordering at compile time via phantom types.

The server composes all eight stages:

Security → Compilation → Execution → Extraction → Authorization → Redaction → Hydration → Output

The CLI can compose just the stages it needs:

Compilation → Execution → Extraction → Output

Same framework, same shared stages, different composition.

Formatting (Graph, Goon) is no longer a pipeline stage — it happens after the pipeline returns, so the caller decides how to present results.

Crate structure

graph TD
    QP["querying-pipeline — framework<br/>(PipelineStage, PipelineRunner, context, observer)"]
    QT["querying-types — data model<br/>(QueryResult, NodeRef, ResourceCheck)"]
    QSS["querying-shared-stages — reusable stages<br/>(Compilation, Extraction, Output + inter-stage types)"]
    QF["querying-formatters — output formatting<br/>(GraphFormatter, GoonFormatter, row_to_json)"]
    SRV["gkg-server/pipeline — server stages<br/>(Security, ClickHouse executor, Authorization,<br/>Redaction, Hydration)"]

    QP --> QT
    QSS --> QP
    QSS --> QT
    QF --> QSS
    QF --> QT
    SRV --> QP
    SRV --> QSS
    SRV --> QF

Other notable changes

• query_pipeline/ renamed to pipeline/ in gkg-server
• All stages are stateless unit structs — per-request data (claims, gRPC stream, ClickHouse client) lives in a typed extension map on the pipeline context
• Removed the RedactionMessage generic that previously threaded through the entire pipeline — the authorization stage now uses ExecuteQueryMessage directly
• QueryPipelineService is no longer generic over a formatter type

Testing

All 124 unit tests pass. Full workspace clippy clean. Integration tests updated for new imports.

Related

Enables MR 590 (local DuckDB query execution in CLI).

Michael Usachenko (3f30fb60) at 18 Mar 23:56

... and 1 more commit

Not a serious concern right now - will look into profiling soon

Michael Usachenko (50ba1344) at 18 Mar 23:48

Michael Usachenko (e42542b9) at 18 Mar 23:42

I ended up removing mutex all together

we can eventually rename it - but this is a good first step

Michael Usachenko (ba3d8445) at 18 Mar 23:28

... and 2 more commits

Michael Usachenko (65f5e359) at 18 Mar 23:27

I think remove-llm-comments in AGENTS.md did this unknowningly, will revert

I don't think it makes sense - query engine should be a pure function - separation of concerns

Michael Usachenko (da444520) at 18 Mar 23:12

... and 2 more commits

Michael Usachenko (0b8a962f) at 18 Mar 22:55

Michael Usachenko (970dde94) at 18 Mar 22:55

... and 1 more commit

Summary

• Install ast-grep 0.42.0 via mise (cargo backend) for AST-based code search and rewrite
• Add .claude/skills/ast-grep/SKILL.md teaching the agent when and how to use ast-grep instead of text-based Grep/Edit for structural code transformations
• Update AGENTS.md/CLAUDE.md with ast-grep references in "Where to find things" and "Code quality" sections

What the skill covers

• Pattern search (ast-grep run -p) and batch rewrite (-r ... -U)
• YAML inline rules with relational (has/inside + stopBy: end) and composite (all/any/not) logic
• --json output, --debug-query, --selector, constraints
• Key Rust node kinds (non-obvious mappings like try_expression for ?, function_modifiers for async)
• Shell escaping guidance

Testing

All examples validated against the codebase:

Michael Usachenko (0b8a962f) at 18 Mar 22:44

Michael Usachenko (e083f272) at 18 Mar 22:43

... and 1 more commit

Michael Usachenko (0ee8a9d5) at 18 Mar 22:23

Michael Usachenko (d98e92eb) at 18 Mar 21:53