brew install cmdrvl/tap/vacuum

You receive a data delivery. 47 files in a directory — CSVs, Excel workbooks, PDFs, a stray .tmp someone forgot to delete. Before you can hash them, fingerprint them, or lock them down, you need the answer to the simplest question in data: what's actually there?

vacuum walks directories and emits a deterministic JSONL manifest — the same files in, the same manifest out, every time. Each file gets a record with its path, size, modification time, and MIME type. No content reading, no transformation, no heuristics. Just enumeration — the reliable starting point for everything downstream.

• Deterministic by construction — records sorted by (relative_path, root) byte-order. Same directory always produces byte-identical output.
• Multi-root scanning — scan /data/q3 and /data/q4 in one invocation; records from all roots interleave deterministically in a single manifest.
• Nothing is silently dropped — files that can't be stat'd produce _skipped records with warnings, not silent omissions.
• Pipeline native — JSONL output feeds directly into hash, fingerprint, and lock via Unix pipes.

$ vacuum /data/dec

{"version":"vacuum.v0","path":"/data/dec/model.xlsx","relative_path":"model.xlsx","root":"/data/dec","size":2481920,"mtime":"2025-12-31T12:00:00.000Z","extension":".xlsx","mime_guess":"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet","tool_versions":{"vacuum":"0.1.0"}}
{"version":"vacuum.v0","path":"/data/dec/tape.csv","relative_path":"tape.csv","root":"/data/dec","size":847201,"mtime":"2025-12-15T08:30:00.000Z","extension":".csv","mime_guess":"text/csv","tool_versions":{"vacuum":"0.1.0"}}

Two files inventoried — sorted, typed, timestamped, ready for hash.

# Filter to specific extensions:
$ vacuum /data/dec --include "*.csv" --include "*.xlsx"

# Exclude scratch files:
$ vacuum /data --exclude "*.tmp" --exclude ".DS_Store"

# Multiple roots, single manifest:
$ vacuum /data/q3 /data/q4 > q3q4-manifest.jsonl

# Full pipeline into lockfile:
$ vacuum /data/dec | hash | fingerprint --fp csv.v0 \
    | lock --dataset-id "dec" > dec.lock.json

vacuum is the first tool in the stream pipeline — it discovers what exists.

vacuum  →  hash  →  fingerprint  →  lock  →  pack
(scan)    (hash)    (template)     (pin)    (seal)

Each tool reads JSONL from stdin and emits enriched JSONL to stdout. vacuum starts the chain by walking directories and producing the initial manifest.

vacuum does not replace downstream tools.

vacuum only answers: what files exist, how big are they, and when were they last modified?

vacuum emits exactly one domain outcome. Note: there is no exit code 1 — either the scan completes or it refuses.

All roots were enumerated. Individual file-level failures (e.g., permission denied on a single file) are recorded as _skipped records in the output stream — they don't prevent a successful scan.

$ vacuum /data/dec
# exit 0 — all files inventoried (some may be _skipped)

Cannot begin scanning. The root directory doesn't exist, isn't readable, or a filesystem error prevents the scan from starting.

{
  "code": "E_ROOT_NOT_FOUND",
  "message": "Root path does not exist",
  "detail": { "root": "/data/nonexistent/" },
  "next_command": null
}

Refusals always include the error code and detail. Mechanical misuse cases, like passing a file instead of a directory, also include a concrete next_command.

When to use vacuum:

• Start of a data pipeline — discover what artifacts exist before hashing and locking
• Audit and compliance — produce a reproducible inventory of a directory
• CI automation — machine-readable manifests that feed into downstream tools

When vacuum might not be ideal:

• You need recursive file search with complex predicates — use find
• You need file content analysis — use downstream tools (hash, fingerprint, shape)
• You need real-time filesystem watching — vacuum is a point-in-time snapshot

brew install cmdrvl/tap/vacuum

curl -fsSL https://raw.githubusercontent.com/cmdrvl/vacuum/main/scripts/install.sh | bash

cargo build --release
./target/release/vacuum --help

vacuum <ROOT>... [OPTIONS]
vacuum witness <query|last|count> [OPTIONS]

• <ROOT>...: One or more directories to scan. At least one required.

• stdout: JSONL manifest records (one per file)
• stderr: progress diagnostics (with --progress) or warnings

Every discovered file produces one vacuum.v0 record:

{
  "version": "vacuum.v0",
  "path": "/data/dec/tape.csv",
  "relative_path": "tape.csv",
  "root": "/data/dec",
  "size": 847201,
  "mtime": "2025-12-15T08:30:00.000Z",
  "extension": ".csv",
  "mime_guess": "text/csv",
  "tool_versions": { "vacuum": "0.1.0" }
}

Files that can't be stat'd (permission denied, broken symlinks) produce a skipped record:

{
  "version": "vacuum.v0",
  "path": "/data/dec/protected.xlsx",
  "relative_path": "protected.xlsx",
  "root": "/data/dec",
  "size": null,
  "mtime": null,
  "extension": ".xlsx",
  "mime_guess": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
  "_skipped": true,
  "_warnings": [
    { "tool": "vacuum", "code": "E_FILE_PERMISSION", "message": "Cannot read file metadata", "detail": { "error": "Permission denied" } }
  ],
  "tool_versions": { "vacuum": "0.1.0" }
}

Skipped records flow downstream — hash passes them through, lock collects them in the skipped array.

Multiple roots: fail-fast on the first failing root.

Check that the path is correct and the directory exists:

ls -la /data/dec/  # verify directory exists
vacuum /data/dec

You don't have permission to read the root directory:

ls -la /data/  # check permissions on parent
# Fix: adjust permissions or run with appropriate access

vacuum scans directories, not individual files. If you passed a file path like vacuum analysis_results.json, rerun it against the parent directory or .:

vacuum .
# or
vacuum reports/

Individual files may be skipped due to permission issues or broken symlinks. These still produce records with _skipped: true — they don't prevent the scan from completing (exit 0). Check the _warnings array for details:

vacuum /data/dec | jq 'select(._skipped == true) | .path'

The directory exists but contains no files matching your include/exclude patterns:

# Check what's actually in the directory:
ls -la /data/dec/

# Check your patterns:
vacuum /data/dec --include "*.csv"  # too narrow?

Patterns are matched against relative_path (forward-slash normalized), not the absolute path:

# Wrong — matching against absolute path:
vacuum /data --include "/data/*.csv"

# Right — matching against relative path:
vacuum /data --include "*.csv"
vacuum /data --include "subdir/*.csv"

vacuum includes an s3-to-vacuum adapter script that converts aws s3 ls output into vacuum.v0 JSONL, enabling S3 prefix scanning without native AWS SDK dependencies.

# Scan an S3 prefix
scripts/s3-to-vacuum s3://my-bucket/data/2025/

# Multiple prefixes
scripts/s3-to-vacuum s3://bucket/q3/ s3://bucket/q4/

# Check delivery completeness before downloading (with fingerprint struct-check)
scripts/s3-to-vacuum s3://bucket/delivery/ \
  | fingerprint struct-check --rules package.sf.yaml

# If complete, pull down and run full pipeline locally
aws s3 sync s3://bucket/delivery/ /data/q3/
vacuum /data/q3 | hash | fingerprint --fp appraisal.v1 \
  | lock --dataset-id q3 > q3.lock.json

Requirements: aws CLI (authenticated) and jq.

The adapter emits identical vacuum.v0 records — path uses s3:// URIs, relative_path is the key minus prefix, and all downstream tools that consume vacuum.v0 work unchanged. Records are deterministically sorted by (relative_path, root).

Note: The full vacuum → hash → fingerprint → lock pipeline requires local files (hash reads file contents). The S3 adapter is designed for inventory and completeness checking — use aws s3 sync before running the content pipeline.

find produces unstructured text that requires parsing. vacuum produces deterministic JSONL with rich metadata (size, mtime, MIME type, extension) that pipes directly into the rest of the pipeline. Same directory always produces identical output.

vacuum's job is enumeration, not transformation. Either the scan starts (exit 0) or it can't (exit 2). Per-file issues like permission denied are recorded as _skipped records in the output stream — they don't prevent the scan from completing.

Deterministic ordering. Records are sorted by (relative_path, root) byte-order, which requires seeing all files before emitting any. This trades latency for reproducibility.

Records from all roots are interleaved by relative_path, with root as tiebreaker. This means files with the same relative path from different roots appear adjacent in the output.

Yes — pass multiple roots:

vacuum /data/q3 /data/q4 /data/q1

All files from all roots appear in a single sorted manifest.

By default, vacuum follows symlinks and resolves targets to canonical paths. Use --no-follow to skip symlinks entirely.

Extension-based lookup from a built-in table: .csv, .tsv, .txt, .json, .jsonl, .xml, .pdf, .xlsx, .xls, .parquet, .zip, .gz, .yaml/.yml, and others. Unknown extensions produce null.

For the full toolchain guide, see the Agent Operator Guide.

$ vacuum --describe | jq '.exit_codes'
{
  "0": { "meaning": "SCAN_COMPLETE" },
  "2": { "meaning": "REFUSAL" }
}

$ vacuum --describe | jq '.pipeline'
{
  "upstream": [],
  "downstream": ["hash", "fingerprint", "lock", "pack"]
}

# 1. Scan directory
vacuum /data/dec > manifest.jsonl

case $? in
  0) echo "scan complete"
     wc -l manifest.jsonl ;;
  2) echo "refusal"
     cat manifest.jsonl | jq '.code'
     exit 1 ;;
esac

# 2. Count skipped files
skipped=$(jq -s '[.[] | select(._skipped == true)] | length' manifest.jsonl)
echo "skipped: $skipped"

# 3. Pipe into rest of pipeline
cat manifest.jsonl | hash | lock --dataset-id "dec" > dec.lock.json

• Exit codes — 0/2 map to success/error branching (no ambiguous exit 1)
• Structured JSONL only — stdout is always machine-readable
• --describe — prints operator.json so an agent discovers the tool without reading docs
• --schema — prints the record JSON schema for programmatic validation
• Skipped records inline — agents can filter _skipped records without separate error streams

# Query by date range or outcome
vacuum witness query --tool vacuum --since 2026-01-01 --outcome SCAN_COMPLETE --json

# Get the most recent scan
vacuum witness last --json

# Count scans matching a filter
vacuum witness count --since 2026-02-01

vacuum witness query [--tool <name>] [--since <iso8601>] [--until <iso8601>] \
  [--outcome <SCAN_COMPLETE|REFUSAL>] [--input-hash <substring>] \
  [--limit <n>] [--json]

vacuum witness last [--json]

vacuum witness count [--tool <name>] [--since <iso8601>] [--until <iso8601>] \
  [--outcome <SCAN_COMPLETE|REFUSAL>] [--input-hash <substring>] [--json]

The full specification is docs/PLAN.md. This README covers intended v0 behavior; the spec adds implementation details, edge-case definitions, and testing requirements.

cargo fmt --check
cargo clippy --all-targets -- -D warnings
cargo test