Reliable, scalable, fault-tolerant data pipeline simulation project.

This branch supports two deployment implementations, both without Kubernetes:

1. Single-machine
2. Distributed multi-machine

.
├── config
├── data
│   ├── analysis
│   ├── preprocess
│   ├── pyproject.toml
│   └── uv.lock
├── infra
│   ├── clickhouse
│   ├── connectors
│   ├── flink
│   ├── kafka
│   ├── nginx
│   ├── spark
│   └── terraform
├── model
└── services
    ├── flink-job
    ├── generator
    └── ingestor

Core path: generator -> nginx -> ingestor -> kafka -> flink -> clickhouse

Prediction branch (current default): model (ONNX artifact) -> flink -> clickhouse.taxi_predictions

Optional branches:

1. kafka -> kafka connect s3 sink -> S3
2. prometheus + kafka-exporter + grafana

1. Runtime runbook index (scenario split): docs/runbooks/runtime.md
2. Single-machine from scratch: docs/runbooks/runtime-single-machine-from-scratch.md
3. Distributed from scratch: docs/runbooks/runtime-distributed-from-scratch.md
4. Validation checklist: docs/runbooks/validation.md
5. Total architecture overview: docs/system-architecture.md
6. Config model + runtime ownership invariants: config/EXPLANATION.md
7. Refactor history: docs/history/demolish-ops.md, docs/history/merge-issues-2026-02-04.md

1. config/.env는 IP/PORT만 관리한다.
2. non-network 값(topic, tuning, table 등)은 컴포넌트 compose 파일에 하드코딩한다.
3. 각 컴포넌트 compose 파일을 직접 실행한다 (root launcher 없음).
4. docker compose 실행 시 항상 --env-file config/.env를 명시한다.

Single-machine:

cp config/.env.single-machine config/.env

# Run each command in a separate terminal (foreground mode)
docker compose -f infra/kafka/docker-compose.yml --env-file config/.env up
docker compose -f infra/clickhouse/docker-compose.yml --env-file config/.env up
docker compose -f services/ingestor/docker-compose.yml --env-file config/.env up --build
docker compose -f infra/nginx/docker-compose.yml --env-file config/.env up
docker compose -f infra/flink/docker-compose.yml --env-file config/.env up --build

cd services/generator
./build/generate

Distributed commands and machine-by-machine startup order are in docs/runbooks/runtime-distributed-from-scratch.md.

1. flink-job: build with JDK 17+, target bytecode Java 11 (Flink 1.17.2 compatibility)
2. ingestor: JDK 17+ (Spring Boot 3.2)
3. generator: C++ toolchain (Conan + CMake)
4. Python workspace: data/ with uv (data/pyproject.toml, data/uv.lock)

cd services/flink-job
mvn clean package

Version Compatibility:

Common issues:

• NoSuchFieldError: JCTree$JCImport → Lombok version too old for your JDK. Upgrade Lombok.
• ClassNotFoundException at runtime → Connector version mismatch with Flink runtime.

cd services/ingestor
./gradlew build
# or via Docker
docker build -t ingestor .

Requires JDK 17+ (Spring Boot 3.2).