Ingest OpenTelemetry logs via OTLP/HTTP and query them with ClickHouse.

flowchart LR
    OTLP[OTLP/HTTP] --> OC[otel-collector]
    OC --> O2P[otlp2parquet]
    O2P --> S3[(S3<br/>rustfs)]
    SYNC[log-sync] <--> S3
    CAT[ice-rest-catalog] <--> S3
    CH[(ClickHouse)] <--> S3
    SYNC <--> CAT
    CH <--> CAT

• Docker and Docker Compose
• Minimum 4GB RAM

docker compose up -d

Services:

• otel-collector: localhost:4318 - OTLP/HTTP ingestion (batches requests for efficient Parquet file sizes)
• otlp2parquet: Converts OTLP to Parquet and writes to S3
• otelgen: Generates test telemetry data (10k logs/sec)
• ClickHouse: localhost:8123
• Grafana: localhost:3000 - Dashboard UI
• rustfs console: localhost:9001 - S3 web UI (rustfsuser/rustfspassword)
• ice-rest-catalog: localhost:5001 - Iceberg REST catalog
• log-sync: Automatically syncs parquet files to Iceberg (every 60s by default)

Later, we can hook up OpenTelemetry SDKs or Collectors, but for now let's send some test logs to verify our set-up. Run this command 2-3 times:

curl -X POST http://localhost:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{
    "resourceLogs": [{
      "resource": {
        "attributes": [{"key": "service.name", "value": {"stringValue": "my-app"}}]
      },
      "scopeLogs": [{
        "scope": {"name": "my-scope"},
        "logRecords": [{
          "timeUnixNano": "'$(date +%s)'000000000",
          "severityText": "INFO",
          "body": {"stringValue": "Hello from my-app!"}
        }]
      }]
    }]
  }'

The log-sync service automatically registers new parquet files with Iceberg every 60 seconds (configurable via LOG_SYNC_INTERVAL env var).

Just send logs and query - no manual registration needed!

Logs are batched and flushed every 10 seconds (or 200k rows / 128MB), so they may take a moment to appear.

There is a Grafana dashboard with some basic panels for viewing logs (available at localhost:3000)

Alternatively, you can query the Iceberg tables directly from ClickHouse:

# Using clickhouse client
clickhouse client --query "SELECT service_name, severity_text, body, timestamp FROM ice.\`otel.logs\`"

The ice database is configured as a DataLakeCatalog pointing to ice-rest-catalog.

The schema follows the OpenTelemetry Collector Exporter for ClickHouse (snake_case). For example, otel.logs:

-- Recent logs
SELECT timestamp, service_name, severity_text, body
FROM ice.`otel.logs`
ORDER BY timestamp DESC
LIMIT 10;

-- Severity count by service
SELECT service_name, severity_text, count()
FROM ice.`otel.logs`
GROUP BY service_name, severity_text;

The otlp2parquet service exposes OTLP-compatible endpoints that can talk with any OTLP-compatible source. You can export telemetry from an application or agent, or you can use something like otelgen to generate more test telemetry to experiment with.

Configure your app's OTLP exporter:

# environment variables
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/json

exporters:
  otlphttp:
    endpoint: http://localhost:4318
    tls:
      insecure: true

service:
  pipelines:
    logs:
      exporters: [otlphttp]

You shouldn't need to do this, but if you want to manually register files:

# List parquet files
docker run --rm --network demo-otel-parquet-antalya_default --entrypoint="" minio/mc sh -c "
mc alias set local http://rustfs:9000 rustfsuser rustfspassword >/dev/null 2>&1
mc find local/bucket1/logs --name '*.parquet'
"

# Create namespace (first time only)
docker compose run --rm ice create-namespace otel

# Insert files using HTTP URLs (use http://rustfs:9000/bucket1/... instead of s3://bucket1/...)
docker compose run --rm ice insert -p --skip-duplicates otel.logs \
  "http://rustfs:9000/bucket1/logs/my-app/year=2026/month=01/day=12/hour=16/file.parquet"

Some data is stored locally in ./data:

# Stop services
docker compose down

# Remove all data
docker compose down -v
rm -rf ./data

• otlp2parquet: many thanks to @smithclay for creating the OTLP to Parquet converter used in this project.

This project is a demo created by Altinity to demonstrate using OpenTelemetry, ClickHouse, Iceberg, and Parquet for a scalable open-source observability data lakehouse. If you have questions, you can join us on Slack or log an issue on GitHub.

If you have any ideas to make the project better (especially if you have the code that makes the project better), just log an issue or submit a PR. We'd love to hear from you!

All code, unless specified otherwise, is licensed under the Apache-2.0 license. Copyright (c) 2026 Altinity, Inc. Altinity.Cloud®, and Altinity Stable® are registered trademarks of Altinity, Inc. ClickHouse® is a registered trademark of ClickHouse, Inc.; Altinity is not affiliated with or associated with ClickHouse, Inc. Kubernetes, MySQL, and PostgreSQL are trademarks and property of their respective owners.