Automated setup script for deploying OpenTelemetry Operator, Collector, Kubernetes monitoring, and Events collection to your Kubernetes cluster with Last9 integration.

• ✅ One-command installation - Deploy everything with a single command
• ✅ Flexible deployment options - Install only what you need (logs, traces, metrics, events)
• ✅ Auto-instrumentation - Automatic instrumentation for Java, Python, Node.js, and more
• ✅ Kubernetes monitoring - Full cluster observability with kube-prometheus-stack
• ✅ Events collection - Capture and forward Kubernetes events
• ✅ Cluster identification - Automatic cluster name detection and attribution
• ✅ Tolerations support - Deploy on tainted nodes (control-plane, spot instances, etc.)
• ✅ Environment customization - Override deployment environment and cluster name

• kubectl configured to access your Kubernetes cluster
• helm (v3+) installed

Installs OpenTelemetry Operator, Collector, Kubernetes monitoring stack, and Events agent:

./last9-otel-setup.sh \
  token="Basic <your-base64-token>" \
  endpoint="<your-otlp-endpoint>" \
  monitoring-endpoint="<your-metrics-endpoint>" \
  username="<your-username>" \
  password="<your-password>"

curl -fsSL https://raw.githubusercontent.com/last9/l9-otel-operator/main/last9-otel-setup.sh | bash -s -- \
  token="Basic <your-token>" \
  endpoint="<your-otlp-endpoint>" \
  monitoring-endpoint="<your-metrics-endpoint>" \
  username="<user>" \
  password="<pass>"

For applications that need distributed tracing:

./last9-otel-setup.sh operator-only \
  token="Basic <your-token>" \
  endpoint="<your-otlp-endpoint>"

For log collection use cases:

./last9-otel-setup.sh logs-only \
  token="Basic <your-token>" \
  endpoint="<your-otlp-endpoint>"

For cluster metrics and monitoring:

./last9-otel-setup.sh monitoring-only \
  monitoring-endpoint="<your-metrics-endpoint>" \
  username="<your-username>" \
  password="<your-password>"

For Kubernetes events collection:

./last9-otel-setup.sh events-only \
  endpoint="<your-otlp-endpoint>" \
  token="Basic <your-base64-token>" \
  monitoring-endpoint="<your-metrics-endpoint>"

./last9-otel-setup.sh \
  token="..." \
  endpoint="..." \
  cluster="prod-us-east-1"

If not provided, the cluster name is auto-detected from kubectl config current-context.

./last9-otel-setup.sh \
  token="..." \
  endpoint="..." \
  env="production"

Default: staging for collector, local for auto-instrumentation.

For deploying on nodes with taints (e.g., control-plane, monitoring nodes):

./last9-otel-setup.sh \
  token="..." \
  endpoint="..." \
  tolerations-file=/path/to/tolerations.yaml

Example tolerations files are provided in the examples/ directory:

• tolerations-all-nodes.yaml - Deploy on all nodes including control-plane
• tolerations-monitoring-nodes.yaml - Deploy on dedicated monitoring nodes
• tolerations-spot-instances.yaml - Deploy on spot/preemptible instances
• tolerations-multi-taint.yaml - Handle multiple taints
• tolerations-nodeSelector-only.yaml - Use nodeSelector without tolerations
• tolerations-gpu-nodes.yaml - Deploy on GPU nodes with nvidia.com/gpu taints

The following placeholders are automatically replaced during installation:

• {{AUTH_TOKEN}} - Your Last9 authorization token
• {{OTEL_ENDPOINT}} - Your OTEL endpoint URL
• {{MONITORING_ENDPOINT}} - Your metrics endpoint URL

./last9-otel-setup.sh uninstall-all

# Uninstall only monitoring stack
./last9-otel-setup.sh uninstall function="uninstall_last9_monitoring"

# Uninstall only events agent
./last9-otel-setup.sh uninstall function="uninstall_events_agent"

# Uninstall OpenTelemetry components (operator + collector)
./last9-otel-setup.sh uninstall

After installation, verify the deployment:

# Check all pods in last9 namespace
kubectl get pods -n last9

# Check collector logs
kubectl logs -n last9 -l app.kubernetes.io/name=opentelemetry-collector

# Check monitoring stack
kubectl get prometheus -n last9

# Check events agent
kubectl get pods -n last9 -l app.kubernetes.io/name=last9-kube-events-agent

The script automatically sets up instrumentation for:

• ☕ Java - Automatic OTLP export
• 🐍 Python - Automatic OTLP export
• 🟢 Node.js - Automatic OTLP export
• 🔵 Go - Manual instrumentation supported
• 💎 Ruby - Coming soon

The OpenTelemetry Collector can automatically discover and scrape application metrics using Kubernetes service discovery with Prometheus-compatible scraping.

Note: This is an optional feature. Use last9-otel-collector-metrics-values.yaml to enable metrics scraping.

To enable application metrics scraping, deploy with the additional metrics configuration file:

# Deploy with metrics scraping enabled
helm upgrade --install last9-opentelemetry-collector open-telemetry/opentelemetry-collector \
  --namespace last9 \
  --version 0.125.0 \
  --values last9-otel-collector-values.yaml \
  --values last9-otel-collector-metrics-values.yaml

Configure Last9 Metrics Endpoint:

Before deploying, update these placeholders in last9-otel-collector-metrics-values.yaml:

• {{LAST9_METRICS_ENDPOINT}} - Your Last9 Prometheus remote write URL
• {{LAST9_METRICS_USERNAME}} - Your Last9 metrics username
• {{LAST9_METRICS_PASSWORD}} - Your Last9 metrics password

Add these annotations to your pod template or service to enable automatic metrics scraping:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"  # Optional, defaults to /metrics

That's it! Your application metrics will be automatically:

• Discovered - No manual configuration needed
• Scraped - Every 30 seconds by default
• Enriched - With pod, namespace, node labels
• Exported - To Last9 via Prometheus remote write

1. Automatic Discovery - OTel Collector watches Kubernetes API for all pods/services
2. Annotation-Based Filtering - Only scrapes resources with prometheus.io/scrape: "true"
3. Metadata Enrichment - Adds Kubernetes labels automatically (pod, namespace, node, app)
4. Direct Export - Sends metrics to Last9 Prometheus endpoint

This setup scales automatically:

• 1 service → Automatically scraped
• 1000 services → Automatically scraped
• No configuration changes needed when adding new services

Base Configuration: last9-otel-collector-values.yaml

• Traces and logs collection
• Basic OTLP receiver
• No metrics scraping

App Metrics Configuration: last9-otel-collector-metrics-values.yaml

• Prometheus receiver with kubernetes_sd_configs for auto-discovery
• prometheusremotewrite exporter for sending to Last9
• RBAC for Kubernetes API access
• Increased resource limits for collector pods
• BasicAuth extension for Last9 metrics endpoint

GPU + App Metrics Configuration: last9-otel-collector-gpu-values.yaml

• Everything in the app metrics configuration, plus:
• DCGM GPU metrics scraping (NVIDIA GPU Operator or GKE-managed DCGM — see variants in values file)
• Ray head/worker metrics scraping (KubeRay Operator)
• Cardinality control via metric keep-list for DCGM

Choose ONE metrics overlay:

• App metrics only: --values last9-otel-collector-values.yaml --values last9-otel-collector-metrics-values.yaml
• App + GPU metrics: --values last9-otel-collector-values.yaml --values last9-otel-collector-gpu-values.yaml

Check if metrics are being scraped:

# Check collector logs for scraping
kubectl logs -n last9 -l app.kubernetes.io/name=last9-otel-collector | grep kubernetes-pods

# Port-forward to collector metrics endpoint
kubectl port-forward -n last9 daemonset/last9-otel-collector 8888:8888

# Check scrape status
curl http://localhost:8888/metrics | grep scrape_samples_scraped

GPU and Ray metrics collection is opt-in. Use last9-otel-collector-gpu-values.yaml instead of the base metrics file to enable these scrape jobs. They use label-based discovery — no annotation changes needed on DCGM or Ray pods.

helm upgrade --install last9-opentelemetry-collector open-telemetry/opentelemetry-collector \
  --namespace last9 \
  --version 0.125.0 \
  --values last9-otel-collector-values.yaml \
  --values last9-otel-collector-gpu-values.yaml

Note: Use last9-otel-collector-gpu-values.yaml instead of last9-otel-collector-metrics-values.yaml — the GPU file already includes all application metrics scrape jobs.

• DCGM metrics (pick one):
  • Self-managed (EKS, AKS, bare-metal): NVIDIA GPU Operator installed (includes DCGM Exporter with app.kubernetes.io/name=nvidia-dcgm-exporter label)
  • GKE: GPU node pools enabled — GKE auto-deploys DCGM exporter in gke-managed-system namespace (label: app.kubernetes.io/name=gke-managed-dcgm-exporter)
• Ray metrics: KubeRay Operator installed (Ray pods carry ray.io/node-type and ray.io/cluster labels)

Important: The values file ships with two DCGM variants (A and B). Variant A (self-managed NVIDIA GPU Operator) is enabled by default. If you're on GKE, comment out Variant A and uncomment Variant B. See the inline comments in last9-otel-collector-gpu-values.yaml for details.

The DCGM job includes a cardinality keep-list limiting collection to 18 key metrics:

• Utilization: DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_MEM_COPY_UTIL, DCGM_FI_DEV_ENC_UTIL, DCGM_FI_DEV_DEC_UTIL
• Memory: DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_TOTAL
• Temperature & Power: DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_MEMORY_TEMP, DCGM_FI_DEV_POWER_USAGE
• Errors: DCGM_FI_DEV_XID_ERRORS, DCGM_FI_DEV_ECC_SBE_VOL_TOTAL, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL
• PCIe: DCGM_FI_DEV_PCIE_TX_THROUGHPUT, DCGM_FI_DEV_PCIE_RX_THROUGHPUT
• Clock & Performance: DCGM_FI_DEV_SM_CLOCK, DCGM_FI_DEV_MEM_CLOCK, DCGM_FI_DEV_PSTATE

To add more DCGM metrics, extend the metric_relabel_configs regex in last9-otel-collector-gpu-values.yaml.

If your GPU nodes have nvidia.com/gpu taints, the OTel Collector and node exporter need tolerations to schedule on those nodes:

./last9-otel-setup.sh \
  token="..." \
  endpoint="..." \
  monitoring-endpoint="..." \
  username="..." \
  password="..." \
  tolerations-file=examples/tolerations-gpu-nodes.yaml

See examples/tolerations-gpu-nodes.yaml for the toleration configuration.

For clusters with many GPU nodes, the collector handles additional scrape targets. Suggested resource scaling:

Override resources in your Helm values or pass a custom values file.

Before deploying, check which DCGM exporter is running in your cluster to pick the right variant:

# Check for GKE-managed DCGM exporter (Variant B)
kubectl get pods -n gke-managed-system -l app.kubernetes.io/name=gke-managed-dcgm-exporter

# Check for self-managed NVIDIA GPU Operator DCGM exporter (Variant A)
kubectl get pods -A -l app.kubernetes.io/name=nvidia-dcgm-exporter

• If the first command returns pods → use Variant B (uncomment it, comment out Variant A)
• If the second command returns pods → use Variant A (the default, no changes needed)
• If neither returns pods → your DCGM exporter is not yet installed (see Prerequisites)

# Verify DCGM exporter pods are discovered
kubectl logs -n last9 -l app.kubernetes.io/name=last9-otel-collector | grep dcgm-gpu-metrics

# Verify Ray head/worker pods are discovered
kubectl logs -n last9 -l app.kubernetes.io/name=last9-otel-collector | grep ray-head
kubectl logs -n last9 -l app.kubernetes.io/name=last9-otel-collector | grep ray-workers

# Check DCGM metrics are being scraped
kubectl port-forward -n last9 daemonset/last9-otel-collector 8888:8888
curl -s http://localhost:8888/metrics | grep -c "DCGM_FI_DEV"

# Check Ray metrics are being scraped
curl -s http://localhost:8888/metrics | grep scrape_samples_scraped | grep ray