Skip to content

fix(helm): allow overriding CODER_PPROF_ADDRESS and CODER_PROMETHEUS_ADDRESS#21714

Merged
uzair-coder07 merged 5 commits intomainfrom
fix/helm-pprof-prometheus-address-override
Feb 3, 2026
Merged

fix(helm): allow overriding CODER_PPROF_ADDRESS and CODER_PROMETHEUS_ADDRESS#21714
uzair-coder07 merged 5 commits intomainfrom
fix/helm-pprof-prometheus-address-override

Conversation

@blinkagent
Copy link
Contributor

@blinkagent blinkagent bot commented Jan 27, 2026

Summary

Previously, CODER_PPROF_ADDRESS and CODER_PROMETHEUS_ADDRESS were hardcoded in the Helm chart template to 0.0.0.0:6060 and 0.0.0.0:2112 respectively. These values could not be overridden via coder.env values because the hardcoded values were set first in the template, and Kubernetes uses the first occurrence of duplicate env vars.

This was a security concern because binding to 0.0.0.0 exposes these endpoints to any pod in the cluster:

  • pprof can expose sensitive runtime information (goroutine stacks, heap profiles, CPU profiles that may contain memory contents)
  • Prometheus metrics may contain sensitive operational data

Changes

  1. helm/coder/templates/_coder.tpl: Added logic to check if the user has set CODER_PPROF_ADDRESS or CODER_PROMETHEUS_ADDRESS in coder.env before applying the default values. If the user provides a value, the hardcoded default is skipped.

  2. helm/coder/values.yaml: Updated documentation to:

    • Remove these vars from the "cannot be overridden" list
    • Add them to a new "can be overridden" section with security recommendations

  3. Tests: Added test cases for both override scenarios with corresponding golden files.

Usage

Users can now restrict pprof and prometheus to localhost only:

coder:
  env:
    - name: CODER_PPROF_ADDRESS
      value: "127.0.0.1:6060"
    - name: CODER_PROMETHEUS_ADDRESS  
      value: "127.0.0.1:2112"

Local Testing

To verify the fix locally:

# Update helm dependencies
cd helm/coder && helm dependency update

# Test default behavior (should show 0.0.0.0)
helm template coder . -f tests/testdata/default_values.yaml --namespace default | grep -A1 'CODER_PPROF_ADDRESS\|CODER_PROMETHEUS_ADDRESS'

# Test pprof override (should show 127.0.0.1:6060)
helm template coder . -f tests/testdata/pprof_address_override.yaml --namespace default | grep -A1 'CODER_PPROF_ADDRESS'

# Test prometheus override (should show 127.0.0.1:2112)
helm template coder . -f tests/testdata/prometheus_address_override.yaml --namespace default | grep -A1 'CODER_PROMETHEUS_ADDRESS'

# Run Go tests
cd tests && go test . -v

Fixes #21713

@blink-so
fix(helm): allow overriding CODER_PPROF_ADDRESS and CODER_PROMETHEUS_…
9503e84 
…ADDRESS

Previously, these environment variables were hardcoded in the Helm chart
template and could not be overridden via coder.env values. This was a
security concern because binding to 0.0.0.0 exposes these endpoints to
any pod in the cluster.

This change:
- Checks if the user has set CODER_PPROF_ADDRESS or CODER_PROMETHEUS_ADDRESS
  in coder.env before applying the default values
- Updates values.yaml documentation to reflect that these can now be
  overridden (and recommends 127.0.0.1 for security)
- Adds test cases for both override scenarios

Users can now restrict pprof and prometheus to localhost only:

```yaml
coder:
  env:
    - name: CODER_PPROF_ADDRESS
      value: "127.0.0.1:6060"
    - name: CODER_PROMETHEUS_ADDRESS
      value: "127.0.0.1:2112"
```

Fixes #21713
@coder-tasks
Copy link
Contributor

coder-tasks bot commented Jan 27, 2026 •
edited by uzair-coder07
Loading

Documentation Check

Updates Needed

  • docs/admin/integrations/prometheus.md - Update the Kubernetes deployment section (lines 31-37) to reflect that CODER_PROMETHEUS_ADDRESS can now be overridden. The current text says it "cannot be changed" which is no longer accurate after this PR. Should mention the security best practice of setting it to 127.0.0.1:2112 to restrict access to localhost only.

Recommendations

Consider adding a security note to docs/admin/integrations/prometheus.md about the security implications of binding to 0.0.0.0 vs 127.0.0.1 for both pprof and Prometheus endpoints in Kubernetes deployments, similar to what's documented in the PR description and the updated values.yaml.

Automated review via Coder Tasks

@uzair-coder07
Copy link
Contributor

uzair-coder07 commented Jan 27, 2026

@deansheather Although the PR was created by Blink, I have reviewed it's work. Would appreciate a review from you if you can. Please let me know.

@deansheather
Copy link
Member

deansheather commented Jan 29, 2026

If people rebind to localhost, can they even scrape the data into prometheus or whatever? To me it sounds like it wouldn't work, and at that point why even leave them enabled? I think we support disabling these, right?

@blinkagent
Copy link
Contributor Author

blinkagent bot commented Jan 29, 2026

You're right that binding to 127.0.0.1 limits the use cases significantly. Here's when it would still be useful:

  1. kubectl port-forward - You can still access pprof/prometheus from localhost via port-forward for debugging, even when bound to 127.0.0.1. This is the main use case for pprof (ad-hoc debugging).

  2. Sidecar containers - Sidecars in the same pod share the network namespace, so they can access 127.0.0.1 endpoints. Some setups use sidecars for metrics collection.

But you raise a valid point - for Prometheus scraping via a ServiceMonitor or external scraper, 127.0.0.1 won't work. In that case, users should either:

  • Keep the default 0.0.0.0 and rely on NetworkPolicies to restrict access
  • Disable the endpoint entirely if they don't need it

The original issue was specifically about pprof where the customer wanted to prevent other pods from grabbing profiles (security concern), but still wanted the ability to use kubectl port-forward for debugging when needed.

Should I update the PR description and values.yaml docs to be clearer about these tradeoffs? Or do you think this flexibility isn't worth the added complexity and we should just keep them non-overridable?

@rowansmithau
Copy link
Contributor

rowansmithau commented Jan 29, 2026

I think this is more about allowing customers to declare their own desired configuration when it comes to security matters like these. Per Blink there are definitely still use cases for binding to localhost that would be helpful for customers.

@coder-tasks
Copy link
Contributor

coder-tasks bot commented Feb 2, 2026

Documentation Check

Previous Feedback

Not yet addressed - The documentation update identified in the previous review is still needed.

Updates Needed

  • docs/admin/integrations/prometheus.md - Update the Kubernetes deployment section (lines 31-37) to reflect that CODER_PROMETHEUS_ADDRESS can now be overridden. The current text says it "will be set by default" without mentioning that users can override it. Should mention the security best practice of setting it to 127.0.0.1:2112 to restrict access to localhost only (useful with kubectl port-forward or sidecar containers, but note that it prevents external Prometheus scraping via ServiceMonitor).

Recommendations

Consider adding a brief security note explaining the tradeoffs of binding to 0.0.0.0 vs 127.0.0.1:

  • 0.0.0.0 allows external scraping (ServiceMonitor) but exposes metrics to any pod in the cluster
  • 127.0.0.1 restricts access but requires kubectl port-forward or sidecar containers for scraping

Automated review via Coder Tasks

@uzair-coder07 uzair-coder07 merged commit 892b226 into main Feb 3, 2026
27 checks passed
@uzair-coder07 uzair-coder07 deleted the fix/helm-pprof-prometheus-address-override branch February 3, 2026 01:03
@github-actions github-actions bot locked and limited conversation to collaborators Feb 3, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@deansheather deansheather deansheather approved these changes

@rowansmithau rowansmithau Awaiting requested review from rowansmithau

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

bug: Allow overriding CODER_PPROF_ADDRESS via Helm chart values

3 participants

@uzair-coder07 @deansheather @rowansmithau