Skip to content

docs(monitoring): add custom metrics collection guide#444

Merged
kvaps merged 2 commits intomainfrom
docs/custom-metrics-collection
Mar 11, 2026
Merged

docs(monitoring): add custom metrics collection guide#444
kvaps merged 2 commits intomainfrom
docs/custom-metrics-collection

Conversation

@IvanHunters
Copy link
Contributor

@IvanHunters IvanHunters commented Mar 10, 2026

Summary

  • Add a new documentation page "Custom Metrics Collection" to the monitoring section (v0 and v1)
  • Covers how tenants connect their own Prometheus exporters using VMServiceScrape and VMPodScrape CRDs
  • Explains the namespace labeling mechanism (namespace.cozystack.io/monitoring) required for tenant VMAgent target discovery

Test plan

  • Verify page renders correctly with make serve
  • Confirm page appears in sidebar between Logs and Alerting
  • Check that the internal ref link to Setup page resolves

Summary by CodeRabbit

  • Documentation
    • Added comprehensive guides for configuring tenant custom metrics collection via VMServiceScrape and VMPodScrape: prerequisites, namespace-label discovery requirements, example manifests and kubectl commands, key field explanations, verification steps (VMAgent targets, Grafana/PromQL checks), and troubleshooting tips.

Explain how tenants can connect their own Prometheus exporters to the
monitoring stack using VMServiceScrape and VMPodScrape CRDs, covering
namespace labeling, example manifests, and verification steps.

Signed-off-by: ohotnikov.ivan <[email protected]>
@netlify
Copy link

netlify bot commented Mar 10, 2026

Deploy Preview for cozystack ready!

Name Link
🔨 Latest commit c584862
🔍 Latest deploy log https://app.netlify.com/projects/cozystack/deploys/69b078ab6482c700083d8ad6
😎 Deploy Preview https://deploy-preview-444--cozystack.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 10, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ccee2a41-6d47-4128-a8c1-8e48ab04e0a9

📥 Commits

Reviewing files that changed from the base of the PR and between a36d356 and c584862.

📒 Files selected for processing (2)
  • content/en/docs/v0/operations/services/monitoring/custom-metrics.md
  • content/en/docs/v1/operations/services/monitoring/custom-metrics.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • content/en/docs/v0/operations/services/monitoring/custom-metrics.md
  • content/en/docs/v1/operations/services/monitoring/custom-metrics.md

📝 Walkthrough

Walkthrough

Adds two new documentation pages (v0 and v1) describing how to collect custom metrics for tenant monitoring using VictoriaMetrics VMServiceScrape and VMPodScrape. Covers prerequisites, namespace labeling behavior, example manifests, key fields, verification steps, and troubleshooting.

Changes

Cohort / File(s) Summary
Custom Metrics Documentation
content/en/docs/v0/operations/services/monitoring/custom-metrics.md, content/en/docs/v1/operations/services/monitoring/custom-metrics.md
Added comprehensive docs for integrating custom metrics via VictoriaMetrics VMServiceScrape and VMPodScrape: prerequisites, discovery/namespace-labeling rules, YAML examples and field explanations, kubectl commands for apply/verify, Grafana query examples, and troubleshooting steps.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

🐰 I hopped through namespaces, labels tight,
Scraped metrics glowing in the night,
YAML stitched and targets found,
Grafana sings without a sound,
Monitoring hops — everything's bright! ✨

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: adding documentation for custom metrics collection in the monitoring section.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch docs/custom-metrics-collection

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces comprehensive documentation for integrating custom metrics into the Cozystack tenant monitoring stack. It provides clear instructions and examples for tenants to leverage VMServiceScrape and VMPodScrape resources, enabling them to collect metrics from their applications and exporters. The new guide also clarifies the necessary Kubernetes namespace labeling to ensure proper discovery by the tenant VMAgent, significantly enhancing the platform's observability capabilities for custom workloads.

Highlights

  • New Documentation Page: Added a new documentation page titled "Custom Metrics Collection" to the monitoring section for both v0 and v1 of the platform.
  • Prometheus Exporter Integration: The guide details how tenants can connect their own Prometheus exporters using VMServiceScrape and VMPodScrape Custom Resource Definitions (CRDs).
  • Namespace Labeling Mechanism: Explained the essential namespace labeling mechanism (namespace.cozystack.io/monitoring) required for the tenant VMAgent to discover scrape targets.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • content/en/docs/v0/operations/services/monitoring/custom-metrics.md
    • Added a new documentation file detailing custom metrics collection for v0.
  • content/en/docs/v1/operations/services/monitoring/custom-metrics.md
    • Added a new documentation file detailing custom metrics collection for v1.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@IvanHunters IvanHunters marked this pull request as ready for review March 10, 2026 19:59
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds excellent documentation for collecting custom metrics. I've provided a few suggestions to improve the user experience by making the kubectl commands more self-contained and to clarify the usage of port selection in VMPodScrape. Overall, the documentation is clear and well-structured.

Note: Security Review has been skipped due to the limited scope of the PR.

Comment on lines +59 to +81
Suppose you have a Service named `my-app` in namespace `my-app-ns` that exposes metrics on port `metrics` at path `/metrics`:

```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
name: my-app-metrics
namespace: my-app-ns
spec:
selector:
matchLabels:
app: my-app
endpoints:
- port: metrics
path: /metrics
interval: 30s
```

Apply the resource:

```bash
kubectl apply --filename vmservicescrape.yaml --namespace my-app-ns
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To make the instructions easier to follow, consider combining the YAML definition and the kubectl apply command into a single, copy-pastable block using a heredoc. This prevents users from needing to manually create a file and reduces the potential for error.

Suggested change
Suppose you have a Service named `my-app` in namespace `my-app-ns` that exposes metrics on port `metrics` at path `/metrics`:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
name: my-app-metrics
namespace: my-app-ns
spec:
selector:
matchLabels:
app: my-app
endpoints:
- port: metrics
path: /metrics
interval: 30s
```
Apply the resource:
```bash
kubectl apply --filename vmservicescrape.yaml --namespace my-app-ns
```
Suppose you have a Service named `my-app` in namespace `my-app-ns` that exposes metrics on port `metrics` at path `/metrics`. You can create and apply a `VMServiceScrape` resource with the following command:
```bash
cat <<EOF | kubectl apply -n my-app-ns -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
name: my-app-metrics
namespace: my-app-ns
spec:
selector:
matchLabels:
app: my-app
endpoints:
- port: metrics
path: /metrics
interval: 30s
EOF

Comment on lines +98 to +127
Suppose you have Pods labeled `app: my-worker` that expose metrics on port `9090` at path `/metrics`:

```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: my-worker-metrics
namespace: my-app-ns
spec:
selector:
matchLabels:
app: my-worker
podMetricsEndpoints:
- port: "9090"
path: /metrics
```

Apply the resource:

```bash
kubectl apply --filename vmpodscrape.yaml --namespace my-app-ns
```

### Key Fields

| Field | Description |
| --- | --- |
| `spec.selector.matchLabels` | Label selector to find the target Pods |
| `spec.podMetricsEndpoints[].port` | Port name or number on the Pod to scrape |
| `spec.podMetricsEndpoints[].path` | HTTP path for metrics (default: `/metrics`) |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To improve clarity and user experience, this section can be improved in two ways:

  1. The kubectl apply command can be combined with the YAML definition using a heredoc. This creates a single copy-pastable command.
  2. The VMPodScrape example uses port: "9090", which implies the port is named 9090. It's more common and clearer to specify a port number using targetPort. The key fields table should also be updated to reflect this.
Suppose you have Pods labeled `app: my-worker` that expose metrics on port `9090` at path `/metrics`. You can create and apply a `VMPodScrape` resource with the following command:

```bash
cat <<EOF | kubectl apply -n my-app-ns -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
  name: my-worker-metrics
  namespace: my-app-ns
spec:
  selector:
    matchLabels:
      app: my-worker
  podMetricsEndpoints:
    - targetPort: 9090
      path: /metrics
EOF

Key Fields

Field Description
spec.selector.matchLabels Label selector to find the target Pods
spec.podMetricsEndpoints[].targetPort Port number or name on the Pod to scrape
spec.podMetricsEndpoints[].path HTTP path for metrics (default: /metrics)

Comment on lines +59 to +81
Suppose you have a Service named `my-app` in namespace `my-app-ns` that exposes metrics on port `metrics` at path `/metrics`:

```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
name: my-app-metrics
namespace: my-app-ns
spec:
selector:
matchLabels:
app: my-app
endpoints:
- port: metrics
path: /metrics
interval: 30s
```

Apply the resource:

```bash
kubectl apply --filename vmservicescrape.yaml --namespace my-app-ns
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To make the instructions easier to follow, consider combining the YAML definition and the kubectl apply command into a single, copy-pastable block using a heredoc. This prevents users from needing to manually create a file and reduces the potential for error.

Suggested change
Suppose you have a Service named `my-app` in namespace `my-app-ns` that exposes metrics on port `metrics` at path `/metrics`:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
name: my-app-metrics
namespace: my-app-ns
spec:
selector:
matchLabels:
app: my-app
endpoints:
- port: metrics
path: /metrics
interval: 30s
```
Apply the resource:
```bash
kubectl apply --filename vmservicescrape.yaml --namespace my-app-ns
```
Suppose you have a Service named `my-app` in namespace `my-app-ns` that exposes metrics on port `metrics` at path `/metrics`. You can create and apply a `VMServiceScrape` resource with the following command:
```bash
cat <<EOF | kubectl apply -n my-app-ns -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
name: my-app-metrics
namespace: my-app-ns
spec:
selector:
matchLabels:
app: my-app
endpoints:
- port: metrics
path: /metrics
interval: 30s
EOF

Comment on lines +98 to +127
Suppose you have Pods labeled `app: my-worker` that expose metrics on port `9090` at path `/metrics`:

```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: my-worker-metrics
namespace: my-app-ns
spec:
selector:
matchLabels:
app: my-worker
podMetricsEndpoints:
- port: "9090"
path: /metrics
```

Apply the resource:

```bash
kubectl apply --filename vmpodscrape.yaml --namespace my-app-ns
```

### Key Fields

| Field | Description |
| --- | --- |
| `spec.selector.matchLabels` | Label selector to find the target Pods |
| `spec.podMetricsEndpoints[].port` | Port name or number on the Pod to scrape |
| `spec.podMetricsEndpoints[].path` | HTTP path for metrics (default: `/metrics`) |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To improve clarity and user experience, this section can be improved in two ways:

  1. The kubectl apply command can be combined with the YAML definition using a heredoc. This creates a single copy-pastable command.
  2. The VMPodScrape example uses port: "9090", which implies the port is named 9090. It's more common and clearer to specify a port number using targetPort. The key fields table should also be updated to reflect this.
Suppose you have Pods labeled `app: my-worker` that expose metrics on port `9090` at path `/metrics`. You can create and apply a `VMPodScrape` resource with the following command:

```bash
cat <<EOF | kubectl apply -n my-app-ns -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
  name: my-worker-metrics
  namespace: my-app-ns
spec:
  selector:
    matchLabels:
      app: my-worker
  podMetricsEndpoints:
    - targetPort: 9090
      path: /metrics
EOF

Key Fields

Field Description
spec.selector.matchLabels Label selector to find the target Pods
spec.podMetricsEndpoints[].targetPort Port number or name on the Pod to scrape
spec.podMetricsEndpoints[].path HTTP path for metrics (default: /metrics)

The namespace labeling mechanism and VMAgent selector configuration
are internal details that should not be exposed in user-facing docs.

Signed-off-by: ohotnikov.ivan <[email protected]>
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
content/en/docs/v0/operations/services/monitoring/custom-metrics.md (1)

110-112: Consider demonstrating named port usage for consistency.

The VMPodScrape example uses a numeric port "9090", while the VMServiceScrape example uses a named port "metrics". Consider adding a comment or second example showing named port usage (e.g., port: "metrics") to demonstrate both options and maintain consistency with the VMServiceScrape section.

📝 Optional: Add example with named port
   podMetricsEndpoints:
-    - port: "9090"
+    - port: "9090"  # Can also use named port: port: "metrics"
       path: /metrics
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@content/en/docs/v0/operations/services/monitoring/custom-metrics.md` around
lines 110 - 112, Add a second example or inline comment in the VMPodScrape
section showing named-port usage to match VMServiceScrape: modify the
podMetricsEndpoints block (symbol: podMetricsEndpoints) to include an
alternative like port: "metrics" and a short note that named ports are
equivalent to numeric ports, so readers see both options; ensure the example
mirrors VMServiceScrape's use of the named port and clearly labels it as an
alternative to port: "9090".
content/en/docs/v1/operations/services/monitoring/custom-metrics.md (1)

110-112: Consider demonstrating named port usage for consistency.

The VMPodScrape example uses a numeric port "9090", while the VMServiceScrape example uses a named port "metrics". Consider adding a comment or second example showing named port usage (e.g., port: "metrics") to demonstrate both options and maintain consistency with the VMServiceScrape section.

📝 Optional: Add example with named port
   podMetricsEndpoints:
-    - port: "9090"
+    - port: "9090"  # Can also use named port: port: "metrics"
       path: /metrics
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@content/en/docs/v1/operations/services/monitoring/custom-metrics.md` around
lines 110 - 112, Update the VMPodScrape example around podMetricsEndpoints to
show named-port usage for consistency with VMServiceScrape: add either a comment
explaining you can use a named port (e.g., "metrics") or include a second
podMetricsEndpoints example entry using port: "metrics" and the same path:
/metrics; ensure you reference the existing podMetricsEndpoints block so readers
see both numeric ("9090") and named ("metrics") port options.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@content/en/docs/v0/operations/services/monitoring/custom-metrics.md`:
- Around line 110-112: Add a second example or inline comment in the VMPodScrape
section showing named-port usage to match VMServiceScrape: modify the
podMetricsEndpoints block (symbol: podMetricsEndpoints) to include an
alternative like port: "metrics" and a short note that named ports are
equivalent to numeric ports, so readers see both options; ensure the example
mirrors VMServiceScrape's use of the named port and clearly labels it as an
alternative to port: "9090".

In `@content/en/docs/v1/operations/services/monitoring/custom-metrics.md`:
- Around line 110-112: Update the VMPodScrape example around podMetricsEndpoints
to show named-port usage for consistency with VMServiceScrape: add either a
comment explaining you can use a named port (e.g., "metrics") or include a
second podMetricsEndpoints example entry using port: "metrics" and the same
path: /metrics; ensure you reference the existing podMetricsEndpoints block so
readers see both numeric ("9090") and named ("metrics") port options.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: fce1b07a-8b7a-4ece-9df9-8759e32dd420

📥 Commits

Reviewing files that changed from the base of the PR and between 2d43914 and a36d356.

📒 Files selected for processing (2)
  • content/en/docs/v0/operations/services/monitoring/custom-metrics.md
  • content/en/docs/v1/operations/services/monitoring/custom-metrics.md

@kvaps kvaps merged commit f46c4e3 into main Mar 11, 2026
6 checks passed
@kvaps kvaps deleted the docs/custom-metrics-collection branch March 11, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants