fix(dashboard): add FlowSchema to exempt BFF from API throttling#2121
fix(dashboard): add FlowSchema to exempt BFF from API throttling#2121
Conversation
The dashboard BFF service account (incloud-web-web) falls under the default "service-accounts" FlowSchema which maps to the "workload-low" priority level. Under load, this causes API Priority and Fairness to return 429 (Too Many Requests) responses to the BFF, resulting in 500 errors for dashboard users. Add a FlowSchema that maps the BFF service account to the "exempt" priority level to prevent APF throttling of dashboard API requests. Co-Authored-By: Claude <[email protected]> Signed-off-by: Andrei Kvapil <[email protected]>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a critical performance issue where the dashboard's Backend For Frontend (BFF) service was experiencing API throttling, leading to 429 errors under heavy load. By implementing a new Kubernetes FlowSchema, the BFF service account is now explicitly exempted from API Priority and Fairness, ensuring stable operation and improved responsiveness for the dashboard. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
kvaps
requested review from
IvanHunters,
androndo,
lexfrei,
lllamnyp and
sircthulhu
as code owners
dosubot
bot
added
size:S
📝 WalkthroughWalkthroughAdds a new Kubernetes FlowSchema manifest that grants the incloud-web-web ServiceAccount broad API access permissions across all resources and namespaces, linked to the exempt PriorityLevelConfiguration with a matchingPrecedence of 2. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs). Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](proxy.php?url=https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a FlowSchema to exempt the dashboard's backend-for-frontend (BFF) service account from API server throttling. While this addresses the issue of 429 Too Many Requests errors, the implementation introduces potential stability and security risks. The FlowSchema uses overly permissive resource rules, matching all possible API requests, which goes against the principle of least privilege. Additionally, it assigns the requests to the exempt priority level, which is generally discouraged for workloads as it completely bypasses API server protection mechanisms. My review includes suggestions to tighten the resource rules and to consider using a custom priority level with a high limit instead of complete exemption.
| resourceRules: | ||
| - verbs: ["*"] | ||
| apiGroups: ["*"] | ||
| resources: ["*"] | ||
| namespaces: ["*"] | ||
| clusterScope: true |
There was a problem hiding this comment.
The resourceRules are configured to match all possible API requests (verbs: ["*"], apiGroups: ["*"], resources: ["*"], namespaces: ["*"], clusterScope: true). This is overly permissive and goes against the principle of least privilege. While this FlowSchema doesn't grant permissions, it exempts the service account from API throttling entirely. In case of a compromise or a bug in the BFF, this could be abused to flood the API server with requests, potentially leading to a denial-of-service condition for other components in the cluster.
Please scope down these rules to only match the API requests that the incloud-web-web service account is authorized to make and needs exemption for. You should define more specific rules based on the actual permissions granted to the service account via its RBAC roles.
| spec: | ||
| matchingPrecedence: 2 | ||
| priorityLevelConfiguration: | ||
| name: exempt |
There was a problem hiding this comment.
Using the exempt priority level completely bypasses API Priority and Fairness for these requests. This level is typically reserved for the most critical system components. For a workload like the dashboard BFF, it's recommended to create a dedicated PriorityLevelConfiguration with a high concurrency limit instead. This would still prevent throttling under normal load while retaining a safety net against request floods that could impact the API server's stability.
Consider creating a new PriorityLevelConfiguration and referencing it here.
![coderabbitai[bot]](proxy.php?url=https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
packages/system/dashboard/templates/flowschema.yaml (2)
6-6:matchingPrecedence: 2collides with built-inprobesFlowSchema.The built-in
probesFlowSchema also uses precedence 2. When two FlowSchemas share the same precedence, the one with the lexicographically smaller name wins. Sincecozy-dashboard-exempt<probes, this FlowSchema will take priority for any overlapping requests. This is likely fine, but consider using a slightly higher number (e.g., 3-10) to avoid ambiguity and potential conflicts with future built-in FlowSchemas.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/system/dashboard/templates/flowschema.yaml` at line 6, The FlowSchema "cozy-dashboard-exempt" currently sets matchingPrecedence: 2 which collides with the built-in "probes" FlowSchema; update the matchingPrecedence for "cozy-dashboard-exempt" to a higher value (e.g., 3–10) so it no longer shares precedence with "probes" and avoids lexicographic tie-breaker ambiguity.
1-20: Consider addingnonResourceRulesfor complete throttling exemption.The FlowSchema only defines
resourceRules, which matches requests to Kubernetes resources. If the BFF also makes non-resource requests (e.g., discovery endpoints like/api,/apis,/openapi/v2, or health checks), those requests will still fall through to other FlowSchemas and could be throttled.To fully exempt the BFF from throttling, consider adding a
nonResourceRulesentry:Proposed fix to add nonResourceRules
rules: - subjects: - kind: ServiceAccount serviceAccount: name: incloud-web-web namespace: {{ .Release.Namespace }} resourceRules: - verbs: ["*"] apiGroups: ["*"] resources: ["*"] namespaces: ["*"] clusterScope: true + nonResourceRules: + - verbs: ["*"] + nonResourceURLs: ["*"]🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/system/dashboard/templates/flowschema.yaml` around lines 1 - 20, The FlowSchema named "cozy-dashboard-exempt" currently only lists resourceRules so non-resource endpoints remain subject to throttling; update the spec for FlowSchema "cozy-dashboard-exempt" (the block that targets ServiceAccount incloud-web-web in namespace {{ .Release.Namespace }}) to add an appropriate nonResourceRules entry (mirroring the exemption intent of resourceRules) that matches non-resource paths (e.g., "*" or health/discovery paths) so the BFF's non-resource requests are also exempted from throttling.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@packages/system/dashboard/templates/flowschema.yaml`:
- Line 6: The FlowSchema "cozy-dashboard-exempt" currently sets
matchingPrecedence: 2 which collides with the built-in "probes" FlowSchema;
update the matchingPrecedence for "cozy-dashboard-exempt" to a higher value
(e.g., 3–10) so it no longer shares precedence with "probes" and avoids
lexicographic tie-breaker ambiguity.
- Around line 1-20: The FlowSchema named "cozy-dashboard-exempt" currently only
lists resourceRules so non-resource endpoints remain subject to throttling;
update the spec for FlowSchema "cozy-dashboard-exempt" (the block that targets
ServiceAccount incloud-web-web in namespace {{ .Release.Namespace }}) to add an
appropriate nonResourceRules entry (mirroring the exemption intent of
resourceRules) that matches non-resource paths (e.g., "*" or health/discovery
paths) so the BFF's non-resource requests are also exempted from throttling.
|
Successfully created backport PR for |
1 participant
Summary
cozy-dashboard-exemptto exempt the dashboard BFF service account (incloud-web-web) from API Priority and Fairness throttlingservice-accountsFlowSchema →workload-lowpriority level, which causes 429 responses under loadTest plan
kubectl get flowschema cozy-dashboard-exemptSummary by CodeRabbit