fix(sonarqube): bypass 10K result cap by fetching issues per-rule#448
Open
TheAuditorTool wants to merge 1 commit intoOWASP-Benchmark:masterfrom
Open
fix(sonarqube): bypass 10K result cap by fetching issues per-rule#448TheAuditorTool wants to merge 1 commit intoOWASP-Benchmark:masterfrom
TheAuditorTool wants to merge 1 commit intoOWASP-Benchmark:masterfrom
Conversation
6286d30 to
698ae83
Compare
Closes OWASP-Benchmark#33 The issues/search API enforces p*ps <= 10000. With PAGE_SIZE=500, page 21 returns HTTP 400. Fix: iterate per-rule instead of passing all ~600 rules in one query. Each single-rule query stays well under 10K. Also fixes off-by-one in page count (ceiling division) and adds HTTP status checking before reading response body.
698ae83 to
adef7bc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #33
Refs #187
Summary
The SonarQube
/api/issues/searchendpoint enforces a hard server-side limit:p * ps <= 10000. WithPAGE_SIZE = 500, page 21 requests offset 10,500 andreceives HTTP 400. For instances reporting >10K vulnerabilities, results are
either silently truncated or the script crashes.
This was reported in #33 (March 2017) and deferred during the #187 cleanup.
The original reporter's suggestion -- filter by individual rules to keep each
query under the cap -- is exactly what this PR implements.
Problem Chain
&rules=squid:XXXper-rule.32933c4e6addedSonarReport.javawith native Java pagination. Passes ALL rules in one&rules=query -- still hits 10K cap for large result sets.dc9abba63fixed hostname.283e0e61fapplied Spotless formatting.What Changed
Single file changed:
src/main/java/org/owasp/benchmark/report/sonarqube/SonarReport.java1. Per-rule issue fetching (
main())Before: All ~600 Java rules are comma-joined into a single
&rules=queryparameter. If the aggregate result exceeds 10K issues, the API returns HTTP 400
or silently truncates.
After: Each rule is queried individually. A single rule produces far fewer
issues (typically 0-200 for Benchmark's 2,740 test cases), staying well under
the 10K cap.
This also fixes:
&&in the URL (&&rules=->&rules=)Why no duplicates: Each SonarQube issue belongs to exactly one rule.
Per-rule iteration produces disjoint result sets.
Performance: ~600 rules x 1 lightweight API call each. Most rules return 0
results (1 page). Total overhead: 1-5 minutes. The scan itself takes >= 1 hour.
2. Off-by-one in page count (
forAllPagesAt())Before:
(total / PAGE_SIZE) + 1-- always adds an extra page. When totalis an exact multiple of PAGE_SIZE (e.g., 500 issues), fires one unnecessary
empty-page request. At exactly 10K results, this triggers a forbidden page 21.
After:
(total + PAGE_SIZE - 1) / PAGE_SIZE-- standard ceiling division.3. HTTP error handling (
apiCall())Before:
getInputStream()on a non-200 response throws a genericIOExceptionwith "Server returned HTTP response code: 400 for URL: ...".For some error codes, it may silently read an error body that breaks Jackson
deserialization downstream.
After:
getResponseCode()is checked before reading the body. Non-200throws
IOException("SonarQube API returned HTTP " + status + " for " + apiPath)with clear context.
What Was NOT Changed
testcode/(2,740 files)SonarQubeResult.javaKeepAsJsonDeserializer.javarunSonarQube.shpom.xmlsetDoOutput(true)inapiCall()Known Limitations (future work)
hotspots/searchhas the same 10K cap but does not support per-rulefiltering. Benchmark's 2,740 test cases are unlikely to produce 10K+
hotspots, but an Enterprise instance could. Addressing this requires a
different partitioning strategy (by file or security category).
rules/searchalso paginates with the same mechanism. Default SonarQubehas ~3,000 rules total, well under 10K. Only relevant for instances with
10K+ custom rules.
Test Plan
scripts/runSonarQube.shon a machine with Dockerresults/Benchmark_*-sonarqube-v*.jsoncontains bothissuesandhotspotsarrayssonarqube-benchmarkcontainer after script completes