Skip to content

Update Web Spider Param#364

Merged
payoub125 merged 2 commits intodevelopfrom
update-web-spider-param
Apr 3, 2026
Merged

Update Web Spider Param#364
payoub125 merged 2 commits intodevelopfrom
update-web-spider-param

Conversation

@payoub125
Copy link
Copy Markdown
Contributor

@payoub125 payoub125 commented Apr 3, 2026

Note

Medium Risk
Changes the route discovery config/CLI flag semantics and defaults (now ignoring cross-domain routes by default), which can significantly alter scan results and downstream integrations relying on the previous schema/flag behavior.

Overview
Route discovery now treats cross-domain filtering as a first-class setting. The discover route CLI flag is renamed from --ignore-base-url-match to --ignore-cross-domain, with updated default/description so route extraction and spidering can exclude non-target domains by default.

The DiscoverRouteConfig field is renamed from ignoreBaseUrlMatch to ignoreCrossDomain across the Fern definition and Go call sites, and all HTML/JS/network extractors plus redirect/static-asset filtering are updated to use the new field. Also fixes a JS extractor bug where IsURLAllowed was passed the wrong static-asset flag, and includes minor import/formatting cleanups.

Written by Cursor Bugbot for commit 88f873f. This will update automatically on new commits. Configure here.

@payoub125 payoub125 self-assigned this Apr 3, 2026
@payoub125 payoub125 force-pushed the update-web-spider-param branch from eeeb048 to 3791b23 Compare April 3, 2026 15:28
Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

@payoub125 payoub125 merged commit b8a82db into develop Apr 3, 2026
15 checks passed
@payoub125 payoub125 deleted the update-web-spider-param branch April 3, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants