Skip to content

Dev#431

Merged
mayanayza merged 25 commits intomainfrom
dev
Jan 6, 2026
Merged

Dev#431
mayanayza merged 25 commits intomainfrom
dev

Conversation

@mayanayza
Copy link
Collaborator

No description provided.

mayanayza and others added 25 commits January 4, 2026 19:57
- Add radio buttons to choose between generating new key or using existing
- Show text input when "Use existing API key" is selected
- Validate form before accepting pasted key (same pattern as generate flow)
- Reset state when switching between options or closing modal
- Disable radio buttons once key is set to prevent switching
- Check plan_status.is_some() instead of stripe_customer_id.is_some()
- A stripe_customer_id is created when checkout is initiated, before
  subscription is activated, so users who started but didn't complete
  checkout were incorrectly flagged as returning customers
- Now only users who have had an actual subscription (plan_status has
  a value) are considered returning customers and excluded from trials
- Fix race condition where identifyUser() was called before org loaded,
  causing demo users to be identified in PostHog
- Add 6 onboarding telemetry events to match backend tracking:
  - org_created: after registration
  - onboarding_modal_completed: after setup modal
  - plan_selected: on billing plan selection
  - first_api_key_created: on first API key creation
  - first_daemon_registered: when daemons first detected
  - first_topology_rebuild: on first topology rebuild
- Add trackEventOnce() utility for one-time events using localStorage
- Add 5 new "Core" category features included in all plans:
  unlimited_scans, unlimited_hosts, service_definitions,
  docker_integration, real_time_updates
- Update BillingPlanFeatures struct and all plan definitions
- Regenerate billing-plans.json and features.json fixtures
- Add TASK.md with research findings and implementation notes
- Add new arp module with platform-specific implementations:
  - broadcast.rs: pnet-based broadcast ARP for Linux/macOS
  - sendarp.rs: Windows native SendARP API (no Npcap required)
  - Platform dispatch in mod.rs with fallback support

- Change discovery flow from per-IP to batch subnet scanning
  - Groups IPs by subnet, scans all at once
  - ~2-3s for /24 subnet (down from 4-5s)

- Add use_npcap_arp config option (Windows only)
  - CLI: --use-npcap-arp
  - Env: SCANOPY_USE_NPCAP_ARP
  - Default: false (uses native SendARP)

- Remove old per-IP ARP functions from scanner.rs
- Add data parameter to track_event for Plunk contact metadata
- Add TelemetryOperation variants: CheckoutStarted, CheckoutCompleted,
  TrialStarted, TrialEnded, SubscriptionCancelled
- Publish checkout_started when creating Stripe checkout session
- Publish checkout_completed/trial_started on first subscription
- Publish trial_ended on trial-to-active or trial-to-cancelled transition
- Publish subscription_cancelled on subscription deletion
- Email subscriber passes metadata for billing events to enable
  Plunk segmentation and cart abandonment workflows
Restructure daemon logs to provide clear visibility into initialization,
registration, and runtime status:

Startup:
- Add ASCII art banner and structured info showing daemon ID, name,
  and config file path
- Add configuration summary block with server URL, network ID, mode,
  bind address, daemon URL (with auto-detected vs configured indicator),
  heartbeat interval, and concurrent scan setting (shows "auto" when
  using default auto-detection)
- Add detailed Docker capability detection showing connection method
  (local socket, HTTP proxy, or SSL proxy) with helpful error hints

Registration:
- Improve server connection status messages
- Show registration details including version and capabilities
- Display server version and minimum daemon version on success

Runtime:
- Move heartbeat/work-request logs from INFO to DEBUG to reduce noise
- Add periodic health summary every ~5 minutes showing status, uptime,
  heartbeat/poll count, and discovery state
- Add mode-specific ready messages explaining Push vs Pull behavior

General:
- Use consistent log target ("daemon") across all daemon components
- Add shutdown logging when daemon receives termination signal
- Remove redundant "Connecting to Docker" debug log
…iting

Replace sequential ARP scanning with broadcast ARP using a streaming
pipeline architecture. Hosts now flow directly from ARP discovery to
deep scanning without artificial phases, significantly improving
discovery speed.

Key changes:

Broadcast ARP with concurrent send/receive:
- Separate sender and receiver threads to avoid buffer overflow
- Sender thread sends packets at configurable rate limit
- Receiver thread continuously captures responses while sending
- Shared state (Arc<Mutex>) tracks discovered hosts for targeted retries

Targeted retries:
- Only retry IPs that haven't responded (like nmap/arp-scan)
- Configurable via `arp_retries` (default: 2 = 3 total attempts)
- Each round waits 3 seconds for responses before retrying
- 5 second final receive period for late arrivals

Rate limiting for enterprise switch compatibility:
- Configurable via `arp_rate_pps` (default: 50 packets/second)
- Prevents triggering Dynamic ARP Inspection (DAI) on managed switches
- Cisco untrusted ports default to 15 pps before errdisable
- Documentation added explaining the issue and configuration options

Streaming pipeline architecture:
- ARP results feed directly to deep scan queue via tokio channel
- Deep scans start immediately as hosts are discovered
- No waiting for full ARP completion before scanning
- 30-second grace period after last activity for late arrivals

Phase-weighted progress reporting:
- ARP phase (0-30%): Based on elapsed time vs estimated duration
- Deep scan phase (30-95%): Based on scanned/discovered ratio
- Grace period (95-100%): Based on grace period elapsed
- Removed per-host progress from deep_scan_host to avoid conflicts

Windows support:
- Default: SendARP API (sequential, always available)
- Optional: Npcap broadcast ARP via `use_npcap_arp` config
- Graceful fallback if Npcap unavailable

Platform behavior:
| Platform | Default Method       | Optional Method    |
|----------|---------------------|-------------------|
| Linux    | Broadcast ARP (pnet) | -                 |
| macOS    | Broadcast ARP (pnet) | -                 |
| Windows  | SendARP (iphlpapi)   | Broadcast (Npcap) |

Files:
- docs/arp-scanning.md: New documentation on ARP rate limiting
- daemon/utils/arp/broadcast.rs: Core broadcast ARP with retries
- daemon/utils/arp/mod.rs: Platform abstraction, export constants
- daemon/discovery/service/network.rs: Streaming pipeline integration
- daemon/shared/config.rs: New config options (arp_retries, arp_rate_pps)
Replace the markdown issue template with a structured YAML form that
provides clearer guidance for users reporting missing services.

Changes:
- Convert service-detection-issue.md to YAML form (.yml) with dropdowns
  and structured sections
- Split form into two clear paths: "not in library" (service request)
  vs "in library but not detected" (detection bug)
- Add auto-responder workflow that posts standard contribution guidance
  and adds "good first issue" label on new service-request issues
- Add escalation workflow that adds "help wanted" label to service
  requests older than 2 weeks

The previous markdown template caused confusion because:
- Users filled out both "Not Detected" and "False Positive" sections
- Freeform text fields led to inconsistent responses
- Instructions were mixed with response placeholders
- No structured way to capture ports or detection criteria

The new form uses dropdowns for constrained choices, clear section
routing based on the initial selection, and specific fields for the
information needed to action each type of request.
- Lazy-load PostHog via dynamic import to avoid blocking initial bundle
- Defer active-sessions query until browser idle using requestIdleCallback
- Replace billing page API calls with static JSON fixtures
  - Eliminates /api/billing/plans and /api/metadata requests
  - Page renders immediately without loading spinner
- Filter out self-hosted plans from billing page display

Supporting changes:
- Add createStaticHelpers() factory for static fixture data
- Add getPosthogDistinctId() for safe access to lazy-loaded PostHog
- Move billing-plans.json and features.json to ui/src/lib/data/
- Update fixture generation and workflow paths accordingly
- Move "generate new key" / "use existing key" radio buttons from
  CreateDaemonModal to CreateDaemonForm with allowExistingKey prop
- Update RadioGroup component to display options as cards with
  per-option helpText
- Add keySource and existingKeyInput as TanStack form fields
- Show daemonUrl field conditionally when mode is set to Push
- Add showWhen property to FieldDef for conditional field rendering
Progress reporting fixes:
- Use tokio::time::interval instead of sleep in select loop to ensure
  progress ticker fires reliably even during continuous activity
- Add 30-second heartbeat in report_scanning_progress to send updates
  even when progress percentage hasn't changed
- Implement batch-level progress tracking for smoother UX during deep
  scans (~328 batches per host instead of per-host granularity)

Resource management:
- Add ConcurrentPipelineOps struct for precise FD budget calculation
- Account for concurrent ARP channels and non-interfaced port scanning
  when calculating deep scan concurrency
- Buffer hosts when concurrency limit reached instead of dropping them

Stalled session handling:
- Send cancellation request to daemon when session is detected as stalled
- Support both push mode (HTTP request) and pull mode (cancellation flag)

UI improvements:
- Add animated progress bar with tweened transitions and shimmer effect
- Use stable keys in DataControls #each blocks to preserve component
  identity across updates
@mayanayza mayanayza merged commit fedae23 into main Jan 6, 2026
4 checks passed
mayanayza added a commit that referenced this pull request Feb 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant