Skip to content

fix(log): log cleanup sql query#4087

Merged
waleedlatif1 merged 2 commits intostagingfrom
fix/log-cleanup
Apr 10, 2026
Merged

fix(log): log cleanup sql query#4087
waleedlatif1 merged 2 commits intostagingfrom
fix/log-cleanup

Conversation

@TheodoreSpeaks
Copy link
Copy Markdown
Collaborator

Summary

Our log cleanup was failing due to too many params passed:

[ERROR] [LogsCleanupAPI] Error in log cleanup process: {"error":{"query":"select \"id\" from
  \"workspace\" where \"workspace\".\"billed_account_user_id\" in ($1, $2, $3, $4, $5, $6, $7...

This is because our old logic fetched every free user id into memory and then passed it to the second sql query. Changed this to use a subquery to fix free user log cleanup

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Other: ___________

Testing

How has this been tested? What should reviewers focus on?

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Screenshots/Videos

@vercel
Copy link
Copy Markdown

vercel bot commented Apr 9, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Apr 10, 2026 0:50am

Request Review

@TheodoreSpeaks TheodoreSpeaks changed the base branch from main to staging April 9, 2026 22:19
@TheodoreSpeaks
Copy link
Copy Markdown
Collaborator Author

@BugBot review

@cursor
Copy link
Copy Markdown

cursor bot commented Apr 9, 2026

PR Summary

Medium Risk
Changes the selection criteria for which logs are eligible for deletion/archiving by switching to a workspace subquery and using startedAt for retention, which could alter cleanup coverage if the new join/field differs from prior behavior.

Overview
Fixes the log-cleanup cron query to avoid generating oversized IN (...) parameter lists by selecting free-plan workspaces via a workspace/subscription subquery instead of fetching all free user IDs into memory.

Also adjusts the retention predicate to compare against workflowExecutionLogs.startedAt (and simplifies early-return logic/logging), keeping the rest of the archive/delete flow unchanged.

Reviewed by Cursor Bugbot for commit cafe8fe. Configure here.

Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 5ef15c8. Configure here.

@TheodoreSpeaks TheodoreSpeaks marked this pull request as ready for review April 10, 2026 00:11
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Apr 10, 2026

Greptile Summary

This PR replaces a two-step memory-bound query (fetch all free-user IDs → pass as IN params) with a single Drizzle subquery, correctly fixing the "too many params" PostgreSQL error that was breaking the log cleanup cron job. The core fix is sound and the approach is idiomatic for Drizzle ORM.

One incidental change deserves attention: the retention filter was silently switched from createdAt to startedAt in the WHERE clause, which has slightly different semantics and was not called out in the PR description.

Confidence Score: 4/5

Safe to merge after confirming the startedAt vs createdAt filter change is intentional.

The core subquery fix is correct and directly addresses the production breakage. The only unresolved concern is the unannounced semantic change from createdAt to startedAt in the retention filter — practically minor (milliseconds difference) but deserves explicit acknowledgment from the author before merging.

apps/sim/app/api/logs/cleanup/route.ts — confirm the startedAt field change on line 90 is deliberate

Important Files Changed

Filename Overview
apps/sim/app/api/logs/cleanup/route.ts Correctly rewrites the free-workspace lookup as a SQL subquery to fix the "too many params" error; also silently changes the retention filter from createdAt to startedAt

Sequence Diagram

sequenceDiagram
    participant Cron as Cron Job
    participant Route as /api/logs/cleanup
    participant DB as Database

    Note over Route,DB: Before (broken — too many params)
    Cron->>Route: GET request
    Route->>DB: SELECT id FROM user (all free users)
    DB-->>Route: [userId1, userId2, ..., userIdN]
    Route->>DB: SELECT id FROM workspace WHERE billedAccountUserId IN ($1,$2,...,$N)
    Note right of DB: Fails when N is large
    DB-->>Route: Error: too many params

    Note over Route,DB: After (fixed — subquery)
    Cron->>Route: GET request
    Route->>DB: SELECT logs WHERE workspaceId IN (SELECT workspace.id FROM workspace LEFT JOIN subscription WHERE subscription.id IS NULL) AND startedAt < retentionDate LIMIT 2000
    DB-->>Route: [log batch]
    Route->>DB: Archive + delete each log
    Route-->>Cron: results, complete, batchLimitReached
Loading

Reviews (3): Last reviewed commit: "perf(log): use startedAt index for clean..." | Re-trigger Greptile

Copy link
Copy Markdown
Collaborator

@waleedlatif1 waleedlatif1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, let's see if there are any other efficiency issues with this query, in terms of it doing a full scan

@gitguardian
Copy link
Copy Markdown

gitguardian bot commented Apr 10, 2026

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Theodore Li and others added 2 commits April 9, 2026 17:48
Switch cleanup WHERE clause from createdAt to startedAt to leverage
the existing composite index (workspaceId, startedAt), converting a
full table scan to an index range scan. Also remove explanatory comment.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
@waleedlatif1
Copy link
Copy Markdown
Collaborator

@greptile

@waleedlatif1
Copy link
Copy Markdown
Collaborator

@cursor review

@waleedlatif1
Copy link
Copy Markdown
Collaborator

@greptile

@waleedlatif1
Copy link
Copy Markdown
Collaborator

@cursor review

Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit cafe8fe. Configure here.

@waleedlatif1 waleedlatif1 merged commit 4074109 into staging Apr 10, 2026
12 checks passed
@waleedlatif1 waleedlatif1 deleted the fix/log-cleanup branch April 10, 2026 01:04
waleedlatif1 added a commit that referenced this pull request Apr 10, 2026
* fix(log): log cleanup sql query

* perf(log): use startedAt index for cleanup query filter

Switch cleanup WHERE clause from createdAt to startedAt to leverage
the existing composite index (workspaceId, startedAt), converting a
full table scan to an index range scan. Also remove explanatory comment.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Theodore Li <[email protected]>
Co-authored-by: Waleed Latif <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants