Skip to content

perf(delayed): allow to fetch next job when moving job to delayed (python) (elixir)#3870

Merged
roggervalf merged 18 commits intomasterfrom
perf-move-to-delayed-fetch-next
Apr 9, 2026
Merged

perf(delayed): allow to fetch next job when moving job to delayed (python) (elixir)#3870
roggervalf merged 18 commits intomasterfrom
perf-move-to-delayed-fetch-next

Conversation

@roggervalf
Copy link
Copy Markdown
Collaborator

@roggervalf roggervalf commented Mar 11, 2026

Why

  1. Why is this change necessary? When a job is being moved to delayed, a new job can be returned, similar to moveToFailed logic and prevent a new redis roundtrip. It keeps moveToFailed logic consistent when current job is moved to delayed in manual processing, before nothing was returned

How

Enter the implementation details here.

Additional Notes (Optional)

Any extra info here.

Copilot AI review requested due to automatic review settings March 11, 2026 05:27
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the “fetch next job” fast-path to the move-to-delayed flow (primarily for delayed retries), reducing roundtrips by allowing the Lua script to optionally fetch/prepare the next job immediately after moving the current job to the delayed set.

Changes:

  • Add a fetchNext flag to MoveToDelayedOpts and plumb it through the moveToDelayed script arguments.
  • Refactor the “fetch next job” logic into a shared Lua include (includes/fetchNextJob.lua) and reuse it from both moveToFinished and moveToDelayed.
  • Update TS script wrappers and job retry path so delayed retries can optionally return next-job data (consistent with moveToFinished).

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/interfaces/minimal-job.ts Adds fetchNext?: boolean to delayed-move options.
src/commands/moveToFinished-14.lua Replaces inline next-job fetching with shared fetchNextJob include.
src/commands/moveToDelayed-12.lua Adds keys/args + optional next-job fetch via fetchNextJob after moving to delayed.
src/commands/includes/fetchNextJob.lua New shared helper implementing next-job fetching + rate-limit/paused handling.
src/classes/scripts.ts Extends moveToDelayedArgs to pass extra keys and optional packed opts; parses optional next-job return.
src/classes/job.ts Uses { fetchNext } when retrying with delay; removes unused return from Job.moveToDelayed().
Comments suppressed due to low confidence (1)

src/commands/moveToDelayed-12.lua:90

  • New behavior allows fetching the next job as part of moveToDelayed (via ARGV[8]/ARGV[9] and fetchNextJob). There are tests for retrying with backoff, but none appear to assert that moveToFailed(..., fetchNext=true) returns valid next-job data (and that worker attribution like processedBy is set) when the retry path uses moveToDelayed. Adding a focused test here would help prevent regressions in the new fast-path.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/classes/scripts.ts
Comment thread src/interfaces/minimal-job.ts
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread python/bullmq/scripts.py Outdated
Comment thread src/commands/includes/fetchNextJob.lua Outdated
Comment thread src/classes/job.ts
Comment thread elixir/lib/bullmq/worker.ex
Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 11, 2026

@roggervalf I've opened a new pull request, #3871, to work on those changes. Once the pull request is ready, I'll request review from you.

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 11, 2026

@roggervalf I've opened a new pull request, #3874, to work on those changes. Once the pull request is ready, I'll request review from you.

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 11, 2026

@roggervalf I've opened a new pull request, #3875, to work on those changes. Once the pull request is ready, I'll request review from you.

@roggervalf roggervalf force-pushed the perf-move-to-delayed-fetch-next branch from 538d406 to e6d8546 Compare March 12, 2026 01:33
@roggervalf roggervalf requested a review from manast March 12, 2026 01:47
@roggervalf roggervalf changed the title perf(delayed): allow to fetch next job when moving job to delayed perf(delayed): allow to fetch next job when moving job to delayed (python) (elixir) Mar 12, 2026
@manast manast requested a review from Copilot March 23, 2026 14:37
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

src/commands/moveToDelayed-12.lua:88

  • When fetchNextJob returns a next job, this script returns early without calling addDelayMarkerIfNeeded. That can leave the delayed marker unset for the job you just moved to the delayed ZSET, causing sleeping workers to miss the wakeup and potentially delay the retry until the next drainDelay poll. Ensure the delay marker is updated even when returning the fetched next job (e.g., call addDelayMarkerIfNeeded(markerKey, delayedKey) before the early return).
    src/commands/moveToDelayed-12.lua:88
  • The type(result[1]) == "table" guard means you drop non-job fetchNextJob results (rate-limited, paused/maxed, or {0,0,0,nextTimestamp} when only delayed jobs exist). That prevents callers from updating limitUntil/delayUntil based on the combined script call and undermines the intended roundtrip savings. Consider returning result whenever it’s non-nil and let the client interpret the tuple (as moveToFinished does).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/job.test.ts Outdated
Comment thread src/classes/scripts.ts
Comment thread elixir/lib/bullmq/scripts.ex
Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 24, 2026

@roggervalf I've opened a new pull request, #3898, to work on those changes. Once the pull request is ready, I'll request review from you.

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 24, 2026

@roggervalf I've opened a new pull request, #3899, to work on those changes. Once the pull request is ready, I'll request review from you.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@roggervalf roggervalf force-pushed the perf-move-to-delayed-fetch-next branch 2 times, most recently from b28a689 to 7a6b0d9 Compare March 29, 2026 18:50
@roggervalf roggervalf force-pushed the perf-move-to-delayed-fetch-next branch 5 times, most recently from bb0d1cf to 12da7d2 Compare April 8, 2026 06:19
@roggervalf roggervalf force-pushed the perf-move-to-delayed-fetch-next branch from 12da7d2 to d8a753c Compare April 9, 2026 06:13
@roggervalf roggervalf merged commit 0cb0b8c into master Apr 9, 2026
21 checks passed
@roggervalf roggervalf deleted the perf-move-to-delayed-fetch-next branch April 9, 2026 13:57
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 9, 2026

🎉 This PR is included in version 5.73.3 🎉

The release is available on:

Your semantic-release bot 📦🚀

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 9, 2026

🎉 This PR is included in version 2.20.2 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants