perf(delayed): allow to fetch next job when moving job to delayed (python) (elixir)#3870
perf(delayed): allow to fetch next job when moving job to delayed (python) (elixir)#3870roggervalf merged 18 commits intomasterfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR extends the “fetch next job” fast-path to the move-to-delayed flow (primarily for delayed retries), reducing roundtrips by allowing the Lua script to optionally fetch/prepare the next job immediately after moving the current job to the delayed set.
Changes:
- Add a
fetchNextflag toMoveToDelayedOptsand plumb it through themoveToDelayedscript arguments. - Refactor the “fetch next job” logic into a shared Lua include (
includes/fetchNextJob.lua) and reuse it from bothmoveToFinishedandmoveToDelayed. - Update TS script wrappers and job retry path so delayed retries can optionally return next-job data (consistent with
moveToFinished).
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/interfaces/minimal-job.ts | Adds fetchNext?: boolean to delayed-move options. |
| src/commands/moveToFinished-14.lua | Replaces inline next-job fetching with shared fetchNextJob include. |
| src/commands/moveToDelayed-12.lua | Adds keys/args + optional next-job fetch via fetchNextJob after moving to delayed. |
| src/commands/includes/fetchNextJob.lua | New shared helper implementing next-job fetching + rate-limit/paused handling. |
| src/classes/scripts.ts | Extends moveToDelayedArgs to pass extra keys and optional packed opts; parses optional next-job return. |
| src/classes/job.ts | Uses { fetchNext } when retrying with delay; removes unused return from Job.moveToDelayed(). |
Comments suppressed due to low confidence (1)
src/commands/moveToDelayed-12.lua:90
- New behavior allows fetching the next job as part of
moveToDelayed(viaARGV[8]/ARGV[9]andfetchNextJob). There are tests for retrying with backoff, but none appear to assert thatmoveToFailed(..., fetchNext=true)returns valid next-job data (and that worker attribution likeprocessedByis set) when the retry path usesmoveToDelayed. Adding a focused test here would help prevent regressions in the new fast-path.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@roggervalf I've opened a new pull request, #3871, to work on those changes. Once the pull request is ready, I'll request review from you. |
|
@roggervalf I've opened a new pull request, #3874, to work on those changes. Once the pull request is ready, I'll request review from you. |
|
@roggervalf I've opened a new pull request, #3875, to work on those changes. Once the pull request is ready, I'll request review from you. |
538d406 to
e6d8546
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (2)
src/commands/moveToDelayed-12.lua:88
- When
fetchNextJobreturns a next job, this script returns early without callingaddDelayMarkerIfNeeded. That can leave the delayed marker unset for the job you just moved to the delayed ZSET, causing sleeping workers to miss the wakeup and potentially delay the retry until the next drainDelay poll. Ensure the delay marker is updated even when returning the fetched next job (e.g., calladdDelayMarkerIfNeeded(markerKey, delayedKey)before the early return).
src/commands/moveToDelayed-12.lua:88 - The
type(result[1]) == "table"guard means you drop non-job fetchNextJob results (rate-limited, paused/maxed, or{0,0,0,nextTimestamp}when only delayed jobs exist). That prevents callers from updatinglimitUntil/delayUntilbased on the combined script call and undermines the intended roundtrip savings. Consider returningresultwhenever it’s non-nil and let the client interpret the tuple (asmoveToFinisheddoes).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@roggervalf I've opened a new pull request, #3898, to work on those changes. Once the pull request is ready, I'll request review from you. |
|
@roggervalf I've opened a new pull request, #3899, to work on those changes. Once the pull request is ready, I'll request review from you. |
1d8c66c to
a0583ee
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
b28a689 to
7a6b0d9
Compare
bb0d1cf to
12da7d2
Compare
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
…ry and fetchNext=true (#3871)
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
12da7d2 to
d8a753c
Compare
|
🎉 This PR is included in version 5.73.3 🎉 The release is available on: Your semantic-release bot 📦🚀 |
|
🎉 This PR is included in version 2.20.2 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
Why
How
Enter the implementation details here.
Additional Notes (Optional)
Any extra info here.