Skip to content

Fix: Prevent job loss on container shutdown for delayed executions#10922

Merged
stnguyen90 merged 2 commits into1.8.xfrom
copilot/fix-delayed-executions-trigger
Dec 9, 2025
Merged

Fix: Prevent job loss on container shutdown for delayed executions#10922
stnguyen90 merged 2 commits into1.8.xfrom
copilot/fix-delayed-executions-trigger

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Dec 8, 2025

What does this PR do?

Race condition: schedule documents were deleted immediately after spawning coroutines, before jobs were queued. Container shutdown between coroutine spawn and trigger execution caused permanent job loss.

Changes:

  • Move schedule deletion inside coroutine, after trigger() call
  • Move unset() operation into coroutine to maintain consistency
  • Add $dbForPlatform to coroutine closure

Before:

\go(function () use (...) {
    Co::sleep($delay);
    $queueForFunctions->trigger();
});

$dbForPlatform->deleteDocument('schedules', $schedule['$id']);  // ← Deleted before trigger completes

After:

\go(function () use (..., $dbForPlatform) {
    Co::sleep($delay);
    $queueForFunctions->trigger();
    $dbForPlatform->deleteDocument('schedules', $schedule['$id']);  // ← Deleted after trigger completes
    unset($this->schedules[$schedule['$sequence']]);
});

Pattern now matches ScheduleMessages.php implementation for one-time scheduled tasks.

Test Plan

Existing e2e test testCreateScheduledAtExecution in FunctionsScheduleTest.php validates scheduled execution behavior. The fix ensures jobs survive container restarts between enqueue scheduling and execution.

Related PRs and Issues

  • Addresses delayed execution reliability during container lifecycle events

Checklist

  • Have you read the Contributing Guidelines on issues?
  • If the PR includes a change to an API's metadata (desc, label, params, etc.), does it also include updated API specs and example docs?
Original prompt

We have an issue where delayed executions don't trigger. The issue is probably here:

\go(function () use ($queueForFunctions, $schedule, $delay, $data) {
    Co::sleep($delay);

    $queueForFunctions->setType('schedule')
        // Set functionId instead of function as we don't have $dbForProject
        // TODO: Refactor to use function instead of functionId
        ->setFunctionId($schedule['resource']['functionId'])
        ->setExecution($schedule['resource'])
        ->setMethod($data['method'] ?? 'POST')
        ->setPath($data['path'] ?? '/')
        ->setHeaders($data['headers'] ?? [])
        ->setBody($data['body'] ?? '')
        ->setProject($schedule['project'])
        ->setUserId($data['userId'] ?? '')
        ->trigger();
});

$dbForConsole->deleteDocument(
    'schedules',
    $schedule['$id'],
);

Once the coroutine starts, the schedule is deleted. If the container shuts down, the job is never queued and the schedule is deleted so it'll never trigger.

Please create a PR to fix this issue.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Dec 8, 2025

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.


Comment @coderabbitai help to get the list of available commands and usage tips.

Move the schedule document deletion and unset operation inside the coroutine, after the function is triggered. This ensures the schedule is only deleted after the job is successfully queued, preventing job loss if the container shuts down between coroutine start and completion.

This follows the same pattern used in ScheduleMessages.php.

Co-authored-by: stnguyen90 <[email protected]>
Copilot AI changed the title [WIP] Fix issue with delayed executions not triggering Fix: Prevent job loss on container shutdown for delayed executions Dec 8, 2025
Copilot AI requested a review from stnguyen90 December 8, 2025 19:54
@stnguyen90 stnguyen90 marked this pull request as ready for review December 8, 2025 20:15
@stnguyen90
Copy link
Copy Markdown
Contributor

I tested this to confirm that delayed executions still worked. Trying to reproduce the issue was difficult because I couldn't stop the container in the 4 second interval that it picked up the job and deleted the schedule, but was waiting to trigger the execution.

@stnguyen90
Copy link
Copy Markdown
Contributor

@github-actions
Copy link
Copy Markdown

github-actions bot commented Dec 8, 2025

Security Scan Results for PR

Docker Image Scan Results

Package Version Vulnerability Severity
imagemagick 7.1.2.3-r0 CVE-2025-62171 HIGH
imagemagick-c++ 7.1.2.3-r0 CVE-2025-62171 HIGH
imagemagick-dev 7.1.2.3-r0 CVE-2025-62171 HIGH
imagemagick-heic 7.1.2.3-r0 CVE-2025-62171 HIGH
imagemagick-jpeg 7.1.2.3-r0 CVE-2025-62171 HIGH
imagemagick-jxl 7.1.2.3-r0 CVE-2025-62171 HIGH
imagemagick-libs 7.1.2.3-r0 CVE-2025-62171 HIGH
imagemagick-openexr 7.1.2.3-r0 CVE-2025-62171 HIGH
imagemagick-webp 7.1.2.3-r0 CVE-2025-62171 HIGH
libecpg 17.6-r0 CVE-2025-12818 HIGH
libecpg-dev 17.6-r0 CVE-2025-12818 HIGH
libpng 1.6.47-r0 CVE-2025-64720 HIGH
libpng 1.6.47-r0 CVE-2025-65018 HIGH
libpng-dev 1.6.47-r0 CVE-2025-64720 HIGH
libpng-dev 1.6.47-r0 CVE-2025-65018 HIGH
libpq 17.6-r0 CVE-2025-12818 HIGH
libpq-dev 17.6-r0 CVE-2025-12818 HIGH
libxml2 2.13.8-r0 CVE-2025-49794 CRITICAL
libxml2 2.13.8-r0 CVE-2025-49796 CRITICAL
libxml2 2.13.8-r0 CVE-2025-49795 HIGH
libxml2 2.13.8-r0 CVE-2025-6021 HIGH
postgresql17-dev 17.6-r0 CVE-2025-12818 HIGH
github.com/containerd/containerd/v2 v2.0.2 CVE-2024-25621 HIGH
golang.org/x/crypto v0.31.0 CVE-2025-22869 HIGH
golang.org/x/oauth2 v0.24.0 CVE-2025-22868 HIGH
stdlib 1.22.10 CVE-2025-47907 HIGH
stdlib 1.22.10 CVE-2025-58183 HIGH
stdlib 1.22.10 CVE-2025-58186 HIGH
stdlib 1.22.10 CVE-2025-58187 HIGH
stdlib 1.22.10 CVE-2025-61729 HIGH

Source Code Scan Results

🎉 No vulnerabilities found!

Base automatically changed from copilot/fix-schedule-executions-bug to 1.8.x December 8, 2025 20:22
@github-actions
Copy link
Copy Markdown

github-actions bot commented Dec 8, 2025

✨ Benchmark results

  • Requests per second: 1,127
  • Requests with 200 status code: 202,853
  • P99 latency: 0.172316908

⚡ Benchmark Comparison

Metric This PR Latest version
RPS 1,127 1,126
200 202,853 202,644
P99 0.172316908 0.183275904

@stnguyen90 stnguyen90 requested a review from loks0n December 9, 2025 02:13
@stnguyen90 stnguyen90 merged commit 6c1cbfd into 1.8.x Dec 9, 2025
44 checks passed
@stnguyen90 stnguyen90 deleted the copilot/fix-delayed-executions-trigger branch December 9, 2025 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants