fix: floor provisioner job queue wait metric#22184
Merged
Conversation
After a PostgreSQL round-trip, job timestamps lose their monotonic clock component, making the subtraction susceptible to wall-clock adjustments producing a small negative delta. Floor at 1ms since a zero or negative queue wait is meaningless. Fixes TestProvisionerJobQueueWaitMetric flakes where small negative values (~ -2ms) are observed.
zedkipp
force-pushed
the
zedkipp/job-queue-negative
branch
from
4b2cfd9 to
852eabe
Compare
zedkipp
commented
Feb 20, 2026
| // round-trip, so the subtraction is based purely on wall-clock time. Floor at | ||
| // 1ms as a defensive measure against clock adjustments producing a negative | ||
| // delta while acknowledging there's a non-zero queue time. | ||
| queueWaitSeconds := max(job.StartedAt.Time.Sub(job.CreatedAt).Seconds(), 0.001) |
Contributor
Author
There was a problem hiding this comment.
Discussed with @cstyan and decided to make this max against 1ms, since in reality the queue times can't be exactly zero. When looking at the histogram for this metric most will be much larger than 1ms, so any of these instances will appear as outliers.
cstyan
approved these changes
Feb 20, 2026
zedkipp
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After a PostgreSQL round-trip, job timestamps lose their monotonic clock component, making the subtraction susceptible to wall-clock adjustments producing a small negative delta. Floor at zero since negative queue wait is meaningless. Fixes TestProvisionerJobQueueWaitMetric flakes where small negative values (~2ms) are observed.
coder/internal#1354