feat: spill large inputs to disk to prevent IPC frame overflow#2804
Merged
michaeldwan merged 5 commits intomainfrom Mar 6, 2026
Merged
feat: spill large inputs to disk to prevent IPC frame overflow#2804michaeldwan merged 5 commits intomainfrom
michaeldwan merged 5 commits intomainfrom
Conversation
When an input payload exceeds 6 MiB (MAX_INLINE_IPC_SIZE), the parent
serializes it to a file under /tmp/coglet/predictions/{id}/inputs/ and
sends only the file path over the slot socket. The worker reads the file
back, parses it, and deletes it before running the prediction.
This mirrors the existing output-spill mechanism and prevents the 8 MiB
LengthDelimitedCodec frame limit from being exceeded by large inputs.
Key changes:
- protocol.rs: SlotRequest::Predict now has optional input/input_file
fields, prediction_id() accessor, and rehydrate_input() method
- service.rs: build_slot_request() checks size and spills to disk;
prediction dir layout changed to /tmp/coglet/predictions/{id}/
- worker.rs: uses rehydrate_input() with proper failure handling that
sends SlotResponse::Failed instead of silently dropping predictions
- Unified MAX_INLINE_IPC_SIZE constant (was separate input/output consts)
- Integration test with 7 MiB payload via new harness @file curl syntax
- Axum's default 2 MiB body limit rejected large input payloads with HTTP 413 before they could reach the spill-to-disk logic. Set DefaultBodyLimit to 100 MiB on the router. - Fix pre-existing rustfmt comment alignment in truncate_worker_log tests that CI's rustfmt version caught.
markphelps
previously approved these changes
Mar 5, 2026
good find Co-authored-by: Mark Phelps <[email protected]> Signed-off-by: Michael Dwan <[email protected]>
markphelps
reviewed
Mar 5, 2026
…ariable' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> Signed-off-by: Mark Phelps <[email protected]>
markphelps
approved these changes
Mar 5, 2026
tempusfrangit
approved these changes
Mar 6, 2026
Contributor
tempusfrangit
left a comment
There was a problem hiding this comment.
I need to turn off notifications for this repo :P. mark approved, approving to unblock.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/tmp/coglet/predictions/{id}/inputs/and sends only the file path over the IPC slot socket. The worker reads the file back, parses it, and deletes it before running the prediction.LengthDelimitedCodecframe limit from being exceeded by large inputs.MAX_INLINE_IPC_SIZEconstant (previously separate input/output constants that could drift).Changes
Rust (
crates/coglet/)protocol.rs:SlotRequest::Predictnow has optionalinput/input_filefields withskip_serializing_if. Addedprediction_id()accessor andrehydrate_input()method that reads from inline value or spill file. Spill file is cleaned up before JSON parsing (so corrupt files don't leak). 4 unit tests + snapshot.service.rs: Newbuild_slot_request()serializes input, checks against threshold, spills to disk if needed. Prediction dir layout changed from/tmp/coglet/outputs/{id}to/tmp/coglet/predictions/{id}/{inputs,outputs}. 3 unit tests.worker.rs: Usesrehydrate_input()with proper failure handling — sendsSlotResponse::Failedback to parent instead of silently dropping the prediction. Removed duplicateMAX_INLINE_OUTPUT_SIZEin favor of sharedMAX_INLINE_IPC_SIZE.codec.rs: Updated test fixture for new field layout.Cargo.toml: Addedtempfileto dev-dependencies.Integration tests (
integration-tests/)harness.go:cmdCurlnow supports@filesyntax to POST large bodies from files.coglet_large_input.txtar: New test — generates 7 MiB JSON payload, POSTs via@file, verifies prediction succeeds via webhook.Test plan
mise run test:rust— 165/165 passmise run fmt:rust:fix— cleanmise run lint:rust— clean (clippy)coglet_large_inputintegration test — PASScoglet_large_outputintegration test — PASS (regression guard for dir layout change)string_predictorintegration test — PASS (inline input path still works)