feat: add real-time progress tracking for multipart weight uploads#2688
feat: add real-time progress tracking for multipart weight uploads#2688markphelps merged 3 commits intomainfrom
Conversation
Signed-off-by: Mark Phelps <[email protected]>
mfainberg-cf
left a comment
There was a problem hiding this comment.
We may want to eventually setup parallel uploads, small MPUs at 25M could benefit from it.
| // commitUpload finalizes the upload by sending a PUT request with the digest. | ||
| func (c *RegistryClient) commitUpload(ctx context.Context, client *http.Client, location string, digest v1.Hash) error { | ||
| u, err := url.Parse(location) | ||
| if err != nil { | ||
| return fmt.Errorf("parsing location URL: %w", err) | ||
| } | ||
|
|
||
| // Add digest query parameter | ||
| q := u.Query() | ||
| q.Set("digest", digest.String()) | ||
| u.RawQuery = q.Encode() | ||
|
|
||
| req, err := http.NewRequestWithContext(ctx, http.MethodPut, u.String(), nil) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| req.Header.Set("Content-Type", "application/octet-stream") | ||
|
|
||
| resp, err := client.Do(req) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| defer resp.Body.Close() | ||
|
|
||
| return transport.CheckError(resp, http.StatusCreated) | ||
| } |
There was a problem hiding this comment.
If we're terminally failing here maybe we should cancel the MPU? Are we planning to have a resume MPU function in the case the CLI crashes or is SIGINT'd? These are questions not any changes needed.
There was a problem hiding this comment.
good questions. yeah we should do cleanup on early exit. will do in a followup. first let me check if the registry has gc too
its uploading 4 in parallel currently, well 4 layers in parallel. i might tune that to use gomaxprocs, although its not cpu bound but network bound so maybe a hard cap on parallel layers is sufficient |
… logic Address PR feedback: - Set multipart threshold to 50MB (was 25MB) to avoid MPU overhead for smaller files - Keep chunk size at 25MB for actual multipart uploads - Extract multipart fallback logic into tryMultipartWithFallback() for cleaner code
Signed-off-by: Mark Phelps <[email protected]>
Summary
Changes
uploadChunkto wrap the chunk reader with progress tracking, reporting progress as bytes are written to the networkuploadBlobChunksto return the final location URL for proper commit handlingtotalSizeto prevent overshoot