next-core: deduplicate output assets and detect content conflicts on emit#92292
Merged
next-core: deduplicate output assets and detect content conflicts on emit#92292
Conversation
Merging this PR will not alter performance
Comparing Footnotes
|
Collaborator
Stats from current PR✅ No significant changes detected📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles
Server Middleware
Build DetailsBuild Manifests
🔄 Shared (bundler-independent)Runtimes
📎 Tarball URL |
Collaborator
Tests Passed |
lukesandberg
reviewed
Apr 3, 2026
lukesandberg
approved these changes
Apr 3, 2026
Deduplicate assets by path before emitting, and bail with an error when duplicate assets have different content. Cherry-picked emit.rs changes from 576587e. Co-Authored-By: Claude <[email protected]>
Add a dedicated IssueStage::Emit variant instead of using
IssueStage::Other("emit"). Rename check_emit_conflict back to
assets_diff which now only returns the diff description, and emit the
EmitConflictIssue from check_duplicates instead. This avoids passing
asset_path into the turbo_tasks function, reducing task key cardinality.
Co-Authored-By: Claude <[email protected]>
FileSystemPath::extension() now returns Option<&str> after upstream dedup of extension methods. Use unwrap_or_default() to handle the None case. Co-Authored-By: Claude <[email protected]>
cbbc4b2 to
f378f4c
Compare
findmaster969
approved these changes
Apr 5, 2026
eps1lon
pushed a commit
that referenced
this pull request
Apr 7, 2026
…emit (#92292) ### What? Adds deduplication and conflict detection to the asset emission stage in `crates/next-core/src/emit.rs`, and a new `IssueStage::Emit` variant in `turbopack-core`. Before emitting, assets are grouped by their output path. If multiple assets map to the same path: - If their content is identical, one is silently chosen (deduplication). - If their content differs, both versions are written to `<node_root>/<content_hash>.<ext>` and an `EmitConflictIssue` is raised for each conflict. All assets are still emitted — conflicts do not abort the build. ### Why? Previously, duplicate output assets for the same path were emitted unconditionally — whichever write happened last silently won. This masked build graph bugs where two different modules produced conflicting output files. Reporting conflicts as issues (rather than silently overwriting) makes them visible and easy to diagnose without breaking the build. ### How? - Collect all assets with their resolved paths via `try_flat_join`. - Bucket them into two `FxIndexMap<FileSystemPath, Vec<ResolvedVc<Box<dyn OutputAsset>>>>` — one for node-root assets and one for client assets. - For each bucket entry, call `check_duplicates`: compare every asset against the first using `assets_diff`. If content differs, emit an `EmitConflictIssue` as a turbo-tasks collectible — but still return the first asset so emission continues. - `assets_diff` is a `#[turbo_tasks::function]` that takes only `(asset1, asset2, extension, node_root)` — the `asset_path` stays out of the task key to avoid unnecessary task cardinality. When file content differs, it hashes each version with xxh3, writes them to `<node_root>/<hash>.<ext>`, and returns the paths in the detail message so the user can diff them. - `EmitConflictIssue` implements the `Issue` trait with `IssueStage::Emit` (new variant added to `turbopack-core`), `IssueSeverity::Error`, a descriptive title, and a detail message explaining the type of conflict. - Node-root and client assets are emitted in parallel via `futures::join!` (not `try_join!`) to ensure deterministic error reporting — both branches always run to completion so errors are reported in a consistent order. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>
eps1lon
pushed a commit
that referenced
this pull request
Apr 7, 2026
…emit (#92292) ### What? Adds deduplication and conflict detection to the asset emission stage in `crates/next-core/src/emit.rs`, and a new `IssueStage::Emit` variant in `turbopack-core`. Before emitting, assets are grouped by their output path. If multiple assets map to the same path: - If their content is identical, one is silently chosen (deduplication). - If their content differs, both versions are written to `<node_root>/<content_hash>.<ext>` and an `EmitConflictIssue` is raised for each conflict. All assets are still emitted — conflicts do not abort the build. ### Why? Previously, duplicate output assets for the same path were emitted unconditionally — whichever write happened last silently won. This masked build graph bugs where two different modules produced conflicting output files. Reporting conflicts as issues (rather than silently overwriting) makes them visible and easy to diagnose without breaking the build. ### How? - Collect all assets with their resolved paths via `try_flat_join`. - Bucket them into two `FxIndexMap<FileSystemPath, Vec<ResolvedVc<Box<dyn OutputAsset>>>>` — one for node-root assets and one for client assets. - For each bucket entry, call `check_duplicates`: compare every asset against the first using `assets_diff`. If content differs, emit an `EmitConflictIssue` as a turbo-tasks collectible — but still return the first asset so emission continues. - `assets_diff` is a `#[turbo_tasks::function]` that takes only `(asset1, asset2, extension, node_root)` — the `asset_path` stays out of the task key to avoid unnecessary task cardinality. When file content differs, it hashes each version with xxh3, writes them to `<node_root>/<hash>.<ext>`, and returns the paths in the detail message so the user can diff them. - `EmitConflictIssue` implements the `Issue` trait with `IssueStage::Emit` (new variant added to `turbopack-core`), `IssueSeverity::Error`, a descriptive title, and a detail message explaining the type of conflict. - Node-root and client assets are emitted in parallel via `futures::join!` (not `try_join!`) to ensure deterministic error reporting — both branches always run to completion so errors are reported in a consistent order. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>
lukesandberg
pushed a commit
that referenced
this pull request
Apr 7, 2026
…emit (#92292) Adds deduplication and conflict detection to the asset emission stage in `crates/next-core/src/emit.rs`, and a new `IssueStage::Emit` variant in `turbopack-core`. Before emitting, assets are grouped by their output path. If multiple assets map to the same path: - If their content is identical, one is silently chosen (deduplication). - If their content differs, both versions are written to `<node_root>/<content_hash>.<ext>` and an `EmitConflictIssue` is raised for each conflict. All assets are still emitted — conflicts do not abort the build. Previously, duplicate output assets for the same path were emitted unconditionally — whichever write happened last silently won. This masked build graph bugs where two different modules produced conflicting output files. Reporting conflicts as issues (rather than silently overwriting) makes them visible and easy to diagnose without breaking the build. - Collect all assets with their resolved paths via `try_flat_join`. - Bucket them into two `FxIndexMap<FileSystemPath, Vec<ResolvedVc<Box<dyn OutputAsset>>>>` — one for node-root assets and one for client assets. - For each bucket entry, call `check_duplicates`: compare every asset against the first using `assets_diff`. If content differs, emit an `EmitConflictIssue` as a turbo-tasks collectible — but still return the first asset so emission continues. - `assets_diff` is a `#[turbo_tasks::function]` that takes only `(asset1, asset2, extension, node_root)` — the `asset_path` stays out of the task key to avoid unnecessary task cardinality. When file content differs, it hashes each version with xxh3, writes them to `<node_root>/<hash>.<ext>`, and returns the paths in the detail message so the user can diff them. - `EmitConflictIssue` implements the `Issue` trait with `IssueStage::Emit` (new variant added to `turbopack-core`), `IssueSeverity::Error`, a descriptive title, and a detail message explaining the type of conflict. - Node-root and client assets are emitted in parallel via `futures::join!` (not `try_join!`) to ensure deterministic error reporting — both branches always run to completion so errors are reported in a consistent order. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Adds deduplication and conflict detection to the asset emission stage in
crates/next-core/src/emit.rs, and a newIssueStage::Emitvariant inturbopack-core.Before emitting, assets are grouped by their output path. If multiple assets map to the same path:
<node_root>/<content_hash>.<ext>and anEmitConflictIssueis raised for each conflict. All assets are still emitted — conflicts do not abort the build.Why?
Previously, duplicate output assets for the same path were emitted unconditionally — whichever write happened last silently won. This masked build graph bugs where two different modules produced conflicting output files. Reporting conflicts as issues (rather than silently overwriting) makes them visible and easy to diagnose without breaking the build.
How?
try_flat_join.FxIndexMap<FileSystemPath, Vec<ResolvedVc<Box<dyn OutputAsset>>>>— one for node-root assets and one for client assets.check_duplicates: compare every asset against the first usingassets_diff. If content differs, emit anEmitConflictIssueas a turbo-tasks collectible — but still return the first asset so emission continues.assets_diffis a#[turbo_tasks::function]that takes only(asset1, asset2, extension, node_root)— theasset_pathstays out of the task key to avoid unnecessary task cardinality. When file content differs, it hashes each version with xxh3, writes them to<node_root>/<hash>.<ext>, and returns the paths in the detail message so the user can diff them.EmitConflictIssueimplements theIssuetrait withIssueStage::Emit(new variant added toturbopack-core),IssueSeverity::Error, a descriptive title, and a detail message explaining the type of conflict.futures::join!(nottry_join!) to ensure deterministic error reporting — both branches always run to completion so errors are reported in a consistent order.