…handling (#92254)

### What?

Bug fixes and a refactoring in `turbo-tasks-backend` targeting stability
issues that surface when filesystem caching is enabled:

1. **Preserve `cell_type_max_index` on task error** — when a task fails
partway through execution, `cell_counters` only reflects the
partially-executed state. Previously, `cell_type_max_index` was updated
from these incomplete counters, which removed entries for cell types not
yet encountered. This caused `"Cell no longer exists"` hard errors for
tasks that still held dependencies on those cells. The fix skips the
`cell_type_max_index` update on error, keeping it consistent with the
preserved cell data (which already wasn't cleared on error).

This bug manifested specifically with `serialization = "hash"` cell
types (e.g. `FileContent`), where cell data is transient and readers
fall back to `cell_type_max_index` to decide whether to schedule
recomputation.

2. **Fix shutdown hang and cache poisoning for cancelled tasks** — three
related fixes for tasks cancelled during shutdown:
- `task_execution_canceled` now drains and notifies all
`InProgressCellState` events, preventing `stop_and_wait` from hanging on
foreground jobs waiting on cells that will never be filled.
- `try_read_task_cell` bails early (before calling `listen_to_cell`)
when a task is in `Canceled` state, avoiding pointless listener
registrations that would never resolve.
- Cancelled tasks are marked as session-dependent dirty, preventing
cache poisoning where `"was canceled"` errors get persisted as task
output and break subsequent builds. The session-dependent dirty flag
causes the task to re-execute in the next session, invalidating stale
dependents.

3. **Extract `update_dirty_state` helper on `TaskGuard`** — the "read
old dirty state → apply new state → propagate via
`ComputeDirtyAndCleanUpdate`" pattern was duplicated between
`task_execution_canceled` and `task_execution_completed_finish`. The new
`update_dirty_state` default method on `TaskGuard` handles both
transitions (to `SessionDependent` or to `None`) and returns the
aggregation job + `ComputeDirtyAndCleanUpdateResult` for callers that
need post-processing (e.g. firing the `all_clean_event`).

### Why?

These bugs caused observable failures when using Turbopack with
filesystem caching (`--cache` / persistent cache):

- `"Cell no longer exists"` panics/errors on incremental rebuilds after
a task error.
- Hangs on `stop_and_wait` during dev server shutdown.
- Stale `"was canceled"` errors persisted in the cache breaking
subsequent builds until the cache is cleared.

### How?

Changes are in `turbopack/crates/turbo-tasks-backend/src/backend/`:

**`mod.rs`:**
- Guard the `cell_type_max_index` update block inside `if
result.is_ok()` to skip it on error, with a cross-reference comment to
`task_execution_completed_cleanup` (which similarly skips cell data
removal on error — the two must stay in sync).
- Move the `is_cancelled` bail in `try_read_task_cell` before the
`listen_to_cell` call to avoid inserting phantom `InProgressCellState`
events that would never be notified.
- In `task_execution_canceled`: switch to `TaskDataCategory::All`
(needed for dirty state metadata access), notify all pending in-progress
cell events, and mark the task as `SessionDependent` dirty via the new
helper.
- In `task_execution_completed_finish`: replace ~77 lines of inline
dirty state logic with a call to
`task.update_dirty_state(new_dirtyness)`, preserving the
`all_clean_event` post-processing and the `dirty_changed` variable under
`#[cfg(feature = "verify_determinism")]`.

**`operation/mod.rs`:**
- Add `update_dirty_state` default method on `TaskGuard` trait (~60
lines), co-located with the existing `dirty_state()` reader. Takes
`Option<Dirtyness>`, applies the transition, builds
`ComputeDirtyAndCleanUpdate`, and returns
`(Option<AggregationUpdateJob>, ComputeDirtyAndCleanUpdateResult)`.
- Add `ComputeDirtyAndCleanUpdateResult` to the public re-exports.

---------

Co-authored-by: Tobias Koppers <[email protected]>
Co-authored-by: Claude <[email protected]>

…handling (#92254)

### What?

Bug fixes and a refactoring in `turbo-tasks-backend` targeting stability
issues that surface when filesystem caching is enabled:

1. **Preserve `cell_type_max_index` on task error** — when a task fails
partway through execution, `cell_counters` only reflects the
partially-executed state. Previously, `cell_type_max_index` was updated
from these incomplete counters, which removed entries for cell types not
yet encountered. This caused `"Cell no longer exists"` hard errors for
tasks that still held dependencies on those cells. The fix skips the
`cell_type_max_index` update on error, keeping it consistent with the
preserved cell data (which already wasn't cleared on error).

This bug manifested specifically with `serialization = "hash"` cell
types (e.g. `FileContent`), where cell data is transient and readers
fall back to `cell_type_max_index` to decide whether to schedule
recomputation.

2. **Fix shutdown hang and cache poisoning for cancelled tasks** — three
related fixes for tasks cancelled during shutdown:
- `task_execution_canceled` now drains and notifies all
`InProgressCellState` events, preventing `stop_and_wait` from hanging on
foreground jobs waiting on cells that will never be filled.
- `try_read_task_cell` bails early (before calling `listen_to_cell`)
when a task is in `Canceled` state, avoiding pointless listener
registrations that would never resolve.
- Cancelled tasks are marked as session-dependent dirty, preventing
cache poisoning where `"was canceled"` errors get persisted as task
output and break subsequent builds. The session-dependent dirty flag
causes the task to re-execute in the next session, invalidating stale
dependents.

3. **Extract `update_dirty_state` helper on `TaskGuard`** — the "read
old dirty state → apply new state → propagate via
`ComputeDirtyAndCleanUpdate`" pattern was duplicated between
`task_execution_canceled` and `task_execution_completed_finish`. The new
`update_dirty_state` default method on `TaskGuard` handles both
transitions (to `SessionDependent` or to `None`) and returns the
aggregation job + `ComputeDirtyAndCleanUpdateResult` for callers that
need post-processing (e.g. firing the `all_clean_event`).

### Why?

These bugs caused observable failures when using Turbopack with
filesystem caching (`--cache` / persistent cache):

- `"Cell no longer exists"` panics/errors on incremental rebuilds after
a task error.
- Hangs on `stop_and_wait` during dev server shutdown.
- Stale `"was canceled"` errors persisted in the cache breaking
subsequent builds until the cache is cleared.

### How?

Changes are in `turbopack/crates/turbo-tasks-backend/src/backend/`:

**`mod.rs`:**
- Guard the `cell_type_max_index` update block inside `if
result.is_ok()` to skip it on error, with a cross-reference comment to
`task_execution_completed_cleanup` (which similarly skips cell data
removal on error — the two must stay in sync).
- Move the `is_cancelled` bail in `try_read_task_cell` before the
`listen_to_cell` call to avoid inserting phantom `InProgressCellState`
events that would never be notified.
- In `task_execution_canceled`: switch to `TaskDataCategory::All`
(needed for dirty state metadata access), notify all pending in-progress
cell events, and mark the task as `SessionDependent` dirty via the new
helper.
- In `task_execution_completed_finish`: replace ~77 lines of inline
dirty state logic with a call to
`task.update_dirty_state(new_dirtyness)`, preserving the
`all_clean_event` post-processing and the `dirty_changed` variable under
`#[cfg(feature = "verify_determinism")]`.

**`operation/mod.rs`:**
- Add `update_dirty_state` default method on `TaskGuard` trait (~60
lines), co-located with the existing `dirty_state()` reader. Takes
`Option<Dirtyness>`, applies the transition, builds
`ComputeDirtyAndCleanUpdate`, and returns
`(Option<AggregationUpdateJob>, ComputeDirtyAndCleanUpdateResult)`.
- Add `ComputeDirtyAndCleanUpdateResult` to the public re-exports.

---------

Co-authored-by: Tobias Koppers <[email protected]>
Co-authored-by: Claude <[email protected]>

…handling (#92254)

### What?

Bug fixes and a refactoring in `turbo-tasks-backend` targeting stability
issues that surface when filesystem caching is enabled:

1. **Preserve `cell_type_max_index` on task error** — when a task fails
partway through execution, `cell_counters` only reflects the
partially-executed state. Previously, `cell_type_max_index` was updated
from these incomplete counters, which removed entries for cell types not
yet encountered. This caused `"Cell no longer exists"` hard errors for
tasks that still held dependencies on those cells. The fix skips the
`cell_type_max_index` update on error, keeping it consistent with the
preserved cell data (which already wasn't cleared on error).

This bug manifested specifically with `serialization = "hash"` cell
types (e.g. `FileContent`), where cell data is transient and readers
fall back to `cell_type_max_index` to decide whether to schedule
recomputation.

2. **Fix shutdown hang and cache poisoning for cancelled tasks** — three
related fixes for tasks cancelled during shutdown:
- `task_execution_canceled` now drains and notifies all
`InProgressCellState` events, preventing `stop_and_wait` from hanging on
foreground jobs waiting on cells that will never be filled.
- `try_read_task_cell` bails early (before calling `listen_to_cell`)
when a task is in `Canceled` state, avoiding pointless listener
registrations that would never resolve.
- Cancelled tasks are marked as session-dependent dirty, preventing
cache poisoning where `"was canceled"` errors get persisted as task
output and break subsequent builds. The session-dependent dirty flag
causes the task to re-execute in the next session, invalidating stale
dependents.

3. **Extract `update_dirty_state` helper on `TaskGuard`** — the "read
old dirty state → apply new state → propagate via
`ComputeDirtyAndCleanUpdate`" pattern was duplicated between
`task_execution_canceled` and `task_execution_completed_finish`. The new
`update_dirty_state` default method on `TaskGuard` handles both
transitions (to `SessionDependent` or to `None`) and returns the
aggregation job + `ComputeDirtyAndCleanUpdateResult` for callers that
need post-processing (e.g. firing the `all_clean_event`).

### Why?

These bugs caused observable failures when using Turbopack with
filesystem caching (`--cache` / persistent cache):

- `"Cell no longer exists"` panics/errors on incremental rebuilds after
a task error.
- Hangs on `stop_and_wait` during dev server shutdown.
- Stale `"was canceled"` errors persisted in the cache breaking
subsequent builds until the cache is cleared.

### How?

Changes are in `turbopack/crates/turbo-tasks-backend/src/backend/`:

**`mod.rs`:**
- Guard the `cell_type_max_index` update block inside `if
result.is_ok()` to skip it on error, with a cross-reference comment to
`task_execution_completed_cleanup` (which similarly skips cell data
removal on error — the two must stay in sync).
- Move the `is_cancelled` bail in `try_read_task_cell` before the
`listen_to_cell` call to avoid inserting phantom `InProgressCellState`
events that would never be notified.
- In `task_execution_canceled`: switch to `TaskDataCategory::All`
(needed for dirty state metadata access), notify all pending in-progress
cell events, and mark the task as `SessionDependent` dirty via the new
helper.
- In `task_execution_completed_finish`: replace ~77 lines of inline
dirty state logic with a call to
`task.update_dirty_state(new_dirtyness)`, preserving the
`all_clean_event` post-processing and the `dirty_changed` variable under
`#[cfg(feature = "verify_determinism")]`.

**`operation/mod.rs`:**
- Add `update_dirty_state` default method on `TaskGuard` trait (~60
lines), co-located with the existing `dirty_state()` reader. Takes
`Option<Dirtyness>`, applies the transition, builds
`ComputeDirtyAndCleanUpdate`, and returns
`(Option<AggregationUpdateJob>, ComputeDirtyAndCleanUpdateResult)`.
- Add `ComputeDirtyAndCleanUpdateResult` to the public re-exports.

---------

Co-authored-by: Tobias Koppers <[email protected]>
Co-authored-by: Claude <[email protected]>

#92108)

### What?

Re-lands #91576 ("turbo-tasks: add hashed cell mode for hash-based
change detection without cell data"), which was reverted in #92103 due
to a `FATAL` crash in the `filesystem-cache` test suite.

Includes a bug fix on top: in `task_execution_completed_prepare`, skip
updating `cell_type_max_index` when the task completed with an error.

Also adds a `CellHash = [u8; 16]` type alias (requested in review) used
throughout the hash pipeline.

### Why?

**The original feature** (`serialization = "hash"` on `FileContent` and
`Code`) stores a hash of the cell data instead of the full serialized
value. On session restore, the hash is used to detect whether cell
content has changed without needing the full data in memory. This avoids
a large persistent cache size increase.

**The bug** that caused the revert: When a task fails partway through
re-execution (before recreating all the cells from its previous run),
`cell_counters` only reflects the partially-executed state. The old code
used those partial counters to update `cell_type_max_index`, removing
entries for cell types that were not yet created at the point of
failure. This caused downstream tasks that still held cell dependencies
from the previous successful run to hit a hard "Cell no longer exists"
error.

**Concrete failure path** in `filesystem-cache rename app page` test:

1. `get_app_page_entry` runs for `/remove-me/page`, creating two
`FileContent` cells (indices 0 and 1). `cell_type_max_index[FileContent]
= 2` is persisted.
2. The folder is renamed (`app/remove-me` → `app/add-me`), dirtying the
task.
3. On re-execution, `get_app_page_entry` fails at `config.await?` (the
loader tree errors because the directory is gone) — before any
`FileContent::cell()` calls.
4. `cell_counters` has no `FileContent` entry → old code removed
`cell_type_max_index[FileContent]`.
5. The `parse` task tries to read `FileContent` cell 1 from
`get_app_page_entry` → `cell_type_max_index` is `None` → **"Cell no
longer exists" panic → FATAL error**.

**Why it didn't crash before** `serialization = "hash"`: `FileContent`
was previously serializable, so `parse` read stale cell data directly
from `persistent_cell_data`, which `task_execution_completed_cleanup`
already preserves on error. With `serialization = "hash"`, data is
transient — readers fall back to `cell_type_max_index` for range
validation, where a stale `None` caused the crash.

### How?

#### Core feature: `serialization = "hash"` cell mode

- New `SerializationMode::Hash` variant in `turbo-tasks-macros` — marks
a value type as non-serializable but stores a `DeterministicHash` of the
cell data for change detection.
- `VcCellHashedCompareMode<T>` cell mode: compares values via
`PartialEq` when available, falls back to hash comparison when transient
data has been evicted.
- `hashed_compare_and_update` /
`hashed_compare_and_update_with_shared_reference` on `CurrentCellRef`
compute and pass content hashes through the update pipeline.
- Backend `update_cell` uses hash-based comparison to skip invalidation
when the old cell data is unavailable but the hash matches.
- `cell_data_hash: AutoMap<CellId, CellHash>` field in task storage
persists hashes across sessions.
- Stale `cell_data_hash` entries are cleaned up in
`task_execution_completed_cleanup` alongside cell data removal.
- `CellHash = [u8; 16]` type alias keeps alignment at 1 byte to avoid
padding growth in `AutoMap`/`LazyField` enum variants.
- Hash bytes use little-endian encoding (`to_le_bytes`) for
cross-platform cache portability.

#### Bug fix: preserve `cell_type_max_index` on task error

In `task_execution_completed_prepare`, guard the `cell_type_max_index`
update block with `if result.is_ok()`. This mirrors the existing
`task_execution_completed_cleanup` behavior that already skips cell data
removal when `is_error` is true, keeping `cell_type_max_index`
consistent with the preserved transient cell data.

#### Applied to `FileContent` and `Code`

- `FileContent` uses `serialization = "hash"` — full content is
persisted via a separate `PersistedFileContent` type when needed (e.g.,
in `DiskFileSystem::write`).
- `Code` uses `serialization = "hash"` with `Arc<Vec<Mapping>>` for
cheap cloning. `Code::cell_persisted()` creates a `PersistedCode` cell
directly and returns `Vc<Code>` via `PersistedCode::to_code()`, avoiding
an intermediate hash-mode cell.

#### Other improvements

- `DeterministicHash` impls for `SmallVec` and `()`.
- `Xxh3Hash128Hasher::finish_bytes()` method returning `[u8; 16]`.
- `hash = "manual"` option on `#[turbo_tasks::value]` to opt out of
auto-deriving `DeterministicHash`.

**Note:** The shutdown hang and cache poisoning fixes that were
previously on this branch have been merged separately via #92254.

### Test plan

- [x] `test/e2e/filesystem-cache/filesystem-cache.test.ts` passes (all
17 tests)
- [x] New
`turbopack/crates/turbo-tasks-backend/tests/hashed_cell_mode.rs`
integration test verifies hash-based change detection: value changes
trigger invalidation, equal values (same hash) do not
- [x] `cargo check` passes for `turbo-tasks`, `turbo-tasks-backend`,
`turbo-tasks-fs`, `turbopack-core`, `turbopack-ecmascript`
- [x] CI green (attempt 2)

<!-- NEXT_JS_LLM_PR -->

---------

Co-authored-by: Tobias Koppers <[email protected]>
Co-authored-by: Claude <[email protected]>

…handling (#92254)

Bug fixes and a refactoring in `turbo-tasks-backend` targeting stability
issues that surface when filesystem caching is enabled:

1. **Preserve `cell_type_max_index` on task error** — when a task fails
partway through execution, `cell_counters` only reflects the
partially-executed state. Previously, `cell_type_max_index` was updated
from these incomplete counters, which removed entries for cell types not
yet encountered. This caused `"Cell no longer exists"` hard errors for
tasks that still held dependencies on those cells. The fix skips the
`cell_type_max_index` update on error, keeping it consistent with the
preserved cell data (which already wasn't cleared on error).

This bug manifested specifically with `serialization = "hash"` cell
types (e.g. `FileContent`), where cell data is transient and readers
fall back to `cell_type_max_index` to decide whether to schedule
recomputation.

2. **Fix shutdown hang and cache poisoning for cancelled tasks** — three
related fixes for tasks cancelled during shutdown:
- `task_execution_canceled` now drains and notifies all
`InProgressCellState` events, preventing `stop_and_wait` from hanging on
foreground jobs waiting on cells that will never be filled.
- `try_read_task_cell` bails early (before calling `listen_to_cell`)
when a task is in `Canceled` state, avoiding pointless listener
registrations that would never resolve.
- Cancelled tasks are marked as session-dependent dirty, preventing
cache poisoning where `"was canceled"` errors get persisted as task
output and break subsequent builds. The session-dependent dirty flag
causes the task to re-execute in the next session, invalidating stale
dependents.

3. **Extract `update_dirty_state` helper on `TaskGuard`** — the "read
old dirty state → apply new state → propagate via
`ComputeDirtyAndCleanUpdate`" pattern was duplicated between
`task_execution_canceled` and `task_execution_completed_finish`. The new
`update_dirty_state` default method on `TaskGuard` handles both
transitions (to `SessionDependent` or to `None`) and returns the
aggregation job + `ComputeDirtyAndCleanUpdateResult` for callers that
need post-processing (e.g. firing the `all_clean_event`).

These bugs caused observable failures when using Turbopack with
filesystem caching (`--cache` / persistent cache):

- `"Cell no longer exists"` panics/errors on incremental rebuilds after
a task error.
- Hangs on `stop_and_wait` during dev server shutdown.
- Stale `"was canceled"` errors persisted in the cache breaking
subsequent builds until the cache is cleared.

Changes are in `turbopack/crates/turbo-tasks-backend/src/backend/`:

**`mod.rs`:**
- Guard the `cell_type_max_index` update block inside `if
result.is_ok()` to skip it on error, with a cross-reference comment to
`task_execution_completed_cleanup` (which similarly skips cell data
removal on error — the two must stay in sync).
- Move the `is_cancelled` bail in `try_read_task_cell` before the
`listen_to_cell` call to avoid inserting phantom `InProgressCellState`
events that would never be notified.
- In `task_execution_canceled`: switch to `TaskDataCategory::All`
(needed for dirty state metadata access), notify all pending in-progress
cell events, and mark the task as `SessionDependent` dirty via the new
helper.
- In `task_execution_completed_finish`: replace ~77 lines of inline
dirty state logic with a call to
`task.update_dirty_state(new_dirtyness)`, preserving the
`all_clean_event` post-processing and the `dirty_changed` variable under
`#[cfg(feature = "verify_determinism")]`.

**`operation/mod.rs`:**
- Add `update_dirty_state` default method on `TaskGuard` trait (~60
lines), co-located with the existing `dirty_state()` reader. Takes
`Option<Dirtyness>`, applies the transition, builds
`ComputeDirtyAndCleanUpdate`, and returns
`(Option<AggregationUpdateJob>, ComputeDirtyAndCleanUpdateResult)`.
- Add `ComputeDirtyAndCleanUpdateResult` to the public re-exports.

---------

Co-authored-by: Tobias Koppers <[email protected]>
Co-authored-by: Claude <[email protected]>