Skip to content

fix: address delivery dedupe review follow-ups#44666

Merged
frankekn merged 5 commits intomainfrom
fix/44109-review-fixes
Mar 13, 2026
Merged

fix: address delivery dedupe review follow-ups#44666
frankekn merged 5 commits intomainfrom
fix/44109-review-fixes

Conversation

@frankekn
Copy link
Contributor

Summary

  • fix completed direct cron delivery cache pruning so the cap is enforced correctly
  • make transcript mirror idempotency scanning async and tolerant of malformed transcript lines
  • add regression tests for both review findings

Context

This replaces the closed #44109 follow-up branch with a fresh replacement PR.

Testing

  • pnpm exec vitest run --config vitest.unit.config.ts src/config/sessions/sessions.test.ts src/cron/isolated-agent/delivery-dispatch.double-announce.test.ts
  • pnpm tsgo
  • pnpm build
  • pnpm check
  • pnpm test

@openclaw-barnacle openclaw-barnacle bot added gateway Gateway runtime agents Agent runtime and tooling size: M maintainer Maintainer-authored PR labels Mar 13, 2026
@aisle-research-bot
Copy link

aisle-research-bot bot commented Mar 13, 2026

🔒 Aisle Security Analysis

We found 2 potential security issue(s) in this PR:

# Severity Title
1 🟠 High Client-controlled idempotencyKey can suppress mirrored transcript entries (audit/integrity bypass)
2 🟠 High Unbounded transcript scan for idempotency key causes O(n) per append (DoS risk)

1. 🟠 Client-controlled idempotencyKey can suppress mirrored transcript entries (audit/integrity bypass)

Property Value
Severity High
CWE CWE-642
Location src/config/sessions/transcript.ts:183-188

Description

appendAssistantMessageToSessionTranscript now deduplicates mirrored assistant messages by idempotencyKey and skips appending if the key already exists in the transcript.

Trust boundary problem:

  • idempotencyKey originates from gateway request parameters (request.idempotencyKey) and is therefore client-controlled.
  • That client-controlled key is propagated into mirror.idempotencyKey and ultimately into transcript appends.
  • Because transcript dedupe checks the entire persisted transcript file, a client can reuse an old key (e.g., after a gateway restart / in-memory dedupe eviction) to send a new outbound message while preventing the mirror entry from being recorded.

Impact:

  • Enables stealth outbound sends in sessions that rely on mirroring for audit/history.
  • Can desync “what was sent” vs “what was logged” (especially if delivery still occurs but transcript append is skipped).

Vulnerable code:

if (
  params.idempotencyKey &&
  (await transcriptHasIdempotencyKey(sessionFile, params.idempotencyKey))
) {
  return { ok: true, sessionFile };
}

Recommendation

Do not use a client-controlled idempotency key as the authority for transcript de-duplication.

Mitigations (pick one or combine):

  1. Generate a server-side transcript idempotency key that is scoped to the concrete delivery event (e.g., messageId + sessionKey), and keep the client idempotency key only for request retry semantics.

  2. If a client-supplied key must be accepted, make dedupe conditional on the content matching as well (do not suppress when the same key is reused with different mirrored content).

  3. Add strict validation and limits for idempotency keys (format + max length) at the protocol/schema boundary.

Example (server-side key):

// When you have a stable delivery identifier:
const transcriptIdempotencyKey = `mirror:v1:${sessionKey}:${result.messageId}`;

await appendAssistantMessageToSessionTranscript({
  sessionKey,
  text: mirrorText,
  idempotencyKey: transcriptIdempotencyKey,
});

And enforce bounds in the gateway schema, e.g.:

const IdempotencyKey = Type.String({ minLength: 1, maxLength: 128, pattern: "^[a-zA-Z0-9:_-]+$" });

2. 🟠 Unbounded transcript scan for idempotency key causes O(n) per append (DoS risk)

Property Value
Severity High
CWE CWE-400
Location src/config/sessions/transcript.ts:220-238

Description

appendAssistantMessageToSessionTranscript() now supports idempotencyKey de-duplication by scanning the entire session transcript JSONL file on every append.

Impact:

  • Unbounded memory usage: fs.promises.readFile(transcriptPath, "utf-8") loads the entire transcript into memory.
  • Unbounded CPU usage / algorithmic complexity: it then split()s into lines and JSON.parse()s each line, making each append O(file_size). Over time, repeated appends to the same session become O(n²) total work.
  • Malformed line handling does not mitigate resource use: malformed JSON is silently ignored, but still requires scanning/splitting the full file contents.

This is reachable in normal operation because outbound delivery mirroring passes an idempotency key through to transcript appends, and gateway callers can supply idempotency keys as arbitrary non-empty strings.

Vulnerable code:

const raw = await fs.promises.readFile(transcriptPath, "utf-8");
for (const line of raw.split(/\r?\n/)) {
  ...
  const parsed = JSON.parse(line);
  ...
}

Recommendation

Avoid full-file reads/parses on the hot path.

Recommended mitigations (choose one or combine):

  1. Stream the file line-by-line (bounded memory) and short-circuit on match:
import readline from "node:readline";

async function transcriptHasIdempotencyKey(transcriptPath: string, key: string): Promise<boolean> {
  const stream = fs.createReadStream(transcriptPath, { encoding: "utf-8" });
  const rl = readline.createInterface({ input: stream, crlfDelay: Infinity });
  for await (const line of rl) {
    if (!line.trim()) continue;
    try {
      const parsed = JSON.parse(line) as { message?: { idempotencyKey?: unknown } };
      if (parsed.message?.idempotencyKey === key) return true;
    } catch {// optionally count/log bad lines
    }
  }
  return false;
}
  1. Bound the scan (e.g., only last N bytes/lines) since duplicates are typically recent retries.

  2. Cap and validate inputs: enforce a reasonable maximum length/charset for idempotencyKey before storing or scanning.

  3. Introduce transcript rotation/compaction guarantees (or enable disk budgets by default) so individual transcript files cannot grow without bound.


Analyzed PR: #44666 at commit 8e6d254

Last updated on: 2026-03-13T09:16:00Z

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Mar 13, 2026

Greptile Summary

This PR addresses two follow-up issues from a delivery deduplication review: it fixes cache pruning for completed direct cron deliveries so the 2 000-entry cap is reliably enforced, and it makes the transcript mirror idempotency scan async and tolerant of malformed JSONL lines. It also consolidates previously duplicated mirror-payload type definitions into a single src/infra/outbound/mirror.ts module and propagates idempotencyKey through all mirror paths (plugin-handled sends, gateway send, and the core deliver layer).

Key changes:

  • delivery-dispatch.ts: New in-memory COMPLETED_DIRECT_CRON_DELIVERIES map with TTL + size-based pruning (pruneCompletedDirectCronDeliveries). Pruning is invoked both on write (rememberCompletedDirectCronDelivery) and read (getCompletedDirectCronDelivery), correctly keeping the map at ≤ 2 000 entries.
  • transcript.ts: appendAssistantMessageToSessionTranscript now accepts an optional idempotencyKey; before appending it calls transcriptHasIdempotencyKey which scans the transcript JSONL file line-by-line, skipping malformed lines, so a replay of the same message is a no-op.
  • subagent-registry.ts: finalizeAnnounceCleanup wrapper resets cleanupHandled = false and re-persists the registry when finalizeSubagentCleanup itself throws, making the cleanup retryable on the next registry restart.
  • mirror.ts (new): Single canonical home for OutboundMirror and DeliveryMirror types, eliminating four inline duplicate definitions across message.ts, outbound-send-service.ts, delivery-queue.ts, and deliver.ts.

Confidence Score: 4/5

  • Safe to merge; changes are well-tested, logically sound, and limited in scope to deduplication and type consolidation.
  • The pruning logic, idempotency scan, and cleanup retry path are all covered by new regression tests. The two flagged points are minor: the cloneDeliveryResults allocation that is immediately discarded on a cache hit, and the inherent TOCTOU window in the file-based idempotency check (acceptable for best-effort mirror deduplication). No missing tests, no risky behavioral changes in the hot delivery path.
  • No files require special attention; src/cron/isolated-agent/delivery-dispatch.ts and src/config/sessions/transcript.ts carry the two minor style notes above but are otherwise correct.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/cron/isolated-agent/delivery-dispatch.ts
Line: 333-337

Comment:
**Cloned results allocated but immediately discarded**

`getCompletedDirectCronDelivery` deep-clones the cached `OutboundDeliveryResult[]` via `cloneDeliveryResults`, but the returned value is only inspected for truthiness at the call site — the actual array is never read. This allocates a fresh array + object spread on every cache hit for no benefit.

Consider returning a simple boolean from a helper, or refactoring `getCompletedDirectCronDelivery` to return `boolean` when the caller only needs existence:

```suggestion
      const isCachedDelivery = (() => {
        pruneCompletedDirectCronDeliveries(Date.now());
        return COMPLETED_DIRECT_CRON_DELIVERIES.has(deliveryIdempotencyKey);
      })();
      if (isCachedDelivery) {
        // Cached entries are only recorded after a successful non-empty delivery.
        delivered = true;
        return null;
      }
```

Or, more minimally, just inline:

```ts
if (getCompletedDirectCronDelivery(deliveryIdempotencyKey)) {
  delivered = true;
  return null;
}
```
and have `getCompletedDirectCronDelivery` return `boolean` instead of `OutboundDeliveryResult[] | undefined`.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/config/sessions/transcript.ts
Line: 183-188

Comment:
**TOCTOU window between idempotency check and append**

The `transcriptHasIdempotencyKey` read and the `sessionManager.appendMessage` call are not atomic. Two concurrent invocations with the same `idempotencyKey` could both read the file, both find no match, and both append — defeating the deduplication. This is an inherent limitation of the file-based approach without a write lock.

If the transcript write path already holds an exclusive advisory lock (e.g. in `SessionManager.open`), this is fine and can be documented with a comment. If it does not, a note explaining that this is best-effort deduplication would make the intent clearer for future contributors.

How can I resolve this? If you propose a fix, please make it concise.

Last reviewed commit: 5baa548

Comment on lines +333 to +337
const cachedResults = getCompletedDirectCronDelivery(deliveryIdempotencyKey);
if (cachedResults) {
// Cached entries are only recorded after a successful non-empty delivery.
delivered = true;
return null;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cloned results allocated but immediately discarded

getCompletedDirectCronDelivery deep-clones the cached OutboundDeliveryResult[] via cloneDeliveryResults, but the returned value is only inspected for truthiness at the call site — the actual array is never read. This allocates a fresh array + object spread on every cache hit for no benefit.

Consider returning a simple boolean from a helper, or refactoring getCompletedDirectCronDelivery to return boolean when the caller only needs existence:

Suggested change
const cachedResults = getCompletedDirectCronDelivery(deliveryIdempotencyKey);
if (cachedResults) {
// Cached entries are only recorded after a successful non-empty delivery.
delivered = true;
return null;
const isCachedDelivery = (() => {
pruneCompletedDirectCronDeliveries(Date.now());
return COMPLETED_DIRECT_CRON_DELIVERIES.has(deliveryIdempotencyKey);
})();
if (isCachedDelivery) {
// Cached entries are only recorded after a successful non-empty delivery.
delivered = true;
return null;
}

Or, more minimally, just inline:

if (getCompletedDirectCronDelivery(deliveryIdempotencyKey)) {
  delivered = true;
  return null;
}

and have getCompletedDirectCronDelivery return boolean instead of OutboundDeliveryResult[] | undefined.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/cron/isolated-agent/delivery-dispatch.ts
Line: 333-337

Comment:
**Cloned results allocated but immediately discarded**

`getCompletedDirectCronDelivery` deep-clones the cached `OutboundDeliveryResult[]` via `cloneDeliveryResults`, but the returned value is only inspected for truthiness at the call site — the actual array is never read. This allocates a fresh array + object spread on every cache hit for no benefit.

Consider returning a simple boolean from a helper, or refactoring `getCompletedDirectCronDelivery` to return `boolean` when the caller only needs existence:

```suggestion
      const isCachedDelivery = (() => {
        pruneCompletedDirectCronDeliveries(Date.now());
        return COMPLETED_DIRECT_CRON_DELIVERIES.has(deliveryIdempotencyKey);
      })();
      if (isCachedDelivery) {
        // Cached entries are only recorded after a successful non-empty delivery.
        delivered = true;
        return null;
      }
```

Or, more minimally, just inline:

```ts
if (getCompletedDirectCronDelivery(deliveryIdempotencyKey)) {
  delivered = true;
  return null;
}
```
and have `getCompletedDirectCronDelivery` return `boolean` instead of `OutboundDeliveryResult[] | undefined`.

How can I resolve this? If you propose a fix, please make it concise.

Comment on lines +183 to +188
if (
params.idempotencyKey &&
(await transcriptHasIdempotencyKey(sessionFile, params.idempotencyKey))
) {
return { ok: true, sessionFile };
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TOCTOU window between idempotency check and append

The transcriptHasIdempotencyKey read and the sessionManager.appendMessage call are not atomic. Two concurrent invocations with the same idempotencyKey could both read the file, both find no match, and both append — defeating the deduplication. This is an inherent limitation of the file-based approach without a write lock.

If the transcript write path already holds an exclusive advisory lock (e.g. in SessionManager.open), this is fine and can be documented with a comment. If it does not, a note explaining that this is best-effort deduplication would make the intent clearer for future contributors.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/config/sessions/transcript.ts
Line: 183-188

Comment:
**TOCTOU window between idempotency check and append**

The `transcriptHasIdempotencyKey` read and the `sessionManager.appendMessage` call are not atomic. Two concurrent invocations with the same `idempotencyKey` could both read the file, both find no match, and both append — defeating the deduplication. This is an inherent limitation of the file-based approach without a write lock.

If the transcript write path already holds an exclusive advisory lock (e.g. in `SessionManager.open`), this is fine and can be documented with a comment. If it does not, a note explaining that this is best-effort deduplication would make the intent clearer for future contributors.

How can I resolve this? If you propose a fix, please make it concise.

@frankekn frankekn force-pushed the fix/44109-review-fixes branch 3 times, most recently from b9bbd2f to 90b5940 Compare March 13, 2026 06:58
@openclaw-barnacle openclaw-barnacle bot added app: android App: android commands Command implementations size: XL and removed size: M labels Mar 13, 2026
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 415a8a7382

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1665 to +1667
Text("Pairing Required", style = onboardingHeadlineStyle, color = onboardingWarning)
Text("Run these on your gateway host:", style = onboardingCalloutStyle, color = onboardingTextSecondary)
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Show actual connection failure instead of generic pairing prompt

This branch now treats every attempted-but-disconnected state as "Pairing Required," but statusText is no longer surfaced in FinalStep, so users lose the real failure reason. The runtime reports non-pairing errors such as Failed: invalid manual host/port and TLS probe failures in apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt (e.g. around lines 788 and 732), so this UI change can direct users to run pairing commands when the fix is actually endpoint or TLS configuration.

Useful? React with 👍 / 👎.

@frankekn frankekn force-pushed the fix/44109-review-fixes branch from 415a8a7 to 2f17f1a Compare March 13, 2026 07:42
@openclaw-barnacle openclaw-barnacle bot added size: M and removed app: android App: android commands Command implementations size: XL labels Mar 13, 2026
@frankekn frankekn force-pushed the fix/44109-review-fixes branch from 2f17f1a to 8e6d254 Compare March 13, 2026 08:08
@frankekn frankekn merged commit f07033e into main Mar 13, 2026
29 checks passed
@frankekn frankekn deleted the fix/44109-review-fixes branch March 13, 2026 08:18
frankekn added a commit to cheapestinference/openclaw that referenced this pull request Mar 13, 2026
Merged via squash.

Prepared head SHA: 8e6d254
Co-authored-by: frankekn <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
Jerry-Xin pushed a commit to Jerry-Xin/openclaw that referenced this pull request Mar 13, 2026
Merged via squash.

Prepared head SHA: 8e6d254
Co-authored-by: frankekn <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
hougangdev pushed a commit to hougangdev/clawdbot that referenced this pull request Mar 14, 2026
Merged via squash.

Prepared head SHA: 8e6d254
Co-authored-by: frankekn <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
ecochran76 pushed a commit to ecochran76/openclaw that referenced this pull request Mar 14, 2026
Merged via squash.

Prepared head SHA: 8e6d254
Co-authored-by: frankekn <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling gateway Gateway runtime maintainer Maintainer-authored PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant