Secret redaction corrupts JSON output when tool results contain harmless authorization-header code #5092

Describe the bug

Describe the bug

Copilot CLI can emit invalid JSON in --output-format json when a tool result contains harmless source text such as:

self.authorization_prefix = "Bearer "

The observed output path serializes an event, then applies the plain-text secret filter to the serialized JSON. That filter can consume JSON escape characters around a detected authorization value. The result is no longer a valid JSON object, so a strict JSONL consumer rejects the event.

This is distinct from overly aggressive masking alone: masking must not invalidate the surrounding transport.

Affected version

• Confirmed against the published @github/copilot 1.0.94 distribution using its actual event writer and native filter on macOS arm64.
• A separate Linux automation run using 1.0.94 produced malformed tool-completion JSON at authorization-redaction boundaries.
• The same plain-text-filter-after-serialization pattern was found in 1.0.94. That is a code-inspection observation, not a claim that a complete live 1.0.94 session was reproduced.

Affected version

No response

Steps to reproduce the behavior

Low-level diagnostic reproduction

The following diagnostic exercises the problematic serialization/filter sequence directly. It reads the existing native module; it does not modify the distribution or disable protection in a running CLI session.

On macOS arm64, set COPILOT_PACKAGE_DIR to an extracted 1.0.94 package directory containing package.json, app.js, and prebuilds/darwin-arm64/runtime.node, then run:

// Save as repro.cjs and run: node repro.cjs
const path = require("node:path");
const root = process.env.COPILOT_PACKAGE_DIR;
if (!root) throw new Error("Set COPILOT_PACKAGE_DIR to the extracted package");
const filter = require(path.join(root, "prebuilds/darwin-arm64/runtime.node"));

// Fresh diagnostic process, deliberately no registered credential values.
filter.processSecretFilterReset();
const event = {
  type: "tool.execution_complete",
  id: "synthetic-event",
  data: {
    toolCallId: "synthetic-tool",
    success: true,
    result: { content: 'self.authorization_prefix = "Bearer "' }
  }
};
const serialized = JSON.stringify(event);
const emitted = filter.processSecretFilterFilter(serialized).value;
console.log(emitted);
JSON.parse(emitted); // Throws: the redacted event is not valid JSON.

Verification performed

The report's exact JavaScript was executed against an unmodified official 1.0.94 macOS arm64 package, verified against the published SHA-256 checksum.

• The input event parsed successfully before filtering.

• No credential values were registered with the filter.

• After filtering, both Node.js and Python rejected the output at character offset 168. Node.js reported:

  SyntaxError: Expected ',' or '}' after property value
  in JSON at position 168 (line 1 column 169)
  

• A control containing self.authorization_prefix = "ordinary text" remained valid JSON; the otherwise equivalent input containing "Bearer " did not.

• The actual event-writing function extracted from the package produced output byte-for-byte identical to the diagnostic script. Only its event-suppression guards were stubbed to allow the test event through; serialization, filtering, and output-writing logic were unchanged.

• The JavaScript bundle and native filtering module retained their original hashes after execution.

Scope: This confirms corruption in the packaged event-writing/filtering path. It is not an end-to-end CLI-session reproduction, and it does not establish whether version 1.0.94 is affected.

Output illustration

These are internal diagnostic entry points, not a proposed supported application API.

The minimal string-field illustration is:

Before filtering:

{"content":"self.authorization_prefix = \"Bearer \""}

After the problematic filtering:

{"content":"self.authorization_prefix = \"******""}

The quote escape is lost. Harmless quoted headers, multiline diffs, backslashes, and nested tool-result strings also reproduced invalid JSON in the diagnostic cases.

Expected behavior

Expected behavior

• Every emitted JSONL event remains valid JSON.
• Redaction preserves event structure and value types.
• Actual credentials remain redacted.
• Harmless authorization-prefix source literals should not corrupt output.

Additional context and suggested investigation

The distribution also exposes a JSON-aware filter, processSecretFilterFilterJsonString. In isolated synthetic comparisons, that path preserved JSON structure and harmless source text while still masking the synthetic credentials tested. This suggests checking whether all serialized-output writers should use the JSON-aware path, or equivalently redact structured values before serialization.

Please audit the event, summary, and workflow-result serialization paths, not just the reader. Regression coverage should include quotes, backslashes, newlines, nested results, and synthetic secrets, and assert both JSON validity and continued credential masking.

This report does not request disabling redaction, repairing malformed events in consumers, or patching installed CLI bundles. It requests an upstream producer fix and a released version containing it. The isolated comparison is diagnostic evidence, not proof that the suggested change covers every runtime path.

Related: #4065 reports false-positive masking of legitimate authorization-header text. This report isolates the additional JSON transport corruption.

Additional context

No response