Repository navigation
Windows PowerShell UTF-8 Markdown files can render as mojibake in agent context, causing patch mismatch and unreliable edits #15422
Description
Problem
On Windows with PowerShell 5.1, UTF-8 Markdown files sometimes showed up as mojibake in the agent/tool context even though the files themselves were valid UTF-8 on disk.
Typical example:
—showing up as—
This made patch-based edits unreliable, because the tool seemed to be matching against the misdecoded text it was seeing rather than the real file contents.
What I observed
The strange part was that the file was not actually corrupted.
- Normal agent/shell context could show mojibake
Get-Content -Encoding utf8 <file>showed the file correctly- hex inspection confirmed the file bytes were valid UTF-8
So the problem looked less like “the file was broken” and more like “the tool/context path was decoding it incorrectly.”
Environment
- Windows
- PowerShell 5.1
- Initial code page:
437 - Initial console input encoding:
IBM437 - Initial
$OutputEncoding:us-ascii
Why this matters
When the displayed context is wrong, patch/edit operations can fail even though the file itself is fine.
This was especially painful with large .md files, because the agent could end up seeing mojibaked context, fail to match hunks, and behave as if the file content were different from what was really on disk.
How I worked around it
These local environment changes helped a lot:
chcp 65001- set console input/output encoding to UTF-8
- set
$OutputEncodingto UTF-8
After that, UTF-8 reads and edit verification behaved much more reliably.
Expected behavior
The tool should read UTF-8 files correctly for context and patch matching regardless of Windows shell/codepage defaults, or at least use a deterministic UTF-8 file-reading path instead of inheriting console encoding behavior.
Possible related issue
There may also be a separate Windows issue where editing Markdown can normalize line endings in touched regions. That was much less serious than the mojibake problem, but it showed up during the same debugging session.
Activity
github-actions commentedon Mar 21, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
Kohgane commentedon Sep 6, 2026
Hit this exact class of bug on Korean Windows (CP949 instead of 437) and it took a while to accept that the file was never the problem — the decode path was.
Worth separating two traps that look identical from the outside:
Reading. PowerShell 5.1's
Get-Contentdefaults to the system ANSI code page, not UTF-8. A UTF-8 file that goes through read → modify → write comes back mangled, and the damage is now on disk.Get-Content -Encoding UTF8on read, and[IO.File]::WriteAllText($p, $s, (New-Object System.Text.UTF8Encoding($false)))on write if you need to stay BOM-less.Parsing.
powershell.exereads the .ps1 source itself in the ANSI code page. A BOM-less script containing non-ASCII string literals is corrupted before execution — nothing at runtime can repair it. Either save with a BOM, or keep the source pure ASCII and decode non-ASCII strings from base64 at runtime. I went with the second, because the file had to stay BOM-less for other tooling.For an agent this matters more than for a human at a prompt: when the context the model sees is misdecoded, patch matching fails against text that never existed on disk, and the failure surfaces as "the edit didn't apply" rather than "the encoding is wrong" — so people go looking in the wrong place.
Why it's invisible to most users: CP1252 and UTF-8 agree on every byte below 0x80. If your paths and strings are ASCII, the bug is present and produces no symptom.
Wrote both up with the verification steps, along with ~90 other traps hit while running AI coding agents: https://kohgane.github.io/ai-coding-mines/en/01-windows-powershell/