subprocess on Windows returns None instead of raising UnicodeDecodeError

Bug description:

On Windows, Popen.communicate() reads stdout and stderr in background threads when a timeout is given, or when more than one standard stream is a pipe. In text mode, the decoding also happens in these threads, in _readerthread():

cpython/Lib/subprocess.py

Lines 1780 to 1782 in 16b357b

    
           def _readerthread(self, fh, buffer): 
        
               buffer.append(fh.read()) 
        
               fh.close()

If decoding fails, the UnicodeDecodeError stays in the thread. It is only printed as "Exception in thread ... (_readerthread)". The caller never gets it, and communicate() returns None for that stream:

cpython/Lib/subprocess.py

Lines 1849 to 1850 in 16b357b

    
           stdout = stdout[0] if stdout else None 
        
           stderr = stderr[0] if stderr else None

So subprocess.run(..., capture_output=True, text=True) returns normally with stdout=None. The program then fails later with a confusing error, like TypeError: ... 'NoneType'.

The other code paths raise the error. On POSIX, _communicate() decodes in the calling thread. On Windows with one pipe and no timeout, communicate() also reads in the calling thread:

cpython/Lib/subprocess.py

Lines 1372 to 1387 in 16b357b

    
           # Optimization: If we are not worried about timeouts, we haven't 
        
           # started communicating, and we have one or zero pipes, using select() 
        
           # or threads is unnecessary. 
        
           if (timeout is None and not self._communication_started and 
        
               [self.stdin, self.stdout, self.stderr].count(None) >= 2): 
        
               stdout = None 
        
               stderr = None 
        
               if self.stdin: 
        
                   self._stdin_write(input) 
        
               elif self.stdout: 
        
                   stdout = self.stdout.read() 
        
                   self.stdout.close() 
        
               elif self.stderr: 
        
                   stderr = self.stderr.read() 
        
                   self.stderr.close() 
        
               self.wait()

Minimal example:

import subprocess, sys

# The child process writes bytes that are not valid UTF-8.
cmd = [sys.executable, "-c", "import sys; sys.stdout.buffer.write(b'ok \\xff\\n')"]

def run(label, **kwargs):
    try:
        r = subprocess.run(cmd, encoding="utf-8", **kwargs)
        print(label, "-> returned, stdout =", repr(r.stdout), flush=True)
    except UnicodeDecodeError as e:
        print(label, "-> raised", repr(e), flush=True)

run("stdout=PIPE", stdout=subprocess.PIPE)
run("stdout=PIPE, timeout=10", stdout=subprocess.PIPE, timeout=10)
run("capture_output=True", capture_output=True)

Output on Windows 11, Python 3.14.5:

stdout=PIPE -> raised UnicodeDecodeError('utf-8', b'ok \xff\n', 3, 4, 'invalid start byte')
Exception in thread Thread-1 (_readerthread):
Traceback (most recent call last):
  File "...\Lib\threading.py", line 1082, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "...\Lib\threading.py", line 1024, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\subprocess.py", line 1614, in _readerthread
    buffer.append(fh.read())
                  ~~~~~~~^^
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 3: invalid start byte
stdout=PIPE, timeout=10 -> returned, stdout = None
Exception in thread Thread-2 (_readerthread):
Traceback (most recent call last):
  File "...\Lib\threading.py", line 1082, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "...\Lib\threading.py", line 1024, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\subprocess.py", line 1614, in _readerthread
    buffer.append(fh.read())
                  ~~~~~~~^^
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 3: invalid start byte
capture_output=True -> returned, stdout = None

3.12.13, 3.13.13 and 3.15.0b1 give the same three results.

Output on Linux (Ubuntu on WSL 2), Python 3.12.3:

stdout=PIPE -> raised UnicodeDecodeError('utf-8', b'ok \xff\n', 3, 4, 'invalid start byte')
stdout=PIPE, timeout=10 -> raised UnicodeDecodeError('utf-8', b'ok \xff\n', 3, 4, 'invalid start byte')
capture_output=True -> raised UnicodeDecodeError('utf-8', b'ok \xff\n', 3, 4, 'invalid start byte')

I expected all three calls to raise UnicodeDecodeError on Windows too, like on Linux.

This will happen more often with 3.15. Python 3.15 uses UTF-8 as the default encoding (What's New), so text=True now decodes as UTF-8 on Windows. But many Windows programs still write in the console code page. For example, on my Traditional Chinese Windows (code page 950):

import subprocess
r = subprocess.run(["where", "no_such_program"], capture_output=True, text=True)
print("returncode =", r.returncode, "stderr =", repr(r.stderr))

Python 3.14.5:

returncode = 1 stderr = '資訊: 找不到提供模式的檔案。\n'

Python 3.15.0b1:

Exception in thread Thread-2 (_readerthread):
Traceback (most recent call last):
  File "...\Lib\threading.py", line 1218, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "...\Lib\threading.py", line 1160, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\subprocess.py", line 1774, in _readerthread
    buffer.append(fh.read())
                  ~~~~~~~^^
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb8 in position 0: invalid start byte
returncode = 1 stderr = None

With a UnicodeDecodeError, the user would know that they need encoding= or errors=. With None, they don't.

This has been seen before:

Subprocess IndexError possible in _communicate #87589 / bpo-43423 Fix IndexError in subprocess _communicate function #24777 (2021) fixed an IndexError on the same lines by returning None. In the review, it was noted that this only happens when a reader thread fails (bpo-43423 Fix IndexError in subprocess _communicate function #24777 (comment)). A decode error in text mode is an easy way for the reader thread to fail.
When installing certain libraries using pip in Python 3.15.0a2, pip reports an error #141877: a user of 3.15.0a2 saw these _readerthread tracebacks when pip built numpy. It was closed for another reason.
In the PR that made UTF-8 mode the default (GH-133711: Enable UTF-8 mode by default (PEP 686) #133712), a Windows buildbot failed in test_regrtest with TypeError: expected string or bytes-like object, got 'NoneType' after the same thread traceback (GH-133711: Enable UTF-8 mode by default (PEP 686) #133712 (comment)). The test was fixed with errors="backslashreplace" in gh-133711: Fix test_regrtest for PYTHONUTF8=1 #134839.

The open PR GH-155409 (for gh-87512) changes the reader threads to read bytes, and decodes in _communicate() like the POSIX code. I ran my example with Lib/subprocess.py from that PR (without its C changes, which this path does not use), and all three calls raised UnicodeDecodeError. So that PR would also fix this on main. But it is a bigger change, and it has no test for this case.

A smaller fix: in _readerthread(), catch the exception and put it in the buffer. Then _communicate() re-raises it in the calling thread after the threads are joined. I tried this change (about 10 lines) on a copy of Lib/subprocess.py from main, with 3.14.5 and 3.15.0b1. With it, all three calls in the example raise UnicodeDecodeError, and bytes mode and the errors= handlers work as before.

I'd be happy to open a PR with this fix and a test, if this approach is acceptable. If GH-155409 is the preferred way, I could send a test for this case instead.

English is not my first language. I used an AI assistant to help write this issue and the reproduction script. The outputs above are from real runs on my machine.

CPython versions tested on:

3.12, 3.13, 3.14, 3.15

Operating systems tested on:

Windows

@kuotunyu

please assign this issue to me

I changed the Windows subprocess reader path so decode failures are surfaced to the caller instead of being lost in a background thread, and I verified it with the subprocess test file.

What I changed
I fixed the bug in the Windows implementation of Popen.communicate() in subprocess.py:1780-1859.

i will work further on it if you can assign this issue to me @kuotunyu

@kuotunyu

please assign this issue to me

I changed the Windows subprocess reader path so decode failures are surfaced to the caller instead of being lost in a background thread, and I verified it with the subprocess test file.

What I changed
I fixed the bug in the Windows implementation of Popen.communicate() in subprocess.py:1780-1859.

i will work further on it if you can assign this issue to me @kuotunyu

Hi @SuspiciousChip usually we don't assign issue to someone unless he is a project member and an expert on the issue or he is the original reporter.

@kuotunyu said in the original issue he want to work on this, so please find another issue if you want to contribute. :)

@serhiy-storchaka

CC @serhiy-storchaka as the author of PR GH-155409

@aisk

@aisk thanks for telling me about that. as of i recently started contributing in opensource can you tell me which type of issues should i pick and what to should i learn to get assign to issues.

@aisk thanks for telling me about that. as of i recently started contributing in opensource can you tell me which type of issues should i pick and what to should i learn to get assign to issues.

In CPython we usually don't assign issue to someone unless there are some specified reasons. You can just open a PR for an issue if no one else is working on it, the way to fix it is clear, and the reporter hasn't said that they're working on it.

It's usually best to wait a little while after an issue is created, though. Sometimes contributors have already finished a PR and are just waiting for an issue number to use in the NEWS entry.

Some issues are also labeled "easy", which can be a good place to start. See the dev guide: https://devguide.python.org/getting-started/fixing-issues/

@kuotunyu

Nothing links here yet and #155409 has not been updated since 2026-10-12, so unless @kuotunyu or @serhiy-storchaka is already on it, I can open a PR.

The Lib-only variant works and is backportable: _readerthread() keeps the exception in its buffer and _communicate() re-raises it in the caller after joining the reader threads, so 3.13–3.15 get the fix without the _winapi additions #155409 needs. About 10 lines plus a test that fails on unpatched main (FAILED (failures=2)) and passes with it on 3.16.0a1+dev/Windows.

If #155409 is the preferred way, I can send the missing test for that path instead — which do you prefer?

Thanks, please go ahead — I haven't started a PR. The Lib-only fix sounds good to me, but I'll leave the choice between it and #155409 to the core devs.

(English is not my first language; an AI assistant helped me write this comment.)

@serhiy-storchaka

Following the reporter's hand-off above, @serhiy-storchaka — could you say which direction you prefer before I open a PR?

Two ways to fix this:

Lib-only, about 10 lines: _readerthread() keeps the exception in its buffer, and _communicate() re-raises it in the calling thread after joining the reader threads. Nothing else changes, so it should backport to 3.13–3.15. I have this ready, with a test that fails on main on Windows for the capture_output and timeout cases (both return None today).
gh-87512: Fix ignored timeout in subprocess.run() on Windows #155409: it decodes in _communicate(), so it fixes this case too. The reporter confirmed that with your Lib/subprocess.py, without the C changes. It has no test for a decode failure.

Do you prefer the small Lib-only change on its own, or should I send only the test for #155409 and let that PR carry the fix?

Navigation Menu

Repository navigation

subprocess on Windows returns `None` instead of raising `UnicodeDecodeError` #158006

Bug description:

CPython versions tested on:

Operating systems tested on:

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	def _readerthread(self, fh, buffer):
	buffer.append(fh.read())
	fh.close()

	stdout = stdout[0] if stdout else None
	stderr = stderr[0] if stderr else None

	# Optimization: If we are not worried about timeouts, we haven't
	# started communicating, and we have one or zero pipes, using select()
	# or threads is unnecessary.
	if (timeout is None and not self._communication_started and
	[self.stdin, self.stdout, self.stderr].count(None) >= 2):
	stdout = None
	stderr = None
	if self.stdin:
	self._stdin_write(input)
	elif self.stdout:
	stdout = self.stdout.read()
	self.stdout.close()
	elif self.stderr:
	stderr = self.stderr.read()
	self.stderr.close()
	self.wait()

subprocess on Windows returns None instead of raising UnicodeDecodeError #158006

Description

Bug description:

CPython versions tested on:

Operating systems tested on:

Activity

SuspiciousChip commented on Sep 23, 2026

aisk commented on Sep 24, 2026

aisk commented on Sep 24, 2026

SuspiciousChip commented on Sep 24, 2026

aisk commented on Sep 24, 2026

summeriver13 commented on Oct 25, 2026

kuotunyu commented on Oct 26, 2026

summeriver13 commented on Oct 26, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

subprocess on Windows returns `None` instead of raising `UnicodeDecodeError` #158006