Short paths in Windows seem to fail to be collected

This is an initial report of an issue I observed when updating xdoctest to support pytest 8.0.

What I found is that when I ran a test effectivly using the invocation:

C:\hostedtoolcache\windows\Python\3.12.1\x64\python.exe -m pytest C:\Users\RUNNER~1\AppData\Local\Temp\tmpcttcs8zz

Pytest would collect 0 tests even though there was a python file with a test in that directory.

By adding this code to convert the short path to a long path, the xdoctest failure went away, and things seem to be working now.

dpath = tempfile.mkdtemp()
# https://stackoverflow.com/questions/11420689/how-to-get-long-file-system-path-from-python-on-windows
from ctypes import create_unicode_buffer, windll
BUFFER_SIZE = 500
buffer = create_unicode_buffer(BUFFER_SIZE)
get_long_path_name = windll.kernel32.GetLongPathNameW
get_long_path_name(dpath, buffer, BUFFER_SIZE)
dpath = buffer.value

I've documented more of the problem here Erotemic/xdoctest#151

I don't have a MWE as I don't have a windows machine, but I'm reasonably confident that something in pytest 8 broke short path recognition on windows. I'll leave it to other devs to test further.

Thanks for the report.

I don't have a MWE as I don't have a windows machine, but I'm reasonably confident that something in pytest 8 broke short path recognition on windows. I'll leave it to other devs to test further.

In order to narrow down on the issue, we'll probably need to bisect. I don't have windows either, but I can try with Wine. Can you provide instructions for reproducing the CI failure locally? (Doesn't have to be minimal).

I think this should work to reproduce it. On a window machine:

clone the latest version of xdoctest: https://github.com/Erotemic/xdoctest/tree/v1.1.3
pip install -e . to install it in development mode
Apply the following patch to remove the workaround I added to convert short paths into long paths:

diff --git a/src/xdoctest/utils/util_path.py b/src/xdoctest/utils/util_path.py
index d49b371..83a0230 100644
--- a/src/xdoctest/utils/util_path.py
+++ b/src/xdoctest/utils/util_path.py
@@ -34,19 +34,8 @@ class TempDir(object):
 
     def ensure(self):
         import tempfile
-        import sys
         if not self.dpath:
             dpath = tempfile.mkdtemp()
-            if sys.platform.startswith('win32'):
-                # Force a long path
-                # References:
-                # https://stackoverflow.com/questions/11420689/how-to-get-long-file-system-path-from-python-on-windows
-                from ctypes import create_unicode_buffer, windll
-                BUFFER_SIZE = 500
-                buffer = create_unicode_buffer(BUFFER_SIZE)
-                get_long_path_name = windll.kernel32.GetLongPathNameW
-                get_long_path_name(dpath, buffer, BUFFER_SIZE)
-                dpath = buffer.value
             self.dpath = dpath
         return self.dpath

install the version of pytest you want to test
Run the tests that cause the error (relative to the xdoctest repo root):

pytest tests/test_pytest_cli.py -k test_simple_pytest_import_error_cli -s
pytest tests/test_pytest_cli.py -k test_simple_pytest_cli -s

You can likely make this much more minimal by using tempfile.mkdtemp() to make a directory, record the path that it returns (on the actions CLI it was a short path, so I expect it will be the same on a different machine). Then write a small test file that pytest should pickup to that directory and then run pytest <path> to that directory. That is effectively what the above xdoctest instructions are doing.

You can likely make this much more minimal by using tempfile.mkdtemp() to make a directory, record the path that it returns (on the actions CLI it was a short path, so I expect it will be the same on a different machine). Then write a small test file that pytest should pickup to that directory and then run pytest to that directory. That is effectively what the above xdoctest instructions are doing.

I did not investigate into details, but I cannot reproduce the behavior using a very short path:

λ pwd
W:\

λ cat a\test_foo.py
def test(): pass

λ pytest w:\a
======================== test session starts ========================
platform win32 -- Python 3.10.9, pytest-8.0.0, pluggy-1.4.0
rootdir: w:\a
collected 1 item

a\test_foo.py .                                                [100%]

========================= 1 passed in 0.00s =========================

λ pytest a
======================== test session starts ========================
platform win32 -- Python 3.10.9, pytest-8.0.0, pluggy-1.4.0
rootdir: W:\a
collected 1 item

a\test_foo.py .                                                [100%]

========================= 1 passed in 0.00s =========================

I'm afraid we will need more details to reproduce the issue.

There's a misunderstanding on what I mean by "short path". It's not a path with a short length, it's a windows shorthand to refer to a real path that's longer. It's like an autocomlete. E.g.

c:\PROGRA~2\Android\ANDROI~1 resolves to c:\Program Files (x86)\Android\android-sdk

Reference: https://superuser.com/questions/348079/how-can-i-find-the-short-path-of-a-windows-directory-file

It's a silly feature, but it seems tempfile.mkdtemp is able to return one, so it probably makes sense to support it.

Ahh of course, brain fart on my part.

I can reproduce the issue:

(.env310) λ dir c:\PORTAB~1\test\ /b
test_foo.py
__pycache__

>>> from pathlib import Path
>>> p=Path(r'c:\PORTAB~1\test')
>>> p.is_dir()
True
>>> (p / "test_foo.py").is_file()
True
>>>

λ pytest c:\PORTAB~1\test\
======================== test session starts ========================
platform win32 -- Python 3.10.10, pytest-8.1.0.dev108+gca3790102, pluggy-1.4.0
rootdir: e:\projects\pytest
plugins: hypothesis-6.84.0, replay-1.4.0
collected 0 items

======================= no tests ran in 0.02s =======================
ERROR: not found: c:\PORTAB~1\test
(no match in any of [<Dir >])

I will investigate this when I find some time, thanks for the report!

I'm surprised this ever worked. Windows paths - the gift that keeps on giving :)

Add test for pytest-dev#11895

Fix collection of short paths on Windows

Add test for pytest-dev#11895

Fix collection of short paths on Windows

Add test for pytest-dev#11895

Fix collection of short paths on Windows

Fix collection of short paths on Windows (#11936)

Fix collection of short paths on Windows (pytest-dev#11936)

Use latest `pytest` and `xdoctest` (#1632)

Navigation Menu

Repository navigation

Short paths in Windows seem to fail to be collected #11895

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Short paths in Windows seem to fail to be collected #11895

Description

Activity

bluetech commented on Jan 31, 2024

Erotemic commented on Jan 31, 2024

nicoddemus commented on Feb 1, 2024

Erotemic commented on Feb 1, 2024

nicoddemus commented on Feb 1, 2024

bluetech commented on Feb 1, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions