Collection is quadratic in the number of file arguments from one directory: each file argument re-collects its parent directory #15187

Description

When several test files from the same directory are passed as separate command-line arguments, Session.collect re-collects the parent directory once per argument. Each re-collection lists the directory again and offers every entry to pytest_ignore_collect / pytest_collect_file again. Cost is O(files given x entries in the directory).

Passing the directory instead, or the same files through a single directory argument, does not have the problem.

We hit this in CI, where a job passes ~100 explicit test files from a directory with ~940 entries: each xdist worker spent 10-16 s of its 22-33 s collection re-walking that one directory. Keeping the cache for directory nodes (below) gave identical collected node IDs and roughly halved collection time.

Minimal reproduction

mkdir tests
cat > tests/conftest.py <<'EOF'
import collections
calls = collections.Counter()
def pytest_collect_file(file_path, parent):
    calls[file_path.name] += 1
def pytest_collection_finish(session):
    print(f"\npytest_collect_file calls: {sum(calls.values())} "
          f"(test_0000.py offered {calls['test_0000.py']} times)")
EOF
for i in $(seq -f '%04g' 0 299); do printf 'def test_%s():\n    pass\n' "$i" > "tests/test_$i.py"; done

python -m pytest --collect-only -q -p no:cacheprovider tests            # directory argument
python -m pytest --collect-only -q -p no:cacheprovider tests/test_*.py  # 300 file arguments

Output (pytest 9.1.1, CPython 3.12, macOS; same on Linux with 9.0.3):

-- directory argument
pytest_collect_file calls: 301 (test_0000.py offered 1 times)
300 tests collected in 0.12s
-- 300 file arguments
pytest_collect_file calls: 90300 (test_0000.py offered 300 times)
300 tests collected in 3.97s

Same 300 tests both ways; 300x the hook calls and ~33x the time with file arguments.

Cause

src/_pytest/main.py, Session.collect (unchanged from 8.x through 9.1.1 and current main):

                    # For backward compat, files given directly multiple
                    # times on the command line should not be deduplicated.
                    handle_dupes = not (
                        len(matchparts) == 1
                        and isinstance(matchparts[0], Path)
                        and matchparts[0].is_file()
                    )
                    rep, duplicate = self._collect_one_node(matchnode, handle_dupes)

When the only remaining part is a file, matchnode is that file's PARENT Directory/Package, so handle_dupes=False bypasses _collection_cache for the directory, not for the file. The intent (a file named twice runs twice) only needs the FILE not to be deduplicated, which genitems already ensures (handle_dupes = not isinstance(node, nodes.File)).

Possible fix

Use the cache for directory nodes in that branch and leave dedup behaviour for files alone, e.g.:

handle_dupes = (
    isinstance(matchnode, nodes.Directory) and not self.config.option.keepduplicates
) or not (
    len(matchparts) == 1 and isinstance(matchparts[0], Path) and matchparts[0].is_file()
)

One caveat we hit: under --keep-duplicates, a file named twice needs two separate Module nodes (two setup_module/teardown_module cycles), which today only the directory re-collection provides. So the directory cache has to stay off in that mode, or the duplicate handling has to move to the file node. Without --keep-duplicates, duplicate paths are already removed during argument normalisation, so nothing depends on the re-walk.

We run this as a monkeypatch of Session._collect_one_node (force handle_dupes=True for nodes.Directory). Across ~17k tests in ~480 files, with files repeated and node-ID arguments mixed in, the collected node IDs are identical with and without it. It would also make the #14635 reconciliation unnecessary for this path, since the directory is no longer re-collected. Happy to open a PR with a test if this direction is acceptable.

Versions

• pytest 9.0.3 and 9.1.1 (code identical on main as of 2026-10-15)
• CPython 3.12
• Linux (GitHub Actions, ubuntu-latest) and macOS

This report was drafted with help from an AI coding agent; the repro and numbers were run and checked by us.