conftest.py: support importorskip at import, make every other outcome exception a usage error #15142

🤖 Written by Claude Opus 5.5 via Claude Code for the pytest maintainers; I prompted it, it did the work, I read it.

Problem

A conftest.py is plugin code, not a collected module. Today an outcome exception raised while importing it does something different depending on how pytest was invoked:

(checked on main @ e84e063)

Why:

• PytestPluginManager._importconftest wraps except Exception into ConftestImportFailure. OutcomeException derives from BaseException, so Skipped/Failed/XFailed slip past, while unittest.SkipTest (an Exception) is reported as an error.
• During collection the conftest happens to be imported inside the Dir collector's CallInfo (pytest_make_collect_report in runner.py), so an escaped Skipped becomes a skip of the directory.
• Initial conftests (_set_initial_conftests, for command-line args) have no collector around them, so the same exception crashes pytest.
• The allow_module_level check exists only in python.importtestmodule, so a plain pytest.skip("x") in a conftest silently skips a whole package.

Real-world usage

Scan of 1648 conftest.py files in 177 popular repos:

• Import-time outcome exceptions in conftests: 10 files in 6 repos (pandas, fsspec, kombu, ibis, salt, gunicorn). All of them are pytest.importorskip; there was no top-level pytest.skip/pytest.fail/raise SkipTest in any conftest. Typical: pa = pytest.importorskip("pyarrow") in pandas/tests/arrays/arrow/conftest.py, gating the directory and binding the module for the fixtures below in one line.
• Alternatives in use are all worse for this job: conditional collect_ignore/collect_ignore_glob (14 repos, silent, a try/except per dependency, does not bind the module), pytest_ignore_collect (7 repos, silent), skip markers added from pytest_collection_modifyitems/pytest_runtest_setup (~12 repos, verbose, does not protect the conftest's own imports).

So importorskip in a conftest is a legitimate, useful idiom; everything else is an accident.

Proposal

1. Support pytest.importorskip at conftest import. It skips the conftest's directory, consistently: also when the conftest is an initial conftest (command-line path or node id), which fixes Skipped raised by a conftest of a command-line path crashes pytest instead of skipping #15141. A node id below the skipped directory reports the skip instead of ERROR: found no collectors.
2. Every other outcome exception at conftest import is a usage error naming the conftest: pytest.skip() (with or without allow_module_level), pytest.fail(), pytest.xfail(). This resolves pytest.skip in conftest/__init__ files doesn't work as expected #12371.
   • pytest.skip() currently "works" during collection, so it gets a PytestRemovedIn10Warning (keeping today's directory skip) in 9.x and becomes the usage error in 10. fail/xfail already error during collection and crash on initial conftests, so they become the usage error right away.
3. --import-mode=importlib: remove the conftest from sys.modules when its import raises, so a retry never sees a half-executed module (Skipped raised by a conftest of a command-line path crashes pytest instead of skipping #15141, point 3).

Implementation note: importorskip raises a plain Skipped today, so it needs to be told apart, e.g. a private Skipped subclass or flag set only by importorskip (including its minversion skip).

Not affected: pytest.exit() (Exit is not an OutcomeException), outcome exceptions raised from conftest hooks or fixtures at runtime (node-scoped, already well-defined, see test_skip_file_by_conftest), and skips in a package __init__.py (a collected module, governed by allow_module_level).

Docs

Document importorskip in conftest.py as the supported way to gate a directory on an optional dependency, and state that no other skip/fail is allowed there.

Supersedes #12371 and #15141 (both closed in favour of this). Background: #4479.