Typing and public API

pytest's "official" public API is exported by the pytest package, and everything else is defined in the _pytest package.

Currently, pytest only contains APIs that users need to use directly: functions to be called, decorators, classes to be instantiated or subclasssed, globals. Other objects, which the user doesn't interact with directly, are not exported.

Typing extends the range of cases in which some object needs to be referred to by name. As long as the user can somehow reach an object, directly or indirectly, through some public API, they need to be able to refer to its type by name, due to type annotations.

To come up with a list of reachable but unexported types, I used the following categories:

Fixtures: built-in fixtures that are dependency-injected. To benefit from the types the user needs to type annotate them, for example:
```
def test_it(monkeypatch: MonkeyPatch) -> None:
    ...
```
Hooks: the user needs to be able to type annotate the arguments and return values of any hook.
Transitive: types that are referred to transitively by some other public API, e.g. return value of a function, method, context manager, public attribute etc.

See below for the list I came up with.

Questions for discussion:

Is this a problem we want to fix?
Most of these types are classes, that are not intended to be instantiated and/or subclassed by users, only internally.

2.1. Do we want to enforce this?
2.2. How do we enforce this?
How do we want to officialy declare something as public API?
Which types from the list below are we actually ready to export?
Is there a category or specific types I missed?

I'll add my own thoughts on these questions in a separate reply.

Fixtures:

Hooks:

API:

_pytest.mark.structures.ParameterSet

Transitive:

Here are my thoughts on the questions:

Is this a problem we want to fix?

Yes; otherwise users can't utilize types in these cases. Also, if we don't, people will be importing from _pytest anyway, which we don't want to happen.

More generally, I think it will be really good for us to clearly delineate what we consider public API and what is private. Currently it is somewhat murky which makes internal refactorings difficult.

2.1. Do we want to enforce this?

I think we want to enforce it, because otherwise instantiation and subclassing are implicitly assumed to be allowed, and then we are locked into supporting them even when we don't want to.

2.2. How do we enforce this?

My current intention is to steal trio's approach and see how it goes.

How do we want to officialy declare something as public API?

I think there should be three criteria:

It is exported in pytest.
It doesn't have a _ or __ prefix.
It is documented in the reference.

By "it" I mean both the types themselves, and their contents (methods, attributes, etc.).

Some work will be required to align reality with these requirements...

Which types from the list below are we actually ready to export?

The fixtures in the list look good to export, except maybe _pytest.fixtures.FixtureRequest / _pytest.fixtures.SubRequest which I'm not super clear about myself.

Stuff in the hooks needs to be exported; some require care however, and we might decide to leave unexported in the first pass.

Similar for the "Transitive" ones.

I think we can discuss the specifics in future PRs.

Is there a category or specific types I missed?

That's for others to answer :)

@bluetech

Thanks @bluetech for writing up this issue with so much details. 👍

My thoughts:

Most of these types are classes, that are not intended to be instantiated and/or subclassed by users, only internally.

2.1. Do we want to enforce this?
2.2. How do we enforce this?

I think we want to enforce it, because otherwise instantiation and subclassing are implicitly assumed to be allowed, and then we are locked into supporting them even when we don't want to.

I agree that we don't want to allow instantiation and subclassing, specially because we might want to change __init__ methods later to include additional parameters to support new features.

An idea would be to implement declare ABCs, and only export the ABCs. This I believe would remove the need to export the actual type, and hence the __init__ method. It would also be a double-checking of the public API, because changing only the implementation would upset the type-checker and bring to light unintentional changes in the API.

The downside is that we would need to implement ABCs for all the objects you listed here, which is a bunch of work.

My current intention is to steal trio's approach and see how it goes.

This looks very similar to the work done by @RonnyPfannschmidt in the Node.from_parent endeavor.

By "it" I mean both the types themselves, and their contents (methods, attributes, etc.).

Those criteria look good to me!

An idea would be to implement declare ABCs, and only export the ABCs

Doing this would be nice in that it would really highlight public vs. private. However I don't think we want to go this way for these reasons:

It doesn't prevent subclassing; in fact an ABC probably encourages it.
It requires a lot of duplication, scattered code and funny-looking "Impl" classes. I fear our code will look more like Java than Python :)
I'm not sure it's very practical because to gain the benefit we would need to to change all arguments/return values to the ABCs, but often our internal code uses public APIs and then would have to cast or such in order to access the private fields.
Not very newbie friendly.

However I don't think we want to go this way for these reasons

All good points, I had a hunch it was a lot of work but hand't thought through the details, but you confirmed it. 😁

So I agree that ABCs is a no-go.

How about having a export module for types which has an explicitly spec that anything not exported somewhere else is only intended for type declaration and won't have public constructors for now

An idea would be to implement declare ABCs, and only export the protocol definition. https://www.python.org/dev/peps/pep-0544/

What about using protocols instead of ABCs? And we only expose the protocols publicly 🤔 That could even be separated into a typing module:

from pytest.typing import MonkeyPatch, TempPathFactory

This way the fixtures have a very explicit public API, however the implementation does not have to inherit from it. Having the typing namespace makes it explicit that it's not meant for instiation. E.g.:

# pytest/typing.py

class Notset:
    def __repr__(self) -> str:
        return "<notset>"


notset = Notset()

K = TypeVar("K")
V = TypeVar("V")


class MonkeyPatch(Protocol):
    @contextmanager
    def context(self) -> Generator["MonkeyPatch", None, None]:
        ...

    @overload
    def setattr(self, target: str, name: object, value: Notset = ..., raising: bool = ...) -> None:
        ...

    @overload
    def setattr(self, target: object, name: str, value: object, raising: bool = ...) -> None:
        ...

    def setattr(
        self, target: Union[str, object], name: Union[object, str], value: object = ..., raising: bool = ...,
    ) -> None:
        ...

    def delattr(self, target: Union[object, str], name: Union[str, Notset] = notset, raising: bool = ...,) -> None:
        ...

    def setitem(self, dic: MutableMapping[K, V], name: K, value: V) -> None:
        ...

    def delitem(self, dic: MutableMapping[K, V], name: K, raising: bool = ...) -> None:
        ...

I like the idea of protocols as a public api, it gives us a little bit more flexibility on the implementation

the tricky parts become though:

(1) how to ensure we stay in line with the protocols (typing tests?)
(2) if we use the protocols internally, we'd probably have to cast to the concrete types occasionally to use the internal methods? this feels like it defeats some of the purposes of typing in the first place 🤔

I don't think you need to cast protocols, they should be replaceable automatically. You can test it by merging the protocol and implementation via retype into a temporary folder 🤔and running type check on that 🤔

@gaborbernat

@gaborbernat the problem would be if we internally defined things like:

def internal_method(x: ProtocolType) -> ...
    x.other_internal_method()

that would not type check because other_internal_method is not on the public protocol

So we just internally always type as implementation? 🤷‍♂️

The fixtures in the list look good to export, except maybe _pytest.fixtures.FixtureRequest / _pytest.fixtures.SubRequest which I'm not super clear about myself.

I believe _pytest.fixtures.SubRequest is what I would like to import for #3342 (comment):

import pytest


@pytest.fixture(params=["a"])
def example_fixture(request) -> str:  # << type hint for request on this line
    return request.param

@asottile

@asottile

(1) how to ensure we stay in line with the protocols (typing tests?)

@altendky has come up with a neat trick with a decorator for each protocol

this works today

from typing import *
import attr
class Eggs: pass
class HamProtocol(Protocol):
    spam: Eggs

HamProtocolT = TypeVar("HamProtocolT", bound=HamProtocol)

def implements_ham(cls: Type[HamProtocolT]) -> Type[HamProtocolT]:
    return cls


@implements_ham
@attr.s
class ConcreteHam:
    spam: Eggs

It's a bit of boilerplate so would be quite nice, and I assume a few dozen lines of code to do, as a mypy plugin:

from protocol_check import implements


@implements(HamProtocol)
@attr.frozen
class ConcreteHam:
    spam: Eggs

Merge #128

Merge #35 #36

Merge #31

Another public API which is currently not annotated at all: pytest.approx. With all the different values it accepts, I guess it won't be easy to annotate, but I suppose it might still be possible to have something based on a TypeVar or so?

chore(deps): update dependency pytest to v7 (#3)

Export `FixtureDef`

@adamjstewart

@adamjstewart That's already public as pytest.skip.Exception, though I'm not sure if that's documented anywhere.

Document exceptions raised by exit, skip, xfail, fail, and importorskip

Document exceptions raised by exit, skip, xfail, fail, and importorsk…

Another public API which is currently not annotated at all: pytest.approx. With all the different values it accepts, I guess it won't be easy to annotate, but I suppose it might still be possible to have something based on a TypeVar or so?

I ran into this when I tried to enable strict typechecking in a project of mine where I use VSCode and Pylance. It complains about expected being of unknown type. Would annotating object work as a general type allowing usage in strict mode?

I tried to make a partial stub for approx in my project, but because pytest has a py.typed pylance won't apply partial stubs.

I'm finding type checkers don't really understand FixtureRequest.node, as in the following code:

from __future__ import annotations                      # 1
import pytest                                           # 2
                                                        # 3
def test_foo(request: pytest.FixtureRequest) -> None:   # 4
    reveal_type(request.node)                           # 5
    node = request.node                                 # 6
    reveal_type(node)                                   # 7

Pyright's output:

/private/tmp/scrap.py
  /private/tmp/scrap.py:5:17 - information: Type of "request.node" is "Unknown"
  /private/tmp/scrap.py:7:17 - information: Type of "node" is "Unknown"
0 errors, 0 warnings, 2 informations

Mypy (--strict) handles this less badly, but still not well:

/tmp/scrap.py:5: note: Revealed type is "Any"
/tmp/scrap.py:7: note: Revealed type is "Any"

Would it be reasonable to change the definition as follows?

Current:

    @property
    @abc.abstractmethod
    def node(self):
        """Underlying collection node (depends on current request scope)."""
        raise NotImplementedError()

Proposed:

    @property
    @abc.abstractmethod
    def node(self) -> nodes.Node:
#                 ━━━━━━━━━━━━━━
        """Underlying collection node (depends on current request scope)."""
        raise NotImplementedError()

Make PyCollector an implementation detail - don't use in hook type an…

Navigation Menu

Repository navigation

Typing and public API #7469

53 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Typing and public API #7469

Description

Activity

bluetech commented on Jul 9, 2020

nicoddemus commented on Jul 9, 2020

bluetech commented on Jul 10, 2020

nicoddemus commented on Jul 10, 2020

RonnyPfannschmidt commented on Jul 10, 2020

gaborbernat commented on Jul 21, 2020

asottile commented on Jul 24, 2020

gaborbernat commented on Jul 24, 2020

asottile commented on Jul 24, 2020

gaborbernat commented on Jul 24, 2020

adamtheturtle commented on Jul 29, 2020

graingert commented on Aug 28, 2020

53 remaining items

The-Compiler commented on Mar 31, 2023

adamjstewart commented on May 4, 2024

The-Compiler commented on May 4, 2024

thomthom commented on Oct 25, 2024

finite-state-machine commented on Jan 5, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions