Fixtures without name matching magic

I've been happily using pytest for several projects for the past few years.

There's one part about pytest that I still struggle to get behind: The way that fixtures magically match argument names to fixtures -- and apparently I'm not alone in this feeling. I would much rather declare dependencies explicitly in some way using code. I know this would be more verbose, but that's a tradeoff I'm happy to make.

Is it possible to do this in some way with pytest today? If not, would you be open to adding an optional feature for this?

I was thinking perhaps something like the following, using an example adapted from the docs:

import pytest

@pytest.fixture
def smtp_connection():
    import smtplib
    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)

@pytest.use_fixture(smtp_connection)
def test_ehlo(connection):
    response, msg = connection.ehlo()
    assert response == 250
    assert 0 # for demo purposes

@shoyer

Hi @shoyer,

If you don't like the way fixtures are automatically injected, you can be more explicit in a way very similar to the example you provided:

import pytest

@pytest.fixture
def smtp_connection():
    import smtplib
    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)

@pytest.mark.usefixtures("stmp_connection")
def test_ehlo(smtp_connection):
    response, msg = connection.ehlo()
    assert response == 250
    assert 0 # for demo purposes

@nicoddemus

@nicoddemus that's explicit about using a fixture, but it's still using a string to refer to a name variable. Is it possible to do something like that while referencing smtp_connection as a Python object -- preferably with a different, non-conflicting argument name?

Probably you can do something like this:

import pytest

def use_fixture(fixfunc):
    def inner(f):
        return pytest.mark.usefixtures(fixfunc.__name__)(f)
    return inner

@pytest.fixture
def fix():
    return 1

@use_fixture(fix)
def test(fix):
    assert fix == 1

But you still need to use the same name in the argument as the fixture, plus you will need to explicitly import the fixture function into the test module in case it is not defined there, and that is not really recommended because it messes with the internal caching of the fixtures.

that's explicit about using a fixture, but it's still using a string to refer to a name variable.

What is the problem with that in your opinion? That's very explicit IMHO: @pytest.mark.usefixtures("fix") explicitly declares that the test function uses a fixture "fix", and that is passed as argument name.

preferably with a different, non-conflicting argument name?

Not sure what is the reason for that, can you explain?

But you still need to use the same name in the argument as the fixture, plus you will need to explicitly import the fixture function into the test module in case it is not defined there, and that is not really recommended because it messes with the internal caching of the fixtures.

Indeed, but also using fixfunc.__name__ for lookup is a good example of exactly what I was trying to avoid here.

What is the problem with that in your opinion? That's very explicit IMHO: @pytest.mark.usefixtures("fix") explicitly declares that the test function uses a fixture "fix", and that is passed as argument name.

I want to use Python's normal rules for resolving symbols (i.e., standard variable lookup), and I want to keep introspection to the bare minimum needed to run tests. I know pytest uses magic to rewrite assert and decide which tests to run, but in general I like my Python code (even tests) to follow the normal rules of Python code.

Referencing functions by string name instead of object breaks normal code introspection tools (e.g., lint), which can't recognize misspelled names if there's no variable reference. So if I write stmp_connection instead of smtp_connection, nothing catches that mistake until I actually run my tests.

(In my opinion, one of the best things about Python is that dependencies are explicit, unless you use the discouraged import *.)

preferably with a different, non-conflicting argument name?

Not sure what is the reason for that, can you explain?

As one simple example, if you run this example through pylint, you get the following warning:

W0621: Redefining name 'smtp_connection' from outer scope (line 7) (redefined-outer-name)

At Google, we actually strictly enforce running pylint on all Python code. If I wanted to disable this message, I would need to write a comment # pylint: disable=redefined-outer-name.

Now, pylint is known for being overly-aggressive in some cases, but I do think that in general it's best not mask variable names from outer scopes. There's basically no reason why this is ever necessary if not using pytest, and if done unintentionally, this can easily lead to bugs.

Another example of what doing this explicitly could look like, with keyword arguments:

@pytest.mark.usefixtures(conn=smtp_connection)
def test_ehlo(conn):
    ...

Using a fixture imported from another module also becomes very straightforward, e.g.,

@pytest.mark.usefixtures(conn=other_module.smtp_connection)
def test_ehlo(conn):
    ...

@shoyer

Hi @shoyer sorry for taking so long to get back at this.

I'm not sure it has a place in the core though, as it would lead to a different way of doing things, and is solving not really an existing problem but a personal preference which is shared by some people but not by the majority of pytest users (at least that's what I think, no hard data here).

But all you are proposing is probably possible to write as a plugin. I suggest you give this a try (be sure to post here if you need help) and see how that feels in real code.

What do you think?

personally i have been thinking about such a mechanism as well, i just dont have the time to work on even a base for it

Let me add to this that the name-matching does lead to real-world problems: I just spent an hour trying to get my pytest-sanic tests to working, not realising that it's test_client fixture clashes with the test_client fixture of pytest-aiohttp.

@shoyer

This would be really nice. I hate dropping # pylint: disable=redefined-outer-name lines to my test files because pylint will nag about every test with a fixture otherwise.

The original suggestion by @shoyer looks good enough, imo. Anyway, here are some extra thoughts:

# not supported by python(?):

def test_ehlo(@pytest.inject(smtp_connection) connection):
    ...

# dunno if this would be doable, either:

@pytest.fixture
class SmtpConnection:
    def __call__(self):
        import smtplib
        return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)

@pytest.inject
def test_ehlo(conn: SmtpConnection):
    ...

This would be really nice. I hate dropping # pylint: disable=redefined-outer-name lines to my test files because pylint will nag about every test with a fixture otherwise.

Just to mention that fixtures have a name parameter to avoid this problem:

@pytest.fixture(name='smtp_connection')
def smtp_connection_():
    ...

def test_ehlo(smtp_connection):
    ...

Oh, that's a workaround I could use. Good to know.

Though the benefit in being able to do something like what I suggested above would be that when you (have to) import fixtures, they are then actually referenced in the tests file, and your import will not get marked as unused (and removed if you auto-organize imports). Sure I could:

from foo import my_fixture

PYTEST_IMPORTS = my_fixture

But then PYTEST_IMPORTS will be marked as unused :P (or would importing even work together with name=?)

@shoyer

I'll register my support for this issue as well. When users first see tests written in pytest, it is mystifying how the arguments to these functions get populated. At least having the option to be explicit about it in a robust way would be nice.

Another possible benefit is the ability to give a different name to the same fixture in different tests. Or maybe more specifically, the ability to give the same name to different fixtures in different tests. Not sure if this is already possible, but with an explicit mechanism like the one @shoyer is talking about, it could look something like this:

Let's say we have these fixtures for an application that processes log files (this is a barebones example just to illustrate the point):

@pytest.fixture
def simple_logfile():
    return ("2019-08-16 10:35:05 connection established\n"
             "2019-08-16 11:07:16 connection dropped\n")

@pytest.fixture
def empty_logfile():
    return ""

With the current behavior, we could write tests like this:

def test_read_logfile(simple_logfile):
    result = read_logfile(simple_logfile)
    assert result == True

def test_read_empty_logfile(empty_logfile):
    result = read_logfile(empty_logfile)
    assert result == True

In the proposed behavior, it could look like this:

@pytest.mark.usefixtures(logfile=simple_logfile)
def test_read_logfile(logfile):
    result = read_logfile(logfile)
    assert result == True

@pytest.mark.usefixtures(logfile=empty_logfile)
def test_read_empty_logfile(logfile):
    result = read_logfile(logfile)
    assert result == True

The explicit approach allows the generic aspects of the test code to remain generic -- in this example, the test only cares about reading a logfile and asserting that it works. The fact that we have selected this file to be an empty one to test an edge case is already captured in the name of the function, and avoiding the need for different variable names seems desirable since the code is the same.

In fact, doing it this way gives us yet another advantage -- the possibility of parametrizing over fixtures using pytest.mark.parametrize. For instance, the proposed behavior could now look like this:

@pytest.mark.parametrize(
    "logfile", (simple_logfile, empty_logfile),
)
def test_read_logfile(logfile):
    result = read_logfile(logfile)
    assert result == True

... which covers both tests in a single parameterized test.

It looks like there is currently already a way to parametrize fixtures themselves. That's a great feature as well, but I think the ability to parameterize the test rather than the fixture is a different thing which could complement the current functionality and lend additional flexibility.

can you provide a open draft pr for reeference - it may enable us to put focus on the bugs to enable the rest

I think this was my WIP branch: https://github.com/adriangb/di/tree/pytest. But given that it was last touched 3 years ago I don't know the state of it. To be clear I'm not saying there were bugs in pytest, more so that I just didn't understand how things were supposed to work internally, especially when it came to the async plugins.

I have thrown together a quick plugin that allows fixtures to be resolved using Annotated[...] rather than their full name.

e.g.

from typing import Annotated
from pytest_annotated import Fixture

class SomeThing:
    pass

@pytest.fixture
async def somefixture(someotherfixture) -> SomeThing:
    return SomeThing()

async def test_thing(st: Annotated[SomeThing, Fixture(somefixture)]):
    assert isinstance(st, SomeThing)

you can find it here: https://github.com/tristan/pytest-annotated and here: https://pypi.org/project/pytest-annotated/

Unfortunately, I feel it requires the use of too many _private parts of pytest, so I'm a bit scared of the maintenance burden it will require to keep it up to date. Would be super happy to have someone more familiar with the internals of pytest to make suggestions for making it less brittle .

that one is a funky little hack, we ought too figure if we can enable lookup by name, type and annotated type in future

https://pypi.org/project/pytest-unmagic/ is a pytest plugin that implements fixtures with normal import semantics.

Fixtures are applied to tests or other fixtures with a @use decorator. Alternately, a fixture can be called without arguments to get its value inside a test or other fixture function.

@millerdev

@millerdev at first glance pytest-unmagic is a complete deception - using magic globals unrelated to the actual fixtures instead of actual values passed - i strongly discourage anyone from using it - it does the opposite of its name

@RonnyPfannschmidt

@RonnyPfannschmidt Fixtures are global objects. Pytest registers them in an opaque hierarchical namespace, and effectively does a global lookup each time a fixture is requested. The magic that this plugin eliminates is function-scoped local names magically invoking global fixtures. Granted, the active request reference is not the nicest feature. Hopefully we can work together to improve it?

The active request reference allows unmagic.fixture to work with pytest features such as --setup-show and --fixtures. While implemented as a module-level singleton by default, it does provide a mechanism for thread-local or other custom scoping. In most cases this will be unnecessary.

I have a development branch where the global active request reference has been removed, but it has trade-offs. The implementation is more complicated, and it is impossible to use fixtures in the context of unittest.TestCase. I didn't release it (yet) because the test suite that pytest-unmagic was originally intended for uses unittest-style tests extensively, and allowing fixtures to be used there is a big win.

For now—in the very spirit of pytest fixtures—practicality beats purity.

This explanation seems to discard important distinction between concepts like registries, discovery, references,actions at a distance and some more

As far as I'm concerned it's much worse than what we have right now

I'm happy to work towards more explicit control of the registries as well as sane lookup

But injection of dependencies at a distance instead of parameters is absolutely off the table, that's completely unacceptable

@RonnyPfannschmidt

My 2 cents on the plugin -- I agree with @RonnyPfannschmidt overall about the design, however I'm also happy that @millerdev did an experiment to try things out (it is his project after all!).

But regardless I wanted to share my opinions on some points (I'm not speaking for all maintainers or for the pytest project at all, but only how I see this topic):

The existence of an external plugin that does "fixtures without name matching" does not automatically imply endorsement from the pytest project, much less that it will be incorporated at some point.
I think pytest should never stray from the current fixture mechanism, or even support both, even if there are people clamoring for that to be the case -- besides being an controversial topic¹, there's backward compatibility, and having to teach/learn multiple ways of doing the same thing.
I do think, however, that we should improve pytest so plugins can change how fixtures are handled, allowing plugins to implement whatever mechanism one prefers over the current name matching in core pytest.

I think the current fixture mechanism in pytest is absolutely fine, IMO lots of frameworks implement things that seem like "magic" at first glance, for example dataclasses "magically" injecting methods, SQL/Django columns "magically" talking to a DB based on how attributes are declared... if you squint hard enough many framework features can be shrugged off (or even bad mouthed) as magic. ↩

@nicoddemus

@nicoddemus: Clarke's third law states "any sufficiently advanced technology is indistinguishable from magic". That is not at all a bad thing. It just means that as soon as a mechanism is not obvious how it works, it appears as if it were magic. Once you understand how things work, its not magic anymore, but just a great power to wield. And with great power, comes great responsibility.

IMHO, what is wrong with the current fixture mechanism is the single global namespace shared among all participants in a given test session. Thus, this particular great power is hard to guide to its intended target. Rather than a surgical laser beam, these fixtures behave like a virus; once you unleash them, they go everywhere, hitting both intended and unintended targets. It appears magic to the many users, because once that virus arrives at your site (intended or not), you don't see where it comes from. This applies both to humans and to tools (type checkers). More importantly, the design makes it impossible to reason about the area of effect within any subscope, because you always need to know the full testing session before you can start figuring out where fixtures may come from or go to.

This is very different from the magic methods in dataclasses or the various forms of automatic schema validation and column creation in ORMs. Its true that it can appear equally magic to someone who doesn't understand the mechanism, but once you do understand, a) the rules are simple (as in the current pytest fixtures) and b) the amount of inputs you need to consider to apply them are reasonably bounded (unlike in the current pytest fixtures).

@burnpanck

Thanks @burnpanck for the detailed reasoning, you make good points.

Regardless, all my previous points still stand: backward compatibility, more than one way to do the same thing, etc, and those do not change regardless of the discussion about what constitutes a magical behavior or not.

But again, this is all from my POV, I don't know if other core maintainers feel otherwise and think it is OK to add an additional way to use fixtures to core pytest.

@RonnyPfannschmidt

I do think, however, that we should improve pytest so plugins can change how fixtures are handled, allowing plugins to implement whatever mechanism one prefers over the current name matching in core pytest.

Yay! 🎉 This would be wonderful.

what is wrong with the current fixture mechanism is the single global namespace shared among all participants

This is precisely why I think pytest-unmagic is an improvement on pytest magic fixture parameters. "Unmagic" fixtures are always explicitly referenced, and do not occupy a single shared global namespace. The question "where is the code for that?" can be answered without any knowledge of pytest fixture name resolution.

The part that @RonnyPfannschmidt reacted so strongly to (I think) is the active request reference, which is an implementation detail that allows the plugin to bind fixture setup and teardown to the appropriate scope and to retrieve the value from pytest's internal fixture cache. It is easy to reason about the active request within the context of any given test or fixture, so that single global reference does not seem as problematic as a shared global namespace for all fixtures.

Maybe "unmagic" is a bit of a misnomer. The plugin moves the "magic" part from fixture name resolution, which everyone using fixtures is confronted with, to an implementation detail where most people don't need to know much if anything about it. You might need to know if you're running tests in parallel within a single Python interpreter, but that is not a problem I tried to solve in v1.0.

@tristan

I think the annotation way which was already raised there is a very good idea. It is not that magical (pretty much just using typing.get_type_hints(metafunc.definition.function) + a fixture registry) and has already reached significant adoption by projects like SQLAlchemy and FastAPI. The implementation by @tristan https://github.com/tristan/pytest-annotated already looks quite good. It would be great to either open the private APIs which he is currently using or consider incorporating it into core pytest.

Regarding two ways to archive the same thing I also do not really have concerns. There are already other aspects where two approaches are used in pytest. The one that directly comes to my mind is import-mode which has gained importlib and is now recommended.

+1 for the approach pytest-annotated seems to be taking!

pytest-annotated looks like a good prototype

However we are not in a position to open the apis it's using

But I'm more than happy with working towards integrating types with fixture lookup

Navigation Menu

Repository navigation

Fixtures without name matching magic #3834

79 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Fixtures without name matching magic #3834

Description

Activity

nicoddemus commented on Aug 19, 2018

shoyer commented on Aug 19, 2018

nicoddemus commented on Aug 19, 2018

shoyer commented on Aug 20, 2018

nicoddemus commented on Aug 23, 2018

RonnyPfannschmidt commented on Aug 24, 2018

burnpanck commented on Sep 11, 2018

mtvx commented on Oct 11, 2018

nicoddemus commented on Oct 11, 2018

mtvx commented on Oct 11, 2018

countvajhula commented on Aug 16, 2019

79 remaining items

RonnyPfannschmidt commented on Jul 24, 2024

adriangb commented on Jul 24, 2024

tristan commented on Sep 30, 2024

RonnyPfannschmidt commented on Sep 30, 2024

millerdev commented on Oct 22, 2024

RonnyPfannschmidt commented on Oct 23, 2024

millerdev commented on Oct 23, 2024

RonnyPfannschmidt commented on Oct 23, 2024

nicoddemus commented on Oct 24, 2024

Footnotes

burnpanck commented on Oct 24, 2024

nicoddemus commented on Oct 24, 2024

millerdev commented on Oct 24, 2024

septatrix commented on Nov 22, 2024

adriangb commented on Nov 22, 2024

RonnyPfannschmidt commented on Nov 23, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions