Repository navigation
How to mark up fixtures with metadata for segregating collections of fixtures in scope #11462
Description
What's the problem this feature will solve?
The problem to be solved here is how to organize families of fixtures with pure metadata to avoid looking up fixture values from their names to determine if the fixture belongs to a given family or not.
I'm looking for a way that I can mark up a bunch of fixtures spread around a bunch of plugin and conftest files etc some of which are marked up with the simple standard markers pytest.mark.metaFoo. I wish to then lookup those marks from the FixtureDef instances retained in the fixture manager to do some dynamic container creations from these fixture families identified by their markers alone. I wish to do this family identification via markers because fixture naming convention alone is inadequate for my use case where looking up fixture values for those fixtures which match a naming convention can cause recursion loops for those fixtures that hold dependencies on the fixture I'm trying to dynamically populate with the fixtures of a given family but not others. This is where marker metadata would be a powerful differentiator.
Describe the solution you'd like
I'd like to be able to decorate my fixtures with regular pytest markers as such (which is valid syntax today) but with that marker metadata able to be crawled and looked up from the request fixture object and ideally populated in the FixtureDef instances.
# plugin_1.py
import pytest
@pytest.mark.fixA
@pytest.mark.other
@pytest.fixture
def fixture_foo():
return "foo"
@pytest.mark.fixB
@pytest.fixture
def fixture_bar():
return "bar"
# plugin_2.py
import pytest
@pytest.mark.fixA
@pytest.fixture
def fixture_2foo():
return "2foo"
@pytest.mark.fixB
@pytest.fixture
def fixture_2bar():
return "2bar"From the above I need a fixture that dynamically returns a container of the fixture values ["foo", "2foo"] just by discovering the fixtures in scope and contributing to that container the values of all fixtures which match the family criteria corresponding to the pytest markers I wish to filter upon.
Alternative Solutions
So far I've been leaning on a combination of fixture naming conventions and value lookups to get this organization by families accomplished but that will not work when I get to the complex case where the different fixture subfamilies share the same naming convention and thus become undifferentiable. In that scenario the fixtures are given different definitions from alternate plugins where I don't want them to continue to be selected by name anymore because the naming convention then blurs between two families of fixtures that have no way to be differentiated by name alone (hence the need for the markers to do the job).
Here's the partial solution I've attempted so far that lets me organize fixtures by the naming convention pattern matching a suffix
@pytest.fixture(autouse=True)
def fix_list(request):
print("fix_list")
foo_fixtures = [request.getfixturevalue(fix_name) for fix_name in request._fixturemanager._arg2fixturedefs if fix_name.endswith("_foo")]
return foo_fixtureshttps://stackoverflow.com/a/77154959/1330381
Additional context
When I probe the fixture definition from a global scope I see that the fixtures have a pytestmark attribute, eg
fixture_foo.pytestmark
[Mark(name='fixA', args=(), kwargs={}), Mark(name='other', args=(), kwargs={})]
I can't see how to navigate to that data from the request fixture. If there is an existing means today that would solve all my problems and I could mark up my fixtures today and solve some gnarly problems.
Activity
jxramos commentedon Sep 25, 2023
Taking a look at the callstack for where
FixtureDefinstances get constructed, it appears to be located here for my user defined fixtures (the line numbers may be off on account of mypdb.set_trace()inserts and other debug prints)pytest/src/_pytest/fixtures.py
Lines 1698 to 1708 in 1a16bac
I'll see if snooping around any of these objects at this point in the
parsefactoriesmethod can make the markers visible for the fixtures. If so that will make things in reach for adding a markers attribute for the Fixture Definition class if that is agreeable.jxramos commentedon Sep 25, 2023
Aha, just found it, so obj still has the
pytestmarklist associated with it BUT once it gets transformed throughobj = get_real_method(obj, holderobj)the obj no longer has its markers as you can see in the dir output printed belowpresumably I could extract the markers list before the call to
get_real_methodand cache it in the FixtureDef with a new parameter? There could be some edge cases to consider with parametrization and other things I'm not thinking of that could collide. I'll need some help here getting oriented to this all and learning from any foresight folks could share from the dev team.jxramos commentedon Sep 25, 2023
I went ahead and printed any fixture which doesn't have
pytestmark, so the challenge will be how to conditionally include this marker metadata without disrupting the other fixtures which should be oblivious to this world.Perhaps one approach would be to default to the empty list if the pytestmark attribute is missing.
perhaps there's a better way to map this information that could be looked up in an alternate structure beside the fixture manager where I could take the names of the fixtures present in the keys of the fixture manager and just lookup in this alternate datasource the fixtures and their markers,
fixtures_2_markes = {"fixture_name": [<PytestMarker>]}. Then I could just do a straight dict lookup likeget_fixture_markes = fixtures_2_markers.get("fixture_name", [])RonnyPfannschmidt commentedon Sep 25, 2023
Markers intentionally do not work on fixtures and do not transfer
Its not correctly manageable in quite a number of edge cases surrounding fixtures, fixture overrides and scoping/Parameterization
jxramos commentedon Sep 25, 2023
can you elaborate on the terminology of 'work on' and 'do not transfer' here please.
If I'm not mistaken I recall a really good rich conversation over in #1368 that I was keen to tune in on that brought up this transferring terminology in the context of test cases that used a particular marked fixture. If I recall correctly that notion of transferring was the concept of taking markers from fixtures and transitively applying them to any test case which happened to use the marked up fixtures. That's not quite the use case I'm wrestling with however.
I want the markers on the fixtures not to produce a convenience marker test case filtering scheme but form a basis for organizing and filtering the fixtures themselves. My focus isn't about the test cases which may or may not use those fixtures I hope to organize but basically I want metadata to be used to produce a transcendent derivative fixture from other fixtures that are already in scope. In a nutshell Fixtures created by other fixtures.
Maybe I can get some insight into the commit that added
get_real_method, which appears to be work done by @nicoddemus over inc6b11b9#diff-b24b16546a68963d52dedaa79e73881adebdf116c4fdfc3859658879d13b9321
There's a curious comment over there...
for sure the
pytestmarkattribute gets lost passing through that method as I found in my comment above. This commit also links to another interesting comment in the issue it referenceswould you say markers on fixtures is still a grey area or something more rejected at this later date from the time of those above exchanges?
Maybe another syntax that could possibly accomplish this organizational attribute I wish to get access to is to use custom kwargs within the fixtures, such as...
@pytest.fixture(scope="function, meta_id="foo")Is that syntax that I could tie into somehow and fetch from
requestobjects to do my fixture organization with?jxramos commentedon Sep 25, 2023
looks like there's no
kwargsupport around the decorator, the lone*is to perhaps handle the test parameter values?pytest/src/_pytest/fixtures.py
Lines 1259 to 1269 in 1a16bac
Snooping around that class
FixtureFunctionMarkerleads to this curious linepytest/src/_pytest/fixtures.py
Lines 1209 to 1210 in 1a16bac
at which point I can see the connection to deprecated code
pytest/src/_pytest/deprecated.py
Lines 125 to 128 in 1a16bac
but interestingly enough that link doesn't have any hits to this subject any longer, maybe it did in the past? This is a bit surprising https://docs.pytest.org/en/stable/deprecations.html#applying-a-mark-to-a-fixture-function
RonnyPfannschmidt commentedon Sep 25, 2023
It was never support,but people still wrote code as if it was
jxramos commentedon Sep 25, 2023
Another angle I may be able to map against is the
baseid/nodeidvalues if they can connect me to the path to where the fixture was found. I can alternatively embed a keyword in the plugin's filename but I'm finding the empty string being returned for these parameters.This may be by design perhaps for reasons unclear to me For other plugins, the baseid is the empty string (always matches). what is the matching constraint being discussed here?
pytest/src/_pytest/fixtures.py
Lines 987 to 1000 in 1a16bac
jxramos commentedon Sep 26, 2023
come to think of it more generally, baseids, markers, whatever the source, I just need some metadata orthogonal to the fixture name which I can form associations with.
jxramos commentedon Oct 3, 2023
Looks like with
FixtureDefinstances you can actually get the path via the inspect module on itsfuncmember which is pretty neat.Zac-HD commentedon Feb 17, 2024
I think we never intended such code to work, and may deliberately break it in future. Probably the best approach to identifying families of fixtures is to track your metadata in some way unrelated to the pytest internals!