Repository navigation
pytest-xdist regression since execnet 1.3.0 #707
Description
Originally reported by: Marc Schlaich (BitBucket: schlamar, GitHub: schlamar)
With the latest execnet release 1.3.0 a pytest-xdist test in pytest-cov fails with ImportError: no module named py.
This is the first error in this build: https://travis-ci.org/schlamar/pytest-cov/jobs/56056841
Pinning execnet to 1.2.0 solves this issue.
Activity
pytestbot commentedon Mar 27, 2015
Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):
this is a problem triggered by a bugfix in execnet
due to an oversight we would add the site-packages of a different python interpreter to sys.path of the slave interpreter
which is also why other modules would appear there
we need to ensue that on the slave a pylib/pytest of the same version is importable
pytestbot commentedon Mar 27, 2015
Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):
So there was a bugfix in execnet which exhibited a problem in the pytest-xdist code which tries to ensure that py/pytest is importable remotely? If so, can you provide a PR fixing this?
pytestbot commentedon Mar 27, 2015
Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):
the problem is local import-ability if the python of the slave is different from the master
pytestbot commentedon Mar 27, 2015
Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):
to solve this propperly we need to ensure that pylib, pytest and potentially the pytest plugins in use + their dependencies are importable in the slave process
with pytest/pylib, we can probably just create module instance for
_pytest,pyas well aspytestand do something evil with execfile and__path__/__file__as a first approximation,later after we support the wheel distribution format we should look into passing pure python wheels straight to slaves via a execnet mechanism (needs some discussion)
pytestbot commentedon Mar 27, 2015
Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):
Let's keep it simpler as a first iteration: just make sure that py and pytest are importable, no plugins and no considerations of wheels. This just requires rsyncing/copying the installed packages from site-packages and we know that py/pytest are interpreter independent.
pytestbot commentedon Mar 27, 2015
Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):
im not sure what to use as the rsync location, got any suggestions?
pytestbot commentedon Mar 27, 2015
Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):
it's not really rsync but i guess you mean local destination for the copy of py and pytest. We could use a tempdir although it's not very elegant. What about teaching execnet to pre-import a list of modules and temporarily prepending the source's site-packages for that? It's a hack but to do all of this properly we need to create virtualenvs, and then we wouldn't do any copying.
pytestbot commentedon Mar 27, 2015
Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):
i'd like to avid adding such behavior to execnet, i suspect it will bring in fragile misbehavior, however it sounds like a perfect hack for pytest-xdist for handling the situation of different executable until we can create a proper solution
Diaoul commentedon Jul 12, 2015
I just faced the same issue, any updates on this or workaround?
RonnyPfannschmidt commentedon Jul 12, 2015
not yet
RonnyPfannschmidt commentedon Jan 18, 2017
closing as the followup is pytest-dev/pytest-xdist#113