Repository navigation
Pytest test discovery over SSH does not work #19864
Description
Type: Bug
Behaviour
Pytest test discovery does not find any test inside the VS Code python extension when working on a remote machine with the Microsoft SSH Remote extension. No error is reported in the OUTPUT/PYTHON window. Indeed, when I copy and paste the command from OUTPUT/PYTHON in the terminal, it finds the tests just fine. Similarly, when I run pytest --collect-only from the terminal, it finds the test just fine.
I have double checked the configuration and that all subfolders have __init__.py . Indeed, I have tried moving the test files to different places, including right in the folder (not in any subfolder) configured in settings.json and still no test is discovered.
However, it works well without the SSH remote extension.
Expected vs. Actual
I expect the tests to be discovered, or an output error to be displayed in OUTPUT/PYTHON window. However, no test is discovered and no error is displayed, as described above.
Moreover, I would expect OUTPUT/PYTHON to display the same output as when I run the command in the terminal:
~/venvs/my_repo/bin/python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear my_src_folder_with_tests
Interestingly, when I force a bug in one of the tests (like adding a random character after the brakedts in the test definition), the error is catched and displayed in OUTPUT/PYTHON. But when I fix the bug, no error is displayed as expected, but the tests are not discovered.
Steps to reproduce:
- Connect to the remote machine over ssh and open the folder with the repo.
- Create
test_fake.pyfile inside thesrc_with_testfolder configured in"python.testing.pytestArgs". - Be sure
src_with_testhas an__init__.pyfile, and thattest_fake.pyhas a valid test, like:
test_one():
assert True
- Configure pytest (ctrl + shift + p) or in
settings.json:
{
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true,
"python.testing.pytestArgs": [
"humanmodelling"
]
}
- Open the Testing panel and click refresh discovery.
Diagnostic data
- Python version (& distribution if applicable, e.g. Anaconda): 3.8.0
- Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
- Value of the
python.languageServersetting: Default
Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)
XXX
User Settings
languageServer: "Pylance"
testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true
Extension version: 2022.14.0
VS Code version: Code 1.71.2 (74b1f979648cc44d385a2286793c226e611f59e7, 2022-09-14T21:03:37.738Z)
OS version: Windows_NT x64 10.0.22621
Modes:
Sandboxed: No
Remote OS version: Linux x64 5.4.0-1090-azure
System Info
| Item | Value |
|---|---|
| CPUs | 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz (8 x 2995) |
| GPU Status | 2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off |
| Load (avg) | undefined |
| Memory (System) | 31.84GB (16.53GB free) |
| Process Argv | --crash-reporter-id c7209560-d4e0-4b0b-af53-21d213e1c4a6 |
| Screen Reader | no |
| VM | 0% |
| Item | Value |
|---|---|
| Remote | SSH: sergiov-vm-nc6sv3 |
| OS | Linux x64 5.4.0-1090-azure |
| CPUs | Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz (6 x 2593) |
| Memory (System) | 110.06GB (107.62GB free) |
| VM | 0% |
A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30557514
vsaa593:30376534
pythonvs932:30410667
cppdebug:30492333
vsclangdc:30486549
c4g48928:30535728
hb751961:30553087
dsvsc012:30540252
azure-dev_surveyone:30548225
2144e591:30553903
fc301958:30567733
Activity
eleanorjboyd commentedon Sep 20, 2022
Hello, we are aware of buggy behavior on pytest discovery and running in vscode. We are currently work on a rewrite of how pytest works in vscode and hope this rewrite will fix this and similar bugs. The main issue which will be working on is here: #17242, and progress will be updated on this issue. Thank you!
bphunter1972 commentedon Oct 14, 2022
I think I have the same issue.
I wonder if this is at all related to vscode reporting that both pylint and mypy are not installed in the venv on the remote host even though they are. I say this because the output view shows no responses in vscode, but if I run these same commands in my shell they all work just fine:
If it helps, I used redirection to determine that all of the output is going to stdout and nothing goes to stderr.
eleanorjboyd commentedon Oct 17, 2022
Hello! We don’t actually install pytest by default- could this be the problem that you are experiencing?
bphunter1972 commentedon Oct 17, 2022
Nope. Pytest, pylint, and mypy are all installed in the poetry env that the project points to. Run the exact same command-lines from the shell and I see the list/dict of tests.
kdbanman commentedon Nov 15, 2022
I have the identical issue to Sergio Valcarcel Macua (@sergiovalmac).
Project structure:
Local, tests properly displayed
A project's tests are discovered and displayed just fine on a local VS Code window.
Remote, discovered tests NOT displayed
But the same project on a remote SSH VS Code window does not display the tests, even though most of the test discovery code path appears to work fine:
Test adapter payload JSON, remote vs local
Eleanor Boyd (@eleanorjboyd) I am trying to diagnose the specific problem and patch it, because I would really like to interact with my tests in VS Code. I see that the test discovery results are reported via
pythonFiles/testing_tools/adapter/report.py, specifically the_sendcall at the very end, which just prints a JSON payload to stdout.The JSON payload appears to be functionally identical between the local and remote windows, though there are some differences.
I would like to continue tracing this execution path to see why the remote window GUI doesn't display the tests. So what happens with that JSON payload? Is stdout being consumed by some javascript/typescript somewhere?
eleanorjboyd commentedon Dec 15, 2022
Hello! Apologies for the delay but I am unable to replicate this current issue. I have followed the steps outlined in Sergio Valcarcel Macua (@sergiovalmac) description and was able to find all my tests successfully. I tried on a Mac to Windows connection, does everyone seeing this issue has the same remote and local machine? Kirby Banman (@kdbanman) what is your remote and local machine types?
eleanorjboyd commentedon Dec 15, 2022
Kirby Banman (@kdbanman) yes standard out is being consumed. It should be consumed by this file,
src/client/testing/testController/common/server.tsand from there go to thepytestController.pyand then thecontroller.py. Let me know if this helps. We are also in the middle of a re-write for pytest which should fix this problem as well but I am unable to give an estimate on when that may be release. Thanksomidmeh commentedon Feb 6, 2023
I'm seeing the same thing.
Tests discovered locally and in WSL, but not in the Linux remote.
The odd thing is that the Python log shows the following command being executed which when I run directly in the shell does indeed produce results.
eleanorjboyd commentedon Feb 7, 2023
Omid (@omidmeh) can you create a separate issue for the problem you are facing? This will just be easier to keep the conversation straight and help you specifically. Thanks
bphunter1972 commentedon Feb 24, 2023
Eleanor Boyd (@eleanorjboyd): Don't try Mac to Windows. Try Mac to Linux. That's where I'm seeing it as well.
1 remaining item
eleanorjboyd commentedon Mar 8, 2023
Could someone get their system args on their remote machine as they are attempting this? Trying to see where the cwd might have gotten messed up since that seems to be the problem. For linux running
printenv.bphunter1972 commentedon Mar 8, 2023
I cannot provide all of my env variables as that might show some proprietary information. Are there some in particular, such as PWD, that you would be interested in?
Also, is the Remote spawning the preferred shell for the user (mine is zsh), or is it spawning a standard sh or bash?
eleanorjboyd commentedon Mar 22, 2023
Hello! Yes the PWD would be of interest. Then for remote-ssh it is spawning the standard shell for the remote when it is installing the vscode server. After the vscode server is installed it will then use the preferred shell as listed in your vscode settings.
bphunter1972 commentedon Mar 30, 2023
I'm having a hard time figuring out the best way accomplish this.
After Developer: Reload Window, I see this in the Python output:
As you can see, the printEnvVariables.py isn't sending anything to the output (nor is the discover pytest step).
Of course it works just fine when I run this in the terminal, whether that happens in bash or my preferred zsh.
So, how would I inject a 'printenv' into the steps above?
eleanorjboyd commentedon Apr 4, 2023
Hi! No need to inject it- just open your terminal once the connection is set up and then run python to print the variables. Thanks
koujaz commentedon Apr 20, 2023
Just came across this issue and thanks to this #19864 (comment) I was able to get it to work.
In my case the payload adapter also provided strange paths such as:
"source": "../../../opt/project/src/resources_test/usage_test.py"Then I realized I'm opening the project in a directory "/root/current/devel" which is a symbolic link to "/opt/project"
on different filesystem. The above relative path is correct but not what would you expect relative to project root.
Opening the project directly in "/opt/project/" on remote machine yielded correct relative paths:
"source": "./src/resources_tests/usage_test.py"And all tests were correctly discovered.
bphunter1972 commentedon Apr 21, 2023
The PWD once the window is loaded is, as I expected, the absolute path of the workspace folder.
gtkramer commentedon Jun 2, 2023
koujaz you are a life saver! Confirmed that this is a bug. Steps for reproduction:
pwdis ~/drives/dev123/test-prj. Then run pytest in a virtual environment to see that it is working correctly:But, reopen VSCode at the absolute path to this folder, and suddenly everything works! Discovery and test execution works as expected.
I was going crazy. I wiped the contents of .cache, .config, .vscode, and .vscode-server on the Linux machine, reinstalled the extensions on Linux, reinstalled VSCode on Windows, tried a zillion configuration settings for the user and the remote... I even was using what Eleanor Boyd (@eleanorjboyd) was suggesting for others to get more debug information by running python scripts out of the extension directly. Nothing seemed to be working. All I had to do was change where exactly I opened the VSCode workspace folder!
I am using VSCode 1.78.2 (the current release) and the Python extension v2023.8.0 (also the current release). This issue happens irrespective of the Python version in use (I tried 3.9, 3.10, and 3.11). I would expect the extension to work irrespective of whether VSCode has opened a workspace folder on the remote at the absolute path or not.
Just upvoted this and now fingers are crossed :)
bphunter1972 commentedon Jun 2, 2023
Whoa! That's it! I never thought to use the absolute path. Symlinks are just so natural, it never occurred to me.
Thanks so much!
eleanorjboyd commentedon Jun 23, 2023
Hi! We are wrapping up a rewrite on testing infrastructure which is aimed at improving buggy behavior and making pytest work in a more native way. I will be testing discovery in regard to many issues we have in this space (symlinks, discovery etc) on the rewrite to see what issues still exist. If you would like to help and test your issue out on the rewrite (steps explained below) that would be most helpful in understanding the ongoing shortcomings and helping me to determine and execute the next steps. Thanks!
To test you need to be on vscode insiders and then add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. This is currently what is needed but we are in the process of switching all users to the rewrite but are doing so incrementally so if you do not have insiders, watch our release notes to get updated on when it will begin to hit stable. Thanks!
eleanorjboyd commentedon Jul 11, 2023
Consolidating all issues on testing and symlinks to issue #9347. Let me know if you do not think this other issue is representative of the bug you are facing. Thanks!