Repository navigation
Python Unittest isues #7176
Description
Issue Type: Bug
Synopsis: unexpected behaviours - (1) running a file of tests doesn't run all tests in the file, (2) auto-configuration of testing subdir doesn't seem complete/correct; feature request - (3) nice if could have a 'run tests' button for a single file or selected subset of tests.
My test case is biopython - not small or minimal, but easy to access and a good example of large, collaborative project that I don't get to change the structure of to make things work. I believe I have all the Unittest functionality working, below is what I had to do to make it so. I think with the info below #5252 can be closed?
Tests are in a 'Tests' subdirectory, named test_*.py, based on python unittest. Putting this info into the Testing configuration tool, it set up python.unitTest.cwd to ${workspaceFolder}/Tests and the -s option in python.unittest.args to ./Tests.
Issue: In order to make test discovery work, I needed to change .Tests to ${workspaceFolder}/Tests in python.unittest.args -s option.
Issue: When I run a test_*.py file with the green 'play' arrow, the '?' changes to a green checkbox indicating test passed. However, on opening the file in the outline pane, the indication is that only the first test has been run.
Issue: When I click the 'bug' icon next to the 'play' arrow to debug a unittest on a test that reads from a subdirectory, the test is running in ${workspaceFolder} instead of ${workspaceFolder}/Tests -- even though the run test ('play') succeeded fine. To fix this I created a test entry in launch.json (entry found in some old files, don't know where it came from so new user would need more help here) and added a $cwd entry:
{
"name": "Unit Tests",
"type": "python",
"request": "test",
"cwd": "${workspaceFolder}/Tests"
}
Feature Request: Biopython has 287 files in the Tests subdir, it would be really nice if the 'Run tests' button in the bottom status bar could be configured to e.g. just run the 12 files of tests relevant to the section of code that I am working on.
Despite these issues, I really, really applaud the testing framework in VS Code! I have never written unit tests before (suspect I'm not alone in this), and having the nice, slick framework and buttons all set up makes me want to implement and use them. Trying not to overstate this, but having this all working easy and friendly really seems like it could have a huge impact on the world of coding quality.
Extension version: 2019.8.30787
VS Code version: Code 1.37.1 (f06011ac164ae4dc8e753a3fe7f9549844d15e35, 2019-08-15T16:16:34.800Z)
OS version: Darwin x64 18.7.0
System Info
| Item | Value |
|---|---|
| CPUs | Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (4 x 3200) |
| GPU Status | 2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on native_gpu_memory_buffers: enabled oop_rasterization: unavailable_off protected_video_decode: unavailable_off rasterization: unavailable_off skia_deferred_display_list: disabled_off skia_renderer: disabled_off surface_synchronization: enabled_on video_decode: enabled viz_display_compositor: disabled_off webgl: enabled webgl2: enabled |
| Load (avg) | 2, 2, 2 |
| Memory (System) | 32.00GB (9.98GB free) |
| Process Argv | |
| Screen Reader | no |
| VM | 0% |
Activity
karrtikr commentedon Sep 3, 2019
Hi Rob Miller (@rob-miller) 👋
It should be
./Tests, not.Tests, please try that. This doesn't seem like an issue.This is the expected behavior right now. The logic is,
If any test fails, the parent should be failure, otherwise pass if all tests that ran passed, and not run if no tests ran.
So a node is '?' only if all the nodes below it are '?'. If you wish to know more details about a test node, you can just hover over any test item to see if it contains any tests that did not run.
It could be useful, could please open up a new issue on this feature request, so that we can make a decision on whether we want this.
karrtikr commentedon Sep 3, 2019
Btw are the test nodes with '?' being skipped intentionally?(using
unittest.skipdecorator or something) Try running them individually.karrtikr commentedon Sep 4, 2019
I will follow up later regarding this issue.
rob-miller commentedon Sep 4, 2019
Cheers Kartik Raj (@karrtikr),
Sorry, typo in my bug report - it was script-generated as ./Tests and I had to change to ${workspaceFolder}/Tests. When left as ./Tests, test discovery did not find any tests.
No, and this is the issue I am reporting. Referring to the screenshot, when I run (green arrow) test_PDB_parse_pdb_header.py, I get the result shown (only the test_parse_header_line appears to have run, and if I had not opened the selected line I would think that all four sub-tests ran and passed). If I run from the ParseReal line, all four sub-tests run and succeed.
I will open the feature request point separately as you suggest.
karrtikr commentedon Sep 4, 2019
I am able to confirm this, thanks.
thornhillcody commentedon Sep 4, 2019
I've run into this same problem. It really should be treated as a bug, not a request for a new feature. When running a module of tests, it doesn't make sense that only the first test case should run. I've included a zip of a very small project for you to reproduce. In the Test window, just click the run button on the module as shown:
pyunittestproblem.zip
karrtikr commentedon Sep 4, 2019
thornhillcody Like I commented earlier, I am able to confirm this as a bug. This is not expected. Thanks for giving a minimal sample btw, it really helps!
karrtikr commentedon Sep 4, 2019
Rob Miller (@rob-miller) thornhillcody This issue is a duplicate of #4567. Can you guys please upvote that issue, it helps us prioritize it.
karrtikr commentedon Sep 4, 2019
Unfortunately I am not able to reproduce this. I had used the following settings:
I am able to discover and run tests just fine. Can you please paste your
settings.json?5 remaining items
rob-miller commentedon Sep 5, 2019
Thanks, I have upvoted #7204. Sorry I hit the 'close' button
rob-miller commentedon Sep 5, 2019
I was thinking that a 'request: test' entry should be generated in launch.json as needed by test configuration, but did not realise it was not required for the test framework in general. Otherwise it seems like several 'user discovery' steps are still needed to make things work after s/he described the desired situation in the test configuration sequence?
Ideally this debug entry would be auto-selected when debug icon clicked in test explorer, but I can see how this might cause more problems.
rob-miller commentedon Sep 5, 2019
OK, I have done more testing and conclude that the issue is a conflict with the python.testing.cwd entry. There is caching involved to add confusion, so on the Mac I have to close the VS Code window (but not 'quit' from the menu) and re-open at the "Editing evolved" screen where I pick the folder to work on to see the effect of changing python.testing.cwd.
In my case I need "python.testing.cwd": "${workspaceFolder}/Tests" in order for the tests to succeed, as they read files from Tests/subdir. I think setting testing.cwd does not happen in the test configuration process. With this setting, python.testing.unittestArgs "-s ." and "-s ${workspaceFolder}/Tests" work for test discovery, but "-s ./Tests" does not (this now makes sense to me).
Please see if you can reproduce this? I can see that it would be expected behavior but it was confusing.
Not knowing the further ramifications especially for the other testing frameworks, it seems like a better situation here would be for test configuration to set testing.cwd for a testing subdir and leave it out of testing.unittestargs (no -s option at all works for me). This way the user would not have to discover testing.cwd if they need it.
karrtikr commentedon Sep 5, 2019
Yes, it's better not to expose it to users. We add the configuration ourselves internally while sending the debug configurations to debugger.
Will be looking into the other issue shortly.
karrtikr commentedon Sep 6, 2019
If you have both
"python.testing.cwd": "${workspaceFolder}/Tests"and"-s ./Tests", then it won't work because"./Tests"would be resolved to"${workspaceFolder}/Tests/Tests". I think you realise this now.I understand it was confusing but I don't think there's much we can do here.
-soption is definitely needed because it gives the location where to discover tests. It's just for your particular configuration, it didn't work. Also setting"python.testing.cwd"could affect other aspects of testing, so setting it just for your case doesn't seem right.Thanks for reporting these anyways. Closing it now as issues has been addressed.
karrtikr commentedon Sep 6, 2019
You can let me know if there's a better approach here, and we can reconsider doing something about it.
rob-miller commentedon Sep 7, 2019
Yes, I understand the situation.
May I suggest adding some further warning or information about the conflict to the current testing.cwd and testing.unittest text:
Not sure what the best wording is, but somehow something should highlight an issue if testing.cwd is set. Presumably testing.cwd similarly affects the other testing frameworks?
karrtikr commentedon Sep 7, 2019
So basically the message we want to convey is that, if
testing.cwdis set, then.in directory paths resolve to the value of the setting, right? (If you would like this, please open up a new issue on this, marking it as code health). I think it would be better if somehow let user know the directory user is discovering tests in. Maybe in the logs.And yes, it affects other frameworks.