No breakpoint for Uncaught Exceptions while debugging tests

UPDATE: Seems this is an enhancement that may be "coming soon". If you've found this bug report in hopes of a fix, and are willing to accept the risks look at this reply for instructions on manually enabling the functionality

Searched for duplicates, none found. See the following post/thread for back story: #850 (comment)

Environment data

Version: 1.49.0
Commit: e790b931385d72cf5669fcefc51cdf65990efa5d
Date: 2020-09-10T17:39:53.251Z (1 wk ago)
Electron: 9.2.1
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Darwin x64 18.7.0

Extension version (available under the Extensions sidebar): v2020.8.109390
OS and version: macOS Mojave 10.14.6 (bug happens on every OS I've tried - Linux Mint & Windows 10)
Python version (& distribution if applicable, e.g. Anaconda): 3.8.1 (64 bit) - appears unrelated
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
Relevant/affected Python packages and their versions: pytest-5.4.3
Relevant/affected Python-related VS Code extensions and their versions: N/A
Value of the python.languageServer setting: Microsoft Python Language Server version 0.5.51.0

[Info  - 9:39:50 AM] Analysis cache path: /Users/####/Library/Caches/Microsoft/Python Language Server
[Info  - 9:39:50 AM] Microsoft Python Language Server version 0.5.51.0
[Info  - 9:39:50 AM] Workspace root: /Users/####/Documents/Code/####
[Info  - 9:39:51 AM] GetCurrentSearchPaths /usr/local/opt/python@3.8/bin/python3 
[Info  - 9:39:51 AM] Interpreter search paths:
[Info  - 9:39:51 AM]     /usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8
[Info  - 9:39:51 AM]     /usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload
[Info  - 9:39:51 AM]     /Users/####/Library/Python/3.8/lib/python/site-packages
[Info  - 9:39:51 AM]     /usr/local/lib/python3.8/site-packages
[Info  - 9:39:51 AM] User search paths:
[Info  - 9:39:55 AM] Initializing for /usr/local/opt/python@3.8/bin/python3
[Info  - 9:39:55 AM] Analysis caching mode: None.

Expected behaviour

When debugging a unit test with the breakpoint of Uncaught Exceptions enabled, the debugger should break on the exception rather than finishing the test as a failure. This is the behavior when invoking the debugger via the typical way through Run -> Start Debugging.

Actual behaviour

When debugging a unit test with the breakpoint of Uncaught Exceptions enabled, the debugger does not break on the exception and finishes as a failed test.

Steps to reproduce:

Create new folder with the following files:

main.py

def main():
    raise Exception("Demo")


if __name__ == "__main__":
    main()

test_main.py

from main import main


def test_main():
    main()

Begin debugging (F5) main.py with standard debug options
Observe that debugger breaks on exception, allowing you to use the interactive debugging console

-> main
 > <function main at 0x108dbd940>

Install pytest & discover tests (may need to add "python.testing.pytestEnabled": true to settings.json)
Click Debug Test over test_main in test_main.py
Observe that the debugger does not break and instead the test results come back as a failure

============================= test session starts ==============================
platform darwin -- Python 3.8.1, pytest-5.4.3, py-1.8.1, pluggy-0.13.1
rootdir: /Users/####/Documents/Code/test_demo
collected 1 item

test_main.py F                                                           [100%]

=================================== FAILURES ===================================
__________________________________ test_main ___________________________________

    def test_main():
>       main()

test_main.py:5: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def main():
>       raise Exception("Demo")
E       Exception: Demo

main.py:2: Exception
- generated xml file: /var/folders/lz/hcbk8xvj2gdf9htx6r3w6cmrgy1jf9/T/tmp-79341Yr1at8rMslzO.xml -
=========================== short test summary info ============================
FAILED test_main.py::test_main - Exception: Demo
============================== 1 failed in 0.15s ===============================

Logs

Intentionally left out, due to difficulty parsing out private data. See logs above

Dan Taylor (@qubitron)

Update:

In response to Dan Taylor (@qubitron) 's suggestion, I have attempted to disable justMyCode without success.

launch.json

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python: Current File",
      "type": "python",
      "request": "launch",
      "program": "${file}",
      "console": "integratedTerminal",
      "justMyCode": false
    }
  ]
}

Could not find an additional place to configure this setting (i.e. within settings python.test.*)

Update

Oddly enough seems the latest update may have resolved this in spite of the conspicuous omission in the changelog and the fact that it has been a long-standing bug for me spanning multiple systems. I haven't tried on my mac yet (the machine I submitted this report on), but it's now breaking as expected in Windows 10.

Version: 1.49.1 (user setup)
Commit: 58bb7b2331731bf72587010e943852e13e6fd3cf
Date: 2020-09-16T23:27:51.792Z
Electron: 9.2.1
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Windows_NT x64 10.0.18363

Python extension version: v2020.9.111407

Chad Bailey (@ChadBailey)

Chad Bailey (@ChadBailey) We did not change any thing in the extension specific to exception handling. I am not sure what changed to get this working.

The issue here is that the debugger debugpy supports this, we call this feature userUnhandled. When running with pytest all exceptions are technically handled by pytest. From python's point of view there won;t be any uncaught exceptions. But in the debugger there is a mode which treats any exception crossing user code to library code boundary as user-uncaught. Previously we tried to expose this feature in the extension had to immediately revert it because people did not like the default and could not agree on if they want default on or off. So for now we removed it from the extension. Debugger fully supports this as it is needed by Visual Studio 2019 nd can be set via the Exception settings.

It is entirely possible the issue was localized as I did originally have the extension Python Test Explorer installed and removed it as a troubleshooting step. Perhaps the update just managed to cause something to refresh? I'm unsure, I will report back once I'm able to test on my mac.

Meanwhile, if it's of any help I have also been getting some strange console behavior since the update which I've never seen before: Error: TypeError: message must be set

Can you file a separate bug for this: Error: TypeError: message must be set?

Can you file a separate bug for this: Error: TypeError: message must be set?

Done, #14067. I'm not certain the error isn't due to something on my end, so I'm not sure I would have filed if not for you suggesting. Hoping it's helpful.

Update

I've completely wiped all settings and extensions... This is definitely not fixed on my mac yet (which was upgraded to Catalina 10.15.6 since I made this report yesterday)

It would be helpful if someone else could weigh in if they are getting this behavior or not.

The unhandled exception behavior is currently by design in VSCode due to the way "unhandled" is defined, as described above. We're planning to add some UX to expose user-unhandled in exception filters, but that's tracked as a feature.

Pavel Minaev (@int19h)

Pavel Minaev (@int19h) I'm having trouble following your comments. It doesn't matter to me what you call it, I really need a way for the debugger to break on exceptions while running tests. This is almost mandatory for debugging longer integration/E2E tests. Sure in a small unit test it's just a minor inconvenience. When you have an E2E test suite that takes 15 minutes to run though, a silently failing exception when the user explicitly clicked "debug" is not a great user experience.

Luciana Abud (@luabud)

Absolutely! That's why debugpy implemented this feature already. It just takes time to figure out how to best surface it in VSCode UI. I was just saying that this is not a regression or an unexpected malfunction, just the unfortunate consequence of not having the feature there yet.

(Luciana Abud (@luabud) for UX considerations)

Ah, my apologies I understand now. Thanks for clearing that up. I didn't realize the feature didn't originally exist. In my opinion - it's worth as much as you paid for it - I think that following the setting for "Uncaught Exceptions" would be an intuitive approach. The only caveat is, in the event where the end user is issuing a command and intentionally checking for a specific exception that of course should not break (see below).

Example

main.py

def main():
    raise Exception("Demo")


if __name__ == "__main__":
    main()

test_main.py

import pytest
from main import main

# _SHOULD_ break on exception with `Uncaught Exceptions` selected
def test_main():
    main()  # Note: Main raises Exception("Demo")

# Should _NOT_ break on exception with `Uncaught Exceptions` selected
def test_main():
    with pytest.raises(Exception):
        main()  # Note: Main raises Exception("Demo")

I hope this is helpful.

We can't change the meaning of "uncaught" to mean "user-uncaught", because it's a standard exception filter, and its current behavior is required for VS to behave as it should. So this has to be another, distinct filter (which can still be enabled by default). VS, which already supports this, hides that fact by picking one or the other category depending on Just My Code setting. But in VSCode, the filters that are advertised by the debug adapter are exposed directly by debugging UI - the extension doesn't control that part. So, the challenge is to add that third filter in a way that makes the distinction between the two sufficiently clear.

You can edit your local copy of debugpy (bundled with the extension, wherever it's installed) to try it out already. Just uncomment this line:

https://github.com/microsoft/debugpy/blob/cf65911a7acbc7ae8d4edd47d00e84cf27fadada/src/debugpy/adapter/clients.py#L165

This will make the filter show up. However, note that its usability is hampered by microsoft/debugpy#399 at the moment.

So, the challenge is to add that third filter in a way that makes the distinction between the two sufficiently clear.

I see, thank you so much for taking the time to explain. If it helps at all, I often mis-call this Unhandled Exceptions. Perhaps that would be a succinct way of putting it that users would understand. Perhaps equally as likely is that this only makes sense in the reality which resides between my ears.

You can edit your local copy of debugpy (bundled with the extension, wherever it's installed) to try it out already.

This is super helpful! Thank you so much for showing me this, I may actually do that... this can become very disruptive when unexpected things break and I find myself chasing down all the possible places an exception could be raised.

You can edit your local copy of debugpy (bundled with the extension, wherever it's installed) to try it out already. Just uncomment this line:

https://github.com/microsoft/debugpy/blob/cf65911a7acbc7ae8d4edd47d00e84cf27fadada/src/debugpy/adapter/clients.py#L165

Woohoo!! I got it working!

Here's some steps for posterity:

Open VSCode and visit console Ctrl+`
Click the OUTPUT tab, select Python from the dropdown
Hit Ctrl+F and search for pythonFiles (Note: You may need to issue a command such as discover tests first)
Copy discovered system path to pythonFiles (i.e. ~/.vscode/extensions/ms-python.python-2020.9.111407/pythonFiles/) then open this folder using VSCode
Search all files Ctrl+Shift+F for the term userUnhandled
Select the search result found in clients.py and modify line to be the following

              {"filter": "userUnhandled", "label": "User Uncaught Exceptions", "default": True},

NOTE: I think I originally had a syntax error causing this, but it did not work until I toggled default to True

Save file and restart VSCode

Navigation Menu

Repository navigation

No breakpoint for Uncaught Exceptions while debugging tests #14056

Environment data

Expected behaviour

Actual behaviour

Steps to reproduce:

Logs

Update

Update

Example

main.py

test_main.py

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

No breakpoint for Uncaught Exceptions while debugging tests #14056

Description

Environment data

Expected behaviour

Actual behaviour

Steps to reproduce:

Logs

Activity

ChadBailey commented on Sep 23, 2020

ChadBailey commented on Sep 23, 2020

Update

karthiknadig commented on Sep 23, 2020

ChadBailey commented on Sep 24, 2020

karthiknadig commented on Sep 24, 2020

ChadBailey commented on Sep 24, 2020

ChadBailey commented on Sep 24, 2020

Update

int19h commented on Sep 25, 2020

ChadBailey commented on Sep 25, 2020

int19h commented on Sep 25, 2020

ChadBailey commented on Sep 25, 2020

Example

main.py

test_main.py

int19h commented on Sep 25, 2020

ChadBailey commented on Sep 25, 2020

ChadBailey commented on Sep 25, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions