Very long delay when a breakpoint is hit and many (150+) threads are created by the program

Environment

OS and version: Ubuntu 20.04.5 LTS
VS Code: 1.72.1
C/C++ extension: v1.12.4
OS and version of remote machine (if applicable): gdbserver is running on an embedded ARMv7 Linux
GDB version: 7.4

Bug Summary and Steps to Reproduce

Bug Summary:
I am developing a C application with 150+ threads and debug it remotely via gdbserver. When the first breakpoint is hit, vscode requests detailed information about every thread in the application and it takes about 3 minutes to complete. The debugging is very painful due to this.
The communication between gdb and gdbserver is very sensitive to the network latency. It is important to make as few requests to gdb/gdbserver as possible.

Steps to reproduce:
Create an app which starts 150+ threads.
Debug it remotely using gdbserver.
Set a breakpoint to be hit after all threads are created already.
Debug the program.
When the breakpoint is hit you will get a significant delay before vscode UI reports the breakpoint hit.

Debugger Configurations

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "(gdb) Debug app",
            "type": "cppdbg",
            "request": "launch",
            "program": "link.out",
            "stopAtEntry": false,
            "cwd": "${workspaceRoot}",
            "environment": [],
            "externalConsole": true,
            "MIMode": "gdb",
            "miDebuggerPath": "arm-linux-gnueabi-gdb",
            "miDebuggerServerAddress": "10.0.1.10:9099",
            "targetArchitecture": "arm",
            "symbolLoadInfo": {
                "loadAll": false
            },
            "logging": {
                "engineLogging": false
            },
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                },
                {
                    "text": "-interpreter-exec console \"set sysroot build_dir/sysroot\"",
                    "ignoreFailures": true
                }
            ]
        }
    ]
}

Debugger Logs

Here is what happens when a break point is hit and why it takes so long.
The break point is hit:

1: (20384) ->=thread-created,id="2",group-id="i1"
1: (20384) ->~"[New Thread 1800]\n"
1: (20385) <-1023-thread-info 2
[New Thread 1800]
1: (20389) ->=breakpoint-modified,bkpt={number="3",type="breakpoint",disp="keep",enabled="y",addr="0x00b5efb4",func="x",file="x.c",fullname="x.c",line="568",times="1",original-location="x.c:568"}
1: (20390) ->~"[Switching to Thread 1800]\n"
[Switching to Thread 1800]
1: (20466) ->*stopped,reason="breakpoint-hit",disp="keep",bkptno="3",frame={addr="0x00b5efb4",func="x",args=[],file="x.c",fullname="x.c",line="568"},thread-id="2",stopped-threads="all",core="0"

Information about all(!) threads is requested:

1: (20467) <-1024-thread-info

While GDB executes the request above, it reports availability of new threads (158 in my case):

1: (20509) ->=thread-created,id="3",group-id="i1"
1: (20509) ->~"[New Thread 1639]\n"
1: (20509) ->=thread-created,id="4",group-id="i1"
1: (20509) ->~"[New Thread 1640]\n"
1: (20509) ->=thread-created,id="5",group-id="i1"
1: (20510) ->~"[New Thread 1641]\n"

vscode starts to request info about each reported thread. Even though information about all threads has been requested earlier:

1: (20510) <-1025-thread-info 3
1: (20510) ->=thread-created,id="6",group-id="i1"
1: (20510) ->~"[New Thread 1642]\n"
[New Thread 1639]
1: (20510) ->=thread-created,id="7",group-id="i1"
1: (20510) ->~"[New Thread 1649]\n"
1: (20510) ->=thread-created,id="8",group-id="i1"
...

...
1: (20528) <-1178-thread-info 156
[New Thread 1798]
1: (20528) <-1179-thread-info 157
[New Thread 1799]
1: (20528) <-1180-thread-info 158
[New Thread 1801]
1: (20528) 1023: elapsed time 143
1: (20528) Send Event AD7ThreadCreateEvent
...

Information about all 158 threads has been requested.
After 11 seconds GDB returns response to request -thread-info``` and starts to slowly return responses for individual thread info requests:

...
1: (31603) ->1024^done,threads=[{id="158",target-id="Thread 1801",frame={level="0",addr="0xb6e97798",func="??",args=[]},state="stopped",core="0"},{id="157",target-id="Thread 1799",frame={level="0",addr="0xb6fa7ce4",func="??",args=[]},state="stopped",core="0"}, ...
1: (31604) ->(gdb)
1: (31606) 1024: elapsed time 11138
1: (32457) ->1025^done,threads=[{id="3",target-id="Thread 1639",frame={level="0",addr="0xb6e91e7c",func="??",args=[]},state="stopped",core="0"}]
1: (32458) ->(gdb)
1: (32458) 1025: elapsed time 11948
1: (33343) ->1026^done,threads=[{id="4",target-id="Thread 1640",frame={level="0",addr="0xb6e97798",func="??",args=[]},state="stopped",core="0"}]
...

After 151 (!) seconds GDB responds to the last thread info request and vscode is almost ready to inform a user about the break point:

...
1: (171843) ->1180^done,threads=[{id="158",target-id="Thread 1801",frame={level="0",addr="0xb6e97798",func="??",args=[]},state="stopped",core="0"}]
1: (171843) ->(gdb)
1: (171844) 1180: elapsed time 151315
1: (171845) <-1181-stack-list-frames 0 1000

Conclusion: it is not needed to request individual thread info since all thread info is requested anyway and such combined request is executed much much faster compared to many individual requests.

You got my vote - got similar problems on our side.

My vote

you got my vote too, this is a horrible issue that makes vscode much less useful than it should be. We should be able to debug in a timely manner. I'm new to this group. Is there some formal mechanism I can use to show my vote?

+1 the C++ debugging performance is the major workflow pain our team and colleagues experience using VSCode as a C++ IDE. It gets so bad for me that I sometimes switch to command-line GDB or use GDB in Emacs. Everything else for C++ development in VSCode scales much better in my experience.

+1

This has nothing to do with remote debugging. Performance when debugging locally is equally awful.

I have to wait minutes to get local variables to appear (during which time I can do nothing). Closing "locals" in the variables window does nothing because it's automatically reopened when execution next stops.

This effectively makes VSCode unfit for purpose and needs to be fixed.

+1 why should we have to disable everything just to step normally? Not very useful to me with everything hidden

+1 from me as well. Very important to have fast gdb on VSCode.

+1 Just recently started debugging a project again after having it sit around for 2 years and this used to work quite decent and snappy. Now it's almost impossible to debug it at all, I wonder what was introduced in the meantime that caused this

+1 I'm currently debugging a multi-threaded program (80+ threads) that takes a long time to step through the lines after hitting a breakpoint. It makes debugging with VS code a rather painful experience. I have no such performance issues when debugging with gdb on the command line

Georg Lauterbach (@georglauterbach)

Georg Lauterbach (@georglauterbach) I haven't seen had that happen to me... I'm sorry! Maybe there's an LLDB setting you can try?

I switched to using the CodeLLDB extension and the lldb debugger and I haven't had any issues since. Stepping over code and hitting breakpoints is really fast now!

Sadly, I also tried that, but LLDB dies with SIGUSR1 and other oddities... it's a shame really :( Any advice when using LLDB with many threads @carterols ?

I am afraid LLDB would not suit our use cases either with remote debugging using gdbserver and KGDB on ARM and ARM64 platforms with custom firmware.

kyselejsyrecek

kyselejsyrecek

could you share your build configuration for GDB, please?

Just a "./configure", nothing more.

Maybe these can help?
config.log:

This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

It was created by configure, which was
generated by GNU Autoconf 2.69.  Invocation command line was

  $ ./configure 

## --------- ##
## Platform. ##
## --------- ##

hostname = ..
uname -m = x86_64
uname -r = 5.15.0-97-generic
uname -s = Linux
uname -v = #107-Ubuntu SMP Wed Feb 7 13:26:48 UTC 2024

/usr/bin/uname -p = x86_64
/bin/uname -X     = unknown
........................
configure: exit 0

gdb/config.log:

This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

It was created by configure, which was
generated by GNU Autoconf 2.69.  Invocation command line was

  $ /.../binutils-gdb/gdb/configure --srcdir=.././gdb --cache-file=./config.cache --with-gnu-as --with-gnu-ld \
    --program-transform-name=s,y,y, --disable-option-checking --disable-year2038 --build=x86_64-pc-linux-gnu \
    --host=x86_64-pc-linux-gnu --target=x86_64-pc-linux-gnu

## --------- ##
## Platform. ##
## --------- ##

hostname = ..
uname -m = x86_64
uname -r = 5.15.0-97-generic
uname -s = Linux
uname -v = #107-Ubuntu SMP Wed Feb 7 13:26:48 UTC 2024

/usr/bin/uname -p = x86_64
/bin/uname -X     = unknown
........................
configure: exit 0

And a full gdb/config.h: config.zip

Georg Lauterbach (@georglauterbach)

Georg Lauterbach (@georglauterbach) Here is the GDB config in the VSCode project:

"symbolLoadInfo": {
       "loadAll": false,
       "exceptionList": "libc.so;libstdc++.so;<selected symbols to load...>"
},
 "setupCommands": [
            {
                   "description": "Enable pretty-printing for gdb",
                   "text": "-enable-pretty-printing",
                   "ignoreFailures": true
            },
            {
                   "description": "Load gdb configuration",
                   "text": "source ~/.gdbinit",
                   "ignoreFailures": true
            },
            {
                    "description": "Add file path directory",
                    "text": "directory </path/to/my/project>",
                    "ignoreFailures": true
            },
            {
                    "description": "Load sbtools source mapping",
                    "text": "source /path/to/source/mapping/file",
                    "ignoreFailures": true
            }
 ]

The ~/.gdbinit only defines some customized pretty printers.

Yuriy (@yuriy-chumak)

No issue observed with self-built gdb. My setup:

Linux Mint 21

GDB 15.0.50.20240304-git

VSCode 1.77.3

Tested with obs-studio debugging (which was completely impossible before).

launch.json (nothing special, just usual settings):
{
"name": "obs",
"type": "cppdbg",
"request": "launch",
"program": "${workspaceFolder}/Build/rundir/Debug/bin/64bit/obs",
"cwd": "${workspaceFolder}/Build/rundir/Debug/bin/64bit",

"args": [
],
"stopAtEntry": false,
"environment": [
{ "name": "LD_LIBRARY_PATH",
"value": "${workspaceFolder}/Build/rundir/Debug/lib" }
],
"externalConsole": false,
"MIMode": "gdb",
"setupCommands": [
{
"description": "Enable pretty-printing for gdb",
"text": "-enable-pretty-printing",
"ignoreFailures": true
},
],
},

Yuriy (@yuriy-chumak) I came across this problem while using vscode in remote connection mode. Because my code is on linux and I work on windows. I'm not sure if this has anything to do with it.

vipcxj

vipcxj same here. I do remote development.

This issue is related to remote debugging using gdb and gdbserver.

I got here because C++ code stepping was extremely slow for me on a local executable using VSCode with cpptools and gdb. Hiding the "Variables" section of the "Run and Debug" sidebar tab made it faster, but obviously was not a solution. Examining the logging/engineLogging logs found many variable lookups taking 0ms, and many others taking 200-400ms each.

After much searching and pain, I found that adding the GCC compiler flag -femit-class-debug-always fixed it for me. Documentation and caveats for the flag are here: https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html

I am using CMake; here is my entry in my CMakePresets.json file.

        {
            "name": "debug",
            "inherits": "default",
            "displayName": "Debug",
            "description": "Configure debug build",
            "cacheVariables": {
                "CMAKE_BUILD_TYPE": "Debug",
                "CMAKE_CXX_FLAGS_DEBUG": "-O0 -g3 -femit-class-debug-always -fno-merge-debug-strings"
            }
        }

Incidentally, this same flag fixed a problem I had trying to switch to using lldb (with the CodeLLDB vscode extension), where lldb was not able to inspect any class members. I had tried to switch to lldb because of the vscode/gdb issue, but to my amazement found that the flag fixed the original issue.

I hope this can help anyone with a similar problem. I am using g++ 12.3 with gdb 12.1, vscode 1.91.1, cpptools v1.22.10 on Ubuntu 22.04. I'm unsure if it is relevant, but I am using an RPATH with $ORIGIN (in CMake: CMAKE_BUILD_RPATH_USE_ORIGIN).

Thank you! This solved the issue for me. Reduced the step time from 10s to instantaneous.

After much searching and pain, I found that adding the GCC compiler flag -femit-class-debug-always fixed it for me.

Thank you! This solved the issue for me. Reduced the step time from 10s to instantaneous.

After much searching and pain, I found that adding the GCC compiler flag -femit-class-debug-always fixed it for me.

This didn't totally fix it for me. I'm still seeing the same issue when debugging an app with around 35 threads. Hitting the initial breakpoint in a function took around 30 seconds to load the "locals" variables. Stepping seemed to be better though, but still not as fast as LLDB through the CodeLLDB extension. Feels very slow when stepping, although I am using a natvis file to display Qt types.

My setup is GCC 11.4 and GDB 15.1.

Using lldb instead of gdb was one workaround for me, but I wasn't satisfied with it.
Finally, I have succeed to fix this issue by moving my vcpkg folder out of my vscode workspace folder. This also fix my auto-completion lag, my "got to declaration" lag and my "go to definition" lag.

An interesting discovery: not setting solib-absolute-prefix and/or sysroot to local library paths seems to have resolved the issues we were facing when remote debugging.

I noticed the original poster is also calling set sysroot <local dir> in his launch config whereas commenters who are reporting no issues on their end don't have it present in their configs.

Completely contradictory since using the local library files should be much more performant than reading from the remote, but luckily it works for our use case.

Now why VSCode struggles with local sysroot and library paths, I have no idea. Other IDE's and command line GDB handles this without issue. I did notice, however, that the Native Debug extension also experiences massive slowdowns with a local sysroot, so whatever issue we were facing seems to be VSCode-wide, rather than isolated to the cpptools extension.

Pretty printers can significantly slow down the debugging experience. Try disabling pretty printers to see if it improves performance: -exec disable pretty-printer

Pretty printers can significantly slow down the debugging experience. Try disabling pretty printers to see if it improves performance: -exec disable pretty-printer

What about the variable formatting in VS Code with pretty printers disabled? Will you still be able to inspect them properly? Disabling PP would be of no use if variables became unreadable.

This issue still exits . Why are they not considering to solve this issue . Atleast provide a feature to control the request send out by the extention . Currently the extention is sending stack traces 0 - 1000 of depth . Atleat provide a felxible method to reduce the depth .

I just cross compiled by gdbserver
GNU gdbserver (GDB) 17.0.50.20250813-git
Copyright (C) 2024 Free Software Foundation, Inc.
gdbserver is free software, covered by the GNU General Public License.
This gdbserver was configured as "arm-linux-gnueabihf"

This is taking about 10 seconds for next command press . With around 8 threads in parallel . Its too slow . The majority of the waiting comes from the request for stack frames from the extenstion .

Navigation Menu

Repository navigation

Very long delay when a breakpoint is hit and many (150+) threads are created by the program #9988

Environment

Bug Summary and Steps to Reproduce

Debugger Configurations

Debugger Logs

54 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Very long delay when a breakpoint is hit and many (150+) threads are created by the program #9988

Description

Environment

Bug Summary and Steps to Reproduce

Debugger Configurations

Debugger Logs

Activity

reneparis commented on Oct 13, 2022

olgapshen commented on Oct 14, 2022

jaemzfleming commented on Oct 24, 2022

rlivings39 commented on Oct 27, 2022

rcontinuum commented on Oct 28, 2022

gewa24 commented on Nov 10, 2022

muddle216 commented on Nov 14, 2022

martymcbro commented on Dec 2, 2022

krysgor commented on Dec 2, 2022

shaharh9110 commented on Jan 17, 2023

crazydef commented on Jan 18, 2023

timmeh87 commented on Jan 23, 2023

er-mo commented on Feb 14, 2023

marcomeyerVFX commented on Feb 19, 2023

Ubachris commented on Feb 21, 2023

54 remaining items

carterols commented on Oct 13, 2024

kyselejsyrecek commented on Oct 13, 2024

yuriy-chumak commented on Oct 14, 2024

r1250gs commented on Oct 14, 2024

vipcxj commented on Oct 17, 2024

carterols commented on Oct 17, 2024

kyselejsyrecek commented on Oct 17, 2024

blackworm27 commented on Oct 22, 2024

kurts-gmail commented on Nov 5, 2024

carterols commented on Nov 6, 2024

zyrkiel commented on Dec 5, 2024

jpvbarbosa commented on Dec 17, 2024

senyai commented on May 1, 2025

georglauterbach commented on May 2, 2025

AbhiramNS1 commented on Aug 13, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions