[Bug][Cloud] RScan_normal ThreadPool idle workers never shrink; OS threads grow far beyond max_threads=512 #66997

Activity

1. Doris-Breakwater commented on Aug 20, 2026

   

   Doris-Breakwater

   on Aug 20, 2026

   Collaborator

   More actions

   Breakwater-GitHub-Analysis-Slot: slot_c701f28ca940

   Preliminary maintainer analysis: this is a real, high-severity process-thread exhaustion report, but the evidence does not yet establish that one RScan_normal pool exceeded its limit or failed to shrink. That interpretation conflicts with the 3.1.4 source path, so the root cause should remain unconfirmed until pool identity and user-space stacks are collected.

   Verified from the 3.1.4 code

   • RScan_normal is the remote scanner scheduler owned by the normal workload-group object. It is created once per workload-group object and backed by the generic Doris ThreadPool (workload_group.cpp).
   • The pool only creates another worker while num_threads + num_threads_pending_start < max_threads (threadpool.cpp). Idle workers above min_threads exit after the pool's 500 ms idle timeout (threadpool.cpp); the 3.1.4 default remote-scan minimum is 8, not 512.
   • thread_pool_active_threads=0 reports workers currently executing tasks; it is not the total number of live workers. 3.1.4 exposes active_threads and max_threads as metrics, but not num_threads. Therefore those two metric values alone cannot show whether the pool retained workers.
   • Each pool metric entity has a unique id label in addition to thread_pool_name and workload_group (threadpool.cpp). That label was omitted from the issue sample and is important here.
   • On Linux, a pthread created inside an RScan_normal worker may inherit the creator's comm name unless it explicitly renames itself. Consequently, ps ... comm=RScan_normal is strong correlation with the scan path but is not proof that every such TID is a worker tracked by this Doris pool.

   Thus, a single intact pool following this code cannot account for 20k tracked workers with max_threads=512. The two leading unverified explanations are (a) multiple live/stale pool instances with the same display name but different metric IDs, or (b) threads created below the scan worker (for example by an I/O/dependency path) that inherited RScan_normal. A third possibility—pool accounting corruption or a build differing from the published tag—needs evidence before being asserted.

   Released fix / PR assessment

   I could not verify any 3.1.x or 3.0.x release or backport that fixes this exact symptom. The inspected 3.0 and 3.1 code has the same per-pool cap and idle-shrink mechanism. I also reviewed #65112; it releases workload-group scheduler-owned queues during BE shutdown and does not demonstrate a fix for periodic runtime creation of thousands of idle OS threads. It should not be presented as the fix for this issue.

   Accordingly, there is currently no evidence-based patch version or PR to recommend for this report.

   Information needed to prove the owner/root cause

   Please collect the following from one affected BE, ideally immediately before and after two scheduled runs:

   1. The complete raw thread_pool_active_threads and thread_pool_max_threads metric lines for RScan_normal, retaining the id label, plus the number of distinct IDs. Do not aggregate away id.
   2. BE log lines containing [topic_publish_wg]update workload group finish and thread info= over the same time window. In 3.1.4 these include [r_scan num:<...>, real_num:<...>, min_num:<...>, max_num:<...>], which directly reports the pool's tracked worker count and configured minimum/maximum.
   3. User-space backtraces (with symbols if available) from 5–10 representative RScan_normal TIDs: include both old TIDs that survived several jobs and TIDs created by the latest job. /proc/.../stack, wchan=futex, or syscall 202 only shows the kernel wait point and is insufficient; the user-space frames above the futex are decisive.
   4. The exact BE build hash/image digest from startup logs, not only the 3.1.4 tag, and the values of doris_max_remote_scanner_thread_pool_thread_num, doris_scanner_min_thread_pool_thread_num, and doris_scanner_thread_pool_thread_num.
   5. normal workload-group ID/version and its max_remote_scan_thread_num / min_remote_scan_thread_num properties, plus the query IDs and timestamps of two reproducing jobs. Also state the storage-vault credential mode (static AK/SK vs. role/default provider) without including any credentials.

   Interpretation of the first two items is straightforward:

   • Multiple metric IDs or workload-group IDs would support multiple pool instances/lifecycle leakage.
   • One metric ID with r_scan num/real_num <= 512 while /proc shows 20k names would prove that most named TIDs are not workers tracked by that pool; their user-space stacks will identify the real owner.
   • One metric ID with r_scan num/real_num > 512 would contradict the published accounting guard and make the exact build plus a core/backtrace essential.

   Until then, reducing job frequency/concurrency and restarting before approaching pids.max are reasonable containment measures. Raising pids.max only delays the abort. Lowering max_remote_scan_thread_num can be used as a controlled diagnostic—check whether the per-run TID delta scales with it—but should not yet be described as a fix.
2. bigo-family commented on Aug 20, 2026

   

   bigo-family

   on Aug 20, 2026

   Author

   More actions

   Thanks for the clarification — we agree that active_threads=0 does not mean OS thread count, and that ps comm=RScan_normal alone does not prove each TID is a ThreadPool worker.
   Here is the evidence we have so far.

   Version / build

   • Doris BE: doris-3.1.4-rc02 / commit 7f5ba43de6c7f148e1dacdb7cb838394d2ec1537
   • Base image: apache/doris:be-3.1.4 (+ only a paimon-s3 jar in custom_lib)
   • Deploy mode: cloud / compute group

   Workload group (SHOW WORKLOAD GROUPS)

   Only one row:

   field	value
   Id / Name	1 / normal
   cpu_share	1024
   memory_limit	30%
   enable_memory_overcommit	true
   max_concurrency	2147483647
   scan_thread_num	-1 (default)
   max_remote_scan_thread_num	-1 (default)
   min_remote_scan_thread_num	-1 (default)
   spill low/high	50% / 80%
   running_query_num / waiting_query_num	0 / 0
   BE log effective values for the same WG:

   • scan_thread_num=48
   • max_remote_scan_thread_num=512
   • min_remote_scan_thread_num=8
   • TG list size = 1

   ThreadPool metrics (single pool id)

   thread_pool_name="RScan_normal" workload_group="normal" id="cc4d753dbd47f8b5-167c63f4c3af5fb3" max_threads=512 active_threads=0

   No second RScan_* pool id observed.

   BE r_scan accounting (repeated)

   [r_scan num:8, real_num:8, min_num:8, max_num:512]

   real_num stays at 8 even when OS has ~20k threads named RScan_normal.

   OS observation (why we originally suspected the pool)

   • Most threads are named RScan_normal and idle on futex
   • Creation pattern: batches of ~85–111 every ~10 minutes after BE start (not one giant burst)
   • Reducing a heavy ADS wide-table job from 1min → 10min clearly slowed growth
   • Eventually hits pids.max (~37776) → Could not create thread (error 11) → BE abort

   Current interpretation (aligned with your note)

   This looks more like case (b): many OS threads inherit the name RScan_normal, but are not the tracked ThreadPool workers (real_num=8).
   We no longer claim “one RScan pool exceeded max_threads=512 / failed to shrink past 512”.

   Still missing (will follow up)

   Userspace backtraces (bt) for:

   • several old RScan_normal TIDs
   • several newly created TIDs from a fresh batch
     Please let us know if any other FE/BE dump would help before that.
3. bigo-family commented on Aug 20, 2026

   

   bigo-family

   on Aug 20, 2026

   Author

   More actions

   Follow-up on userspace stacks:

   BE has ~23k OS threads named RScan_normal while pool accounting stays
   r_scan real_num≈8 / max=512.

   gdb attach at this scale is not practical (info threads hangs / holds TracerPid).
   perf record -t <tid> gets no samples (threads off-CPU).

   Sampled OLD TIDs: 7713, 7784, 7788
   Sampled NEW TIDs: 3625708, 3625721, 3625785
   All: syscall 202 (futex), same userspace PC.

   PC=0x7f232e7f3117 → libc.so.6 text
   map 7f232e78a000-7f232e91f000 r-xp offset 0x28000
   relative=0x91117

   Dynsym bracketing (stripped libc, no debuginfo):
   prev: __nptl_death_event@0x90f90 (delta +0x187 — NOT inside that fn)
   next: __lll_lock_wait_private@0x91230 (~281 bytes after PC)
   also nearby: __lll_lock_wake_private, __pthread_cleanup_routine,
   __nptl_create_event

   So OLD/NEW threads look idle in glibc NPTL low-level wait/futex path,
   not obviously differentiated by PC. Full Doris bt still missing; will
   retry after BE restart when thread count is low, or via off-CPU BPF.

   Question: is this futex-PC + dynsym bracket enough to proceed, or do you
   need a full bt after we restart one BE?
4. bigo-family commented on Aug 21, 2026

   

   bigo-family

   on Aug 21, 2026

   Author

   More actions

   Follow-up evidence (userspace bt + metrics)

   Thanks for the earlier guidance. We now have a full userspace backtrace after restarting one BE (thread count low enough for gdb).

   Version

   • Doris BE 3.1.4-rc02 / commit 7f5ba43de6c7f148e1dacdb7cb838394d2ec1537
   • Image based on apache/doris:be-3.1.4 (+ paimon-s3 jar only)
   • Symbols path: /home/zcp/repo_center/doris_release/doris/

   Workload group

   Only one WG (SHOW WORKLOAD GROUPS):

   • Id=1 Name=normal
   • scan_thread_num / max_remote_scan_thread_num / min_remote_scan_thread_num = -1 (defaults)
   • BE effective: scan=48, max_remote=512, min_remote=8

   Metrics right after sampling (same window as bt)

   Pool id: a44b3294c7572642-ac5de49ef6b4e883 (RScan_normal / workload normal)

   metric	value
   OS threads named RScan_normal	~235
   active_threads	0
   max_threads	512
   queue_size	0
   task_execution_count_total	260
   submit_failed	0
   (Previously, before restart, OS had ~20k RScan_normal while logs showed r_scan real_num:8 — still unexplained vs OS count; current window shows OS≈235 with many idle workers.)

   Userspace bt (valid)

   TID=2283, comm=RScan_normal, gdb bt / bt full:
   #0 futex in libc #1 pthread_cond_timedwait #2..#6 std::condition_variable::wait_for #7 doris::ThreadPool::dispatch_thread (be/src/util/threadpool.cpp:564) #8 std::function::operator() #9 doris::Thread::supervise_thread (be/src/util/thread.cpp:498)

   From bt full:

   • thread name = "RScan_normal [worker]-2283"
   • task.runnable == nullptr (idle)
   • waiting on pool not_empty condition_variable

   Conclusion from our side

   These OS threads are real ThreadPool workers, not inherited-name aliens.
   They sit idle in ThreadPool::dispatch_thread and do not shrink back toward min_remote_scan_thread_num=8 after load (here ~235 idle workers, active_threads=0).
   Over longer runs they accumulate in batches (≈80–110 / ~10 min when heavy ADS job runs), eventually hit pids.max → Could not create thread (error 11) → BE abort.
   Please check ThreadPool grow/shrink for remote scan (RScan_*) and whether pool accounting (r_scan real_num) can diverge from live OS workers / pool recreations.
   Happy to collect any extra dump you need (e.g. second LWP with forced thread switch, or logs around pool create/destroy).
5. bigo-family commented on Aug 21, 2026

   

   bigo-family

   on Aug 21, 2026

   Author

   More actions

   
6. zhang-arvin commented on Aug 21, 2026

   

   zhang-arvin

   on Aug 21, 2026

   More actions

   I'd like to work on this issue.

   👍React with 👍1Reacted by bigo