BE remote scanner thread pool (rs_normal) never shrinks — thread count grows to 27k+ and BE freezes

BE remote scanner thread pool (`rs_normal`) never shrinks — thread count grows to 27k+ and BE freezes

Environment

Doris version: 4.1.1-rc01
Cluster size: 4 BE + 3 FE, deployed via DorisOperator on Kubernetes
BE image: official apache/doris:doris-4.1.1-rc01-be (x86_64)
Workload: BE queries external tables backed by Paimon on S3 (Doris × Paimon catalog), mixed with normal internal OLAP queries
BE resources: cpu 8 core, memory limit 40G (cgroup), mem_limit=32G

Symptom

BE pod RSS climbs monotonically from ~3G to 6–7G over ~4h, then either:

keeps growing until the pod is OOM-ish and gets killed by cgroup, or
at ~27k threads the BE process freezes: it stops responding to FE heartbeats, SHOW BACKENDS shows Alive=false with SocketTimeoutException: Read timed out, CPU drops to ~1m, readiness probe times out. Not OOMKilled — internal lock/resource exhaustion.

The only recovery is kubectl delete pod (restart the BE process).

Misleading signals

All conventional memory metrics stay green, which made this look like a "memory leak" for a while:

JVM heap used — normal GC sawtooth, returns to baseline
jemalloc resident — stable at ~2G
jvm non-heap — stable at ~130M
mem_tracker — stable

Root cause

/proc/$BE_PID/status shows Threads=16000–27000, of which ~90% are rs_normal [worker] threads in S (sleeping) state. Each thread has a 512K stack → 7–12GB just for thread stacks, which is what pushes RSS up.

Thread census command:

BE_PID=$(pgrep -f doris_be)
ls /proc/$BE_PID/task/ | while read t; do cat /proc/$BE_PID/task/$t/comm; done \
  | sort | uniq -c | sort -rn | head

Typical output on a frozen BE:

27012 rs_normal
   48 olap_scan
   24 brpc
   ...

The rs_normal pool (remote scanner, used for querying external catalogs like Paimon/Hive/Iceberg) only grows, never shrinks. Idle keep-alive eviction either isn't implemented or isn't effective in 4.1.1-rc01.

Configs I tried to cap the pool (all ineffective on rc01)

#	Where	Setting	Result
1	`be.conf`	`doris_scanner_thread_pool_thread_num=256`	Only caps `ls_normal` (local scanner). `rs_normal` still linear-grows.
2	`be.conf`	`doris_max_remote_scanner_thread_pool_thread_num=256`	No effect. `rs_normal` keeps growing past 256.
3	FE	`ALTER WORKLOAD GROUP normal PROPERTIES ('max_remote_scan_thread_num'='256', 'min_remote_scan_thread_num'='48');`	No effect.

Related defaults on this BE:

doris_remote_scanner_thread_pool_thread_num=48 (initial)
doris_max_remote_scanner_thread_pool_thread_num=-1 (unlimited)

Current workaround

CronJob that rotates one BE pod every 4h (kubectl delete pod on the oldest ready BE, with guards: ready BE >= 3 and pod age > 6h). This keeps the leak from ever reaching the freeze threshold, but it's purely symptomatic.

YAML for the workaround is available if useful.

Expected behavior

At least one of these should actually work on rc01:

doris_max_remote_scanner_thread_pool_thread_num=N should hard-cap rs_normal size, excess idle threads should be reaped when the cap is hit and load drops.
Idle rs_normal workers should be evicted after a keep-alive timeout (like ls_normal does), so the pool shrinks back to doris_remote_scanner_thread_pool_thread_num when external-scan load stops.
ALTER WORKLOAD GROUP … max_remote_scan_thread_num=N should bound concurrent remote scans per workload group — currently appears to be ignored or not enforced against the rs_normal pool.

Reproduction

Fresh BE 4.1.1-rc01 with a Paimon catalog on S3.
Run a sustained mixed workload that issues external scans against Paimon (a few QPS is enough; the pool grows even under modest load).
Within 4–8h, Threads= in /proc/$BE_PID/status crosses 20k and RSS climbs correspondingly.
Eventually (usually 24k–28k threads) the BE stops answering FE heartbeats and has to be killed externally.

Happy to provide thread dumps, be.conf, SHOW BACKENDS/SHOW WORKLOAD GROUPS output, or a heap profile if it helps. Just let me know what's most useful.

Question

Is this a known issue in 4.1.1-rc01? Is there a fix in a later 4.1.x or 4.2.x release? If a fix exists, I'd like to upgrade instead of running the rotator CronJob forever.

It is not a known issue. How many CPU cores do you have in your host machine. And could you please provide some be logs from be startup until this issues happens.

@yiguolei

Hi @yiguolei — thanks for taking a look. Answers to both questions below.

1. Host CPU cores

4 BE pods run on 4 different nodes, every node has 32 cores at the host level (status.capacity.cpu = 32).
BE container cgroup limits: 8 cores cpu, 40 GB memory (cgroup), mem_limit = 32G in be.conf.
Worth flagging: when I run SHOW BACKENDS\G from the FE, every BE reports CpuCores: 32 — i.e. the BE is reporting host CPU to the FE, not the cgroup-limited 8. If the remote scanner pool is sized/expanded based on CpuCores, then in a containerized env it'll be over-provisioned 4x from the start, which lines up with the runaway growth we see.

2. BE logs from startup → capture

Captured from doris-be-0 after 4h29m of runtime — not yet at the freeze point, but the leak is already clearly visible. Public gist with the artifacts:

https://gist.github.com/tracy1014-hub/8df6ab002a509132d7ffbac27d0f88d4

Inside the gist:

INSTRUCTIONS.md — layout + how to decode the tarball
runtime-snapshot.txt — plain-text snapshot, browse inline without decoding anything
be.conf — full BE config, browse inline
doris-issue-65416.tar.gz.b64 — base64-encoded tarball, decode with base64 -d doris-issue-65416.tar.gz.b64 | tar xzf - to get:
- be.INFO.log.20260710-071747 — 24 MB, full startup → capture
- be.out — startup banner, JNI init
- be.WARNING.log.20260710-071748
- be.gc.log.20260710-071747 — JVM GC log
- jni.log — 3.2 MB, Paimon scanner activity via JNI. Every Paimon scan spawns a new Thread-N and re-installs HadoopModule + CodecPool.getDecompressor — no JVM-side scanner reuse, which lines up with rs_normal never shrinking.
- be.conf and runtime-snapshot.txt (same as the readable inline copies above)

Snapshot at capture time (pod age 4h29m, BE still `Alive=true`, `HeartbeatFailureCounter=0`)

/proc/950/status:
  Threads: 17192
  VmRSS:    5004868 kB  (~4.77 GB)
  VmSize:   57360836 kB

thread census (top 5):
  15099 rs_normal [work     <-- 87% of all threads, 59x the configured 256 cap
    512 brpc_arrow_flig
    381 doris_be
    256 ls_normal [work     <-- local scanner, 256 cap is enforced here
    128 brpc_light

cgroup memory.current: 5.8 GB

So at 4h29m we already have 15,099 rs_normal threads — doris_max_remote_scanner_thread_pool_thread_num = 256 in be.conf is clearly not being honored. The pod is ~7k threads short of the freeze threshold (24k–28k) we usually see, so it would have frozen in another ~1–2h on its own.

What I can do next

If startup-to-freeze is more useful than startup-to-now, I can:

kubectl patch cronjob doris-be-rotator -p '{"spec":{"suspend":true}}' to disable the workaround
Let doris-be-0 run uninterrupted until it freezes (~1–2h from now given current trajectory)
Capture be.INFO + jni.log at the moment of freeze + a final SHOW BACKENDS\G showing Alive=false

Just say the word and I'll do it. If the current 4h29m log is enough for you to start tracing where the rs_normal workers are being created / why the cap isn't enforced, no need for the freeze run.

add this config to be.conf and restart be. num_cores = 8
and check the thread num

@yiguolei

Hi @yiguolei — added num_cores = 8 to be.conf and restarted doris-be-0. Reporting back the results.

Verification that the config took effect

SHOW BACKENDS\G from FE confirms the new doris-be-0 reports CpuCores: 8 (vs CpuCores: 32 on the other 3 BEs that I left as a control group):

*************************** 1. row ***************************
                   Host: doris-be-0.doris-be-internal.paimon-prod.svc.cluster.local
         LastStartTime: 2026-07-10 12:49:33
                  Alive: true
 HeartbeatFailureCounter: 0
                CpuCores: 8      <-- was 32 before
                  Memory: 40.00 GB

*************************** 2. row ***************************    (doris-be-1, control)
                CpuCores: 32     <-- unchanged
*************************** 3. row ***************************    (doris-be-3, control)
                CpuCores: 32     <-- unchanged
*************************** 4. row ***************************    (doris-be-2, control)
                CpuCores: 32     <-- unchanged

So the lever does what you expected at the report-to-FE level: num_cores = 8 overrides the host CPU count in CpuCores. But the actual leak behavior is unchanged — see below.

rs_normal growth: experiment vs control (~30 min in)

I kept doris-be-1/2/3 on the default config (host = 32 cores, CpuCores: 32) as a control group, and ran all 4 BEs in parallel under the same workload. Side-by-side snapshot at experiment T+30 min:

BE	`num_cores`	uptime	rs_normal	Threads	rate (rs_normal/min)
doris-be-0 (exp)	8	29m54s	1,744	3,761	58.3
doris-be-1 (ctrl)	32 (default)	3h01m	10,382	12,475	57.1
doris-be-2 (ctrl)	32 (default)	5h01m	14,975	17,071	49.6
doris-be-3 (ctrl)	32 (default)	4h02m	16,571	18,667	68.4

Control group avg rate: 58.4 rs_normal/min. Experiment rate: 58.3 rs_normal/min. Essentially identical.

rs_normal trajectory on doris-be-0 (num_cores=8) over 30 min

pod age	rs_normal	Threads	instantaneous rate
0m39s	8	2,014	—
3m10s	164	2,181	62/min
4m06s	224	2,241	60/min
9m07s	494	2,511	54/min
14m10s	794	2,811	60/min
19m13s	1,089	3,106	59/min
24m15s	1,389	3,406	60/min
29m18s	1,684	3,701	59/min

Growth is linear at ~60/min and shows no sign of saturating. If the 256 cap from doris_max_remote_scanner_thread_pool_thread_num = 256 were being honored, rs_normal should have stopped at 256 — instead it's at 1,684 (6.6× the cap) and still climbing linearly.

Observations

num_cores = 8 changes CpuCores but not the leak. The hypothesis that the rs_normal pool is sized on CpuCores doesn't hold — or at least num_cores alone isn't the lever that bounds the remote scanner pool in 4.1.1-rc01.
The 256 cap is still not enforced under num_cores = 8 — rs_normal passed 256 within the first ~5 minutes and kept climbing linearly. So whatever path is spawning rs_normal workers, it isn't consulting doris_max_remote_scanner_thread_pool_thread_num at all (with or without num_cores).
Other thread pool sizes (brpc, ls_normal, EvHttpServer, etc.) also appear unchanged between the experiment BE and the control BEs — num_cores = 8 doesn't seem to affect any pool sizing I can observe.

Cluster status

All 4 BEs are Alive=true, HeartbeatFailureCounter=0. The 3 control BEs are at 12.5k–18.7k threads each — still below the freeze threshold (~27k) we usually see, but doris-be-3 (highest, 18.7k at 68/min) will approach the threshold in ~2h. I have a rotator CronJob that will pick up the oldest BE at the 6h-age mark, and I'll manually rotate doris-be-3 before then if its thread count crosses ~22k.

What's next?

Standing by for direction. Things I can do on request:

Provide be.INFO + jni.log from the experiment BE at any point if you want to trace where the rs_normal workers are being created.
Try other config levers (e.g. doris_remote_scanner_thread_pool_thread_num, doris_scanner_thread_pool_queue_size, parallel_remote_scan_thread_num if it exists in 4.1.1) — let me know which to flip and what value to set.
Let one of the control BEs freeze naturally and capture the freeze-time log + SHOW BACKENDS\G showing Alive=false (this is what I offered in the previous comment). With doris-be-3's current trajectory that would be in ~2h.
Revert num_cores = 8 if you'd prefer to keep the cluster on default config while you investigate.

Quick status update — continuing to monitor, no new findings yet, but a freeze-time capture is coming up.

Experiment status (T+46 min on doris-be-0 with `num_cores = 8`)

rs_normal=2729 threads=4746 rss_kb=2878644 uptime_sec=2804

Rate still ~58/min, exactly matching the control group. No change from the previous update.

Control group snapshot

BE	`num_cores`	uptime	rs_normal	Threads	rate (rs_normal/min)
doris-be-0 (exp)	8	46m44s	2,729	4,746	58.3
doris-be-1 (ctrl)	32	3h18m	11,362	13,455	57.2
doris-be-2 (ctrl)	32	5h18m	15,770	17,866	49.6
doris-be-3 (ctrl)	32	4h18m	17,736	19,832	68.4

Opportunity to capture freeze-time data (the "Path B" from my earlier offer)

doris-be-3 is on track to hit the ~27k-thread freeze threshold in roughly 1h45m (around 15:21 CST, i.e. ~07:21 UTC), assuming the rate holds. The rotator's 6h-age threshold won't pick it up before then (its 6h mark is at 15:17, but it'd still be 5h59m at that minute — the rotator skips and the next run is 16:17, too late).

I'll let doris-be-3 run uninterrupted and capture, at the moment it freezes:

be.INFO from startup to freeze
jni.log over the same window
SHOW BACKENDS\G showing Alive=false + HeartbeatFailureCounter > 0
/proc/$BE_PID/status final Threads= count
thread census showing rs_normal as the dominant thread type

I'll publish those as a follow-up here. If you'd rather I rotate doris-be-3 before it freezes (e.g. you've already seen enough freeze signatures and want me to keep the cluster clean), say the word and I'll rotate it at the ~22k mark.

If, in the meantime, there are other config levers you'd like me to try on doris-be-0 (which is still running with num_cores = 8 and growing linearly), I can patch them in parallel without disturbing the freeze capture on doris-be-3.

using curl be_ip:web_server_port/metrics to get all be metrics to a file and send me the result when there are a lot of threads.

@yiguolei

Hi @yiguolei — captured /metrics from two BEs in parallel at the same workload moment, in response to your request. Gist with the raw files + a summary:

https://gist.github.com/tracy1014-hub/6c8f05326068a446cca6403694292bc7

Files in the gist:

doris-be-0.metrics — experiment BE with num_cores = 8. doris_be_process_thread_num = 8001.
doris-be-3.metrics — control BE (default config, 32-core host). doris_be_process_thread_num = 23502, was just over the 22k "approaching freeze" threshold at capture time.
SUMMARY.md — observations.

rs_normal pool metrics (the smoking gun)

Side-by-side for the rs_normal pool only:

metric (rs_normal)	be-0 (`num_cores=8`)	be-3 (default)
`doris_be_thread_pool_max_threads{...rs_normal...}`	256	256
`doris_be_thread_pool_active_threads{...rs_normal...}`	0	0
`doris_be_thread_pool_queue_size{...rs_normal...}`	0	0
`doris_be_thread_pool_task_execution_count_total{...rs_normal...}`	4,798	15,297
`doris_be_thread_pool_task_wait_worker_count_total{...rs_normal...}`	4,798	15,297
`doris_be_thread_pool_submit_failed{...rs_normal...}`	0	0
`doris_be_process_thread_num` (whole BE)	8,001	23,502
rs_normal count in `/proc/$PID/task/` (OS reality)	~6,039	~21,392
OS-thread / task-count ratio	1.26	1.40

What this tells us

The pool reports max_threads = 256 correctly — matches the doris_max_remote_scanner_thread_pool_thread_num = 256 override in be.conf. So the config is being read into the pool's reported state on both BEs, with or without num_cores = 8.
The pool reports active_threads = 0 — i.e. it thinks no workers are currently busy. queue_size = 0, submit_failed = 0. From the pool's own POV everything is healthy and idle.
But /proc/$PID/task/ shows thousands of rs_normal threads actually living in the OS — ~6k on be-0, ~21k on be-3 — that the pool doesn't account for.
The ratio of OS rs_normal thread count / task_execution_count_total is ~1.26–1.40 on both BEs. Either (a) each task execution causes the pool to spawn a worker that's never reaped, or (b) there's a thread-creation path that bypasses the task_execution_count counter entirely.

Net: the cap (256) is being recorded into the pool's reported state, but it isn't being consulted when threads are spawned. Some path is creating rs_normal worker threads without going through (or without being tracked by) the pool's bookkeeping that owns the max_threads cap. That's why the 256 setting is dead letter — the enforcement point exists, but the spawning path doesn't pass through it.

num_cores = 8 vs default — still no difference

Both BEs report max_threads{rs_normal} = 256 and active_threads{rs_normal} = 0. The other thread pool sizes (EvHttpServer=128, FragmentMgrAsyncWorkThreadPool=2048, ls_normal=256, mf_normal=128, etc.) are also identical between the two BEs — confirming that num_cores = 8 in 4.1.1-rc01 changes the CpuCores field reported to the FE but doesn't actually influence any pool's sizing.

Live state at capture time

doris-be-0 (exp, num_cores=8, uptime ~1h45m): rs_normal=6224, threads=8241, growing at ~60/min
doris-be-3 (ctrl, default, uptime ~5h15m): rs_normal=21392, threads=23502, growing at ~75/min, was already past the 22k warning threshold at capture

The doris-be-3 BE is still running (not yet frozen). I'm letting it run toward the ~27k freeze threshold so I can also provide the freeze-time be.INFO + SHOW BACKENDS\G with Alive=false as I previously offered. Let me know if you'd rather I rotate it before then and skip the freeze capture — the metrics + the earlier 4h29m log may be enough for you to find the leak path.

using pstack DORIS_BE_PID to get all stacks and send me the file.

Quick status note — pstack capture is in progress, freeze-time artifacts captured in parallel.

pstack status

Started gdb -batch -ex "set pagination off" -ex "thread apply all bt" -p 955 on doris-be-3 at 14:55 CST (06:55 UTC). The BE process has 25,035 threads at attach time, so the full thread apply all bt is taking a while — gdb has consumed ~7 min of CPU so far (utime=361s, stime=70s) and is still actively walking thread stacks. Estimated 20–30 min more to finish.

Output is being piped through gdb | gzip | base64 so the file stays at 0 bytes until gdb finishes writing. I'll publish the pstack file in a follow-up comment as soon as it's done.

Freeze-time artifacts captured meanwhile (while gdb is still attached)

/proc/$BE_PID/status:

Name:   doris_be
State:  t (tracing stop)     <-- ptrace stop, gdb is attached
Pid:    955
VmSize: 67149576 kB          (~64 GB)
VmRSS:  7094288 kB           (~6.8 GB)
Threads: 25035

Thread census (top 10) from /proc/$BE_PID/task/:

  22945 rs_normal [work     <-- 91.6% of all threads
    512 brpc_arrow_flig
    378 doris_be
    256 ls_normal [work     <-- local scanner, capped at 256 as configured
    128 brpc_light
    128 brpc_heavy
    128 EvHttpServer [w
     64 p_normal_blocki
     64 SendBatchThread
     64 DownloadThreadP

SHOW BACKENDS\G from FE — doris-be-3 row:

                   Host: doris-be-3.doris-be-internal.paimon-prod.svc.cluster.local
         LastStartTime: 2026-07-10 09:17:42
                  Alive: false
 HeartbeatFailureCounter: 17      (climbing — gdb still attached, BE not heartbeating)
                CpuCores: 32
                  Memory: 40.00 GB

The other 3 BEs (including doris-be-0 with num_cores = 8) are Alive=true, HeartbeatFailureCounter=0. Cluster-wise only doris-be-3 is affected.

Note on the freeze

This freeze is gdb-induced (the BE process is in ptrace stop while gdb walks 25k thread stacks), not the natural leak-induced freeze at ~27k threads. The BE crossed 25k threads during gdb attach. When gdb finishes and detaches, the BE should SIGCONT — I'll watch whether it recovers or stays frozen.

If it recovers, I'll let it run toward 27k to capture the leak-induced freeze state separately. If it doesn't recover (HEARTBEAT_FAIL keeps climbing after gdb detaches), I'll delete the pod and we'll have lost the freeze-time capture but the gdb pstack + the artifacts above are still good.

be.INFO (29 MB, startup → freeze), jni.log (4.8 MB), be.WARNING (107 KB), be.gc.log (3.3 MB), and be.out (881 B) have all been pulled off the pod and will be published together with the pstack file once gdb is done.

@yiguolei

Hi @yiguolei — pstack captured. Gist with the raw pstack + analysis:

https://gist.github.com/tracy1014-hub/7ddbc6340653ead0d701d6ba26550577

Files:

pstack.txt — 3.5 MB, partial pstack of doris-be-3 (3,295 of 25,035 threads captured, 26% coverage)
ANALYSIS.md — key findings + recommended next step

Why partial

gdb -batch -ex "thread apply all bt" -p 955 ran for ~33 min and had only walked 26% of the 25k threads (gdb is slow on this many threads — 100% CPU, ~7 min of CPU per 1k threads). I killed it before completion because waiting for full coverage would have taken hours and you'd be waiting on this reply. The 3,295 threads captured are statistically representative — see below.

Headline finding

Every single thread in the pstack is rs_normal, and 3,294 / 3,295 (99.97%) have an identical backtrace:

Thread 25035 (Thread 0x7f9dc6cb9640 (LWP 443684) "rs_normal [work"):
#0  pthread_cond_wait                                 from /lib/x86_64-linux-gnu/libc.so.6
#1  Parker::park(bool, long)                          from /usr/lib/jvm/java/lib/server/libjvm.so
#2  Unsafe_Park                                       from /usr/lib/jvm/java/lib/server/libjvm.so
#3  ?? (JIT-compiled)
#4  ?? (JIT-compiled)
#5  OptoRuntime::new_instance_C(Klass*, JavaThread*)  from /usr/lib/jvm/java/lib/server/libjvm.so

(1 thread uses pthread_cond_timedwait instead of pthread_cond_wait, otherwise identical.)

Diagnosis

The rs_normal threads are not C++ threads — they're JVM threads (created by the JNI scanner, almost certainly Paimon) parking on Java's Unsafe_Park primitive waiting for work.

This reconciles everything we've seen:

doris_be_thread_pool_active_threads{rs_normal} = 0 — correct from the C++ pool's POV; none of these threads are in that pool.
doris_be_thread_pool_max_threads{rs_normal} = 256 — the cap exists on the C++ pool, but the 22,945 OS threads named rs_normal were never created by that pool. They share the thread name (probably because the JNI scanner uses the same naming convention) but bypass the C++ pool entirely.
task_execution_count_total{rs_normal} = 15,297 on be-3 (vs 22,945 actual threads) — the C++ pool has executed ~15k tasks, but ~22.9k JVM threads exist. The two numbers aren't directly comparable because they're counted by different accounting systems (one C++, one JVM), but the JVM-side count being higher is consistent with the JVM spawning threads that the C++ pool never sees.
num_cores = 8 had no effect on rs_normal growth — because the leak isn't on a path that consults num_cores. The leak is in the JNI scanner's Java-side thread creation logic.

Where to look in the code

The leak path is in the Paimon JNI scanner (or whichever external-catalog scanner is being used), specifically:

A code path that creates a new Java Thread (or submits to an unbounded ExecutorService) per external scan, parks it on Unsafe_Park waiting for work, but never interrupt()s or join()s it when the scan completes. The parked thread waits forever and accumulates.
The OptoRuntime::new_instance_C frame a few levels deeper suggests the parked thread also allocates Java objects while waiting — possibly a fresh Runnable / Callable per scan that gets GC'd but the carrier thread stays around.
Worth grepping for LockSupport.park, BlockingQueue.take, Unsafe.park in the Paimon scanner JARs ($DORIS_HOME/lib/java_extensions/paimon-scanner/) — find the parking site, then check whether the corresponding unpark / put / notify site is reachable on scan completion.

Cluster state

After I killed gdb with kill -9, the BE process was left in T (stopped) state (gdb didn't get a chance to send SIGCONT before dying). A manual kill -CONT 955 from inside the container woke it up — it immediately went back to S (sleeping), FE marked it Alive=true, HeartbeatFailureCounter=0, pod back to 1/1 Running. So the freeze during pstack was purely ptrace-induced, not a real leak-induced freeze at ~27k threads.

However, the k8s liveness probe had failed during the 38-min gdb pause (BE was unresponsive because ptrace-stopped), so k8s restarted the be container shortly after SIGCONT. The pod is now 1/1 Running with RESTARTS=1, BE_PID=992 (was 955), and rs_normal count reset to 8 (fresh start). So the natural-freeze capture I was hoping to get from doris-be-3 won't happen on this pod instance — it would take another 5+ hours of growth.

If you'd like the freeze-time be.INFO + SHOW BACKENDS\G with Alive=false from a natural ~27k-thread freeze, I can either:

Disable the liveness probe on doris-be-3 so it doesn't get killed during the next gdb attach, then re-run pstack on the natural freeze
Or just let doris-be-3 (or one of the other control BEs) run to natural freeze without gdb interference — the rotator is currently suspended, and the natural-freeze log would be different from the gdb-induced freeze log

Let me know which (if any) you want. Otherwise the pstack analysis above is the main deliverable.

It seems they are java threads. Doris use JNI to access paimon file. There should be a jvm inside doris be process. You could find the JVM ID and then you should use jstack to get jvm's stack.

@tracy1014-hub

@tracy1014-hub Do you have a wechat？

And you can also use jmap to print the object num in JVM

@yiguolei

Hi @yiguolei — jstack and jmap captured. You're spot-on: these are Java threads in the embedded JVM inside doris_be. Public gist:

https://gist.github.com/tracy1014-hub/1af1032ea8b8cc08b6cf3d28d790abfe

Files:

jstack-doris-be-0.txt.gz.b64 — full jstack 994 output (15,080 Java threads). Decode: base64 -d file.b64 | gunzip > jstack.txt. jstack ran in 6 seconds total — vastly faster than gdb.
jmap-histo-doris-be-0.txt — jmap -histo 994 output (Java object histogram).
JVM-ANALYSIS.md — analysis.

Thread name distribution (15,080 total Java threads)

7,324 sdk-ScheduledExecutor-N-M     (47.7% of all Java threads)
   51 Thread-N
    1 each: main, Reference Handler, Finalizer, Signal Dispatcher, Service Thread,
          Monitor Deflation Thread, C2 CompilerThread0, Notification Thread,
          pool-1-thread-1, Sweeper thread, MutableQuantiles,
          mysql-cj-abandoned-connection-cleanup,
          org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner,
          Timer for 's3a-file-system' metrics system

7,324 of the 15,080 Java threads are workers in ScheduledThreadPoolExecutor instances, all named sdk-ScheduledExecutor-N-M with unique (N, M) pairs. Each unique N is a distinct ScheduledThreadPoolExecutor instance — so we have thousands of leaked executors in the JVM.

All 7,324 parked threads have an identical stack

Unsafe.park (Native Method)
  - parking to wait for <0x...> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
LockSupport.park (LockSupport.java:341)
AbstractQueuedSynchronizer$ConditionNode.block (AbstractQueuedSynchronizer.java:506)
ForkJoinPool.unmanagedBlock (ForkJoinPool.java:3463)
ForkJoinPool.managedBlock (ForkJoinPool.java:3434)
AbstractQueuedSynchronizer$ConditionObject.await (AbstractQueuedSynchronizer.java:1623)
ScheduledThreadPoolExecutor$DelayedWorkQueue.take (ScheduledThreadPoolExecutor.java:1177)
ScheduledThreadPoolExecutor$DelayedWorkQueue.take (ScheduledThreadPoolExecutor.java:899)
ThreadPoolExecutor.getTask (ThreadPoolExecutor.java:1062)
ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1122)
ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
java.lang.Thread.run (Thread.java:833)

All parked on ScheduledThreadPoolExecutor$DelayedWorkQueue.take — waiting for tasks that never arrive. The executors are never shutdown()'d, so their workers never exit.

jmap histogram confirms the leak

Top objects by bytes/instances:

#	instances	bytes	class
1	569,928	521 MB	`byte[]`
2	5,631,659	135 MB	`org.apache.paimon.data.Timestamp`
3	1,524,665	85 MB	`java.nio.HeapByteBuffer`
5	738,882	24 MB	`java.util.HashMap$Node`
6	634,381	20 MB	`javax.management.MBeanAttributeInfo`
8	634,314	15 MB	`javax.management.Attribute`
9	577,459	14 MB	`org.apache.hadoop.metrics2.impl.MetricCounterLong`
14	75,170	7.8 MB	`java.lang.management.ThreadInfo`
15	15,053	5.5 MB	`java.lang.Thread`
18	90,373	2.9 MB	`org.apache.paimon.shade.org.apache.parquet.io.api.Binary$ByteBufferBackedBinary`
20	61,335	2.0 MB	`org.apache.paimon.memory.MemorySegment`

Key signals:

java.lang.Thread instances: 15,053 — matches the OS-level rs_normal thread count (15,174) within timing delta. Byte-for-byte confirmation that the rs_normal OS threads are Java Thread objects.
org.apache.paimon.data.Timestamp 5.6 M instances — Paimon allocates a Timestamp per row (or per scan?) and they're not being GC'd. Suggests a ThreadLocal cache or per-scan HashMap that holds Timestamps and is never cleared.
javax.management.MBeanAttributeInfo 634k + MetricCounterLong 577k — Hadoop metrics2 MBeans. The S3A client registers MBeans per S3AFileSystem instance. 634k MBean attributes is consistent with thousands of distinct S3A client instances — every Paimon scan creates a new S3A client → new MBean registration → never unregistered.
byte[] 521 MB across 570k instances — Paimon data buffers / Parquet column payloads.
AQS$ConditionNode ~25k instances (in the full histogram) — corresponds to the ~15k parked workers (each holds one AQS$ConditionNode).

Diagnosis

The leak path:

Every external scan against the Paimon catalog creates a fresh Hadoop S3AFileSystem instance (with its own S3 client, ScheduledThreadPoolExecutor for connection pool management, MBean registration for Hadoop metrics2, etc.).
The S3AFileSystem is never close()'d, so its ScheduledThreadPoolExecutor is never shutdown()'d, and its MBeans are never unregistered.
Each leaked executor creates at least one worker thread named sdk-ScheduledExecutor-N-M, parked on DelayedWorkQueue.take waiting for tasks that never come.
Over hours of sustained Paimon scan traffic, the count of leaked ScheduledThreadPoolExecutor instances grows linearly with query rate. By 4-5 hours, we have 7,324 leaked executors → 15k parked worker threads → ~5-7 GB RSS just for thread stacks → freeze or OOM.

The jni.log from earlier also showed this pattern — every Paimon scan re-runs HadoopSecuredFileSystem.trySecureFileSystem + HadoopModule.install + CodecPool.getDecompressor from scratch, no caching of the FileSystem or compressor across scans.

Where to look in the code

The fix has to be on the Paimon JNI scanner side, not in the C++ remote scanner pool. Likely places to look:

Paimon scanner JARs under $DORIS_HOME/lib/java_extensions/paimon-scanner/ — search for classes that create S3AFileSystem, AmazonS3Client, S3Client, or call FileSystem.get(...). Each call site should be checked for whether the returned FileSystem is cached/shared vs. created fresh.
The Hadoop FileSystem class has a global FileSystem.CACHE keyed by URI + UserGroupInformation — if every Paimon scan uses a slightly different URI (e.g., different query parameters, different path prefix), the cache will hold a new entry per scan, and each entry holds a ScheduledThreadPoolExecutor. Worth checking org.apache.hadoop.fs.FileSystem$Cache$CacheCleaner activity and the size of FileSystem.CACHE.
Paimon-side fix: reuse a shared S3AFileSystem (or at least call close() / shutdown() on the per-scan instance when the scan completes).

The C++ remote scanner pool is healthy and bounded (max_threads{rs_normal}=256, active=0, task_execution_count=4,798). The leak is entirely on the JVM side.

WeChat

Will DM you about this separately — I want to keep the technical thread clean.

Live state

doris-be-0 (the experiment BE with num_cores = 8) is still running and growing. Latest monitor sample: rs_normal=15,174, threads=17,197 at T+3h38m. It's on track to hit the natural ~27k-thread freeze in roughly 2-3 more hours. I'm happy to either let it run to capture the natural-freeze be.INFO, or rotate it now that we've nailed the diagnosis — your call.

@tracy1014-hub

@tracy1014-hub Do you use paimon on HDFS or on S3？

@tracy1014-hub Do you use paimon on HDFS or on S3？

paimon on S3

@tracy1014-hub

@tracy1014-hub
have look this issue, maybe u could have a temp test to alter the catalog property. eg: fs.xxx.impl.disable.cache, and xxx could be oss, cos, s3, s3a depends your bucket.

mysql>   ALTER CATALOG paimon_oss_ctl
    ->   SET PROPERTIES (
    ->     "fs.oss.impl.disable.cache" = "false"
    ->   );

pls let me know how the testing goes. If this fix works, will dive deeper and figure out a more complete solution.

@tracy1014-hub

@tracy1014-hub we have find the problem and will solve this problem in next version maybe version 4.1.4.

@tracy1014-hub we have find the problem and will solve this problem in next version maybe version 4.1.4.
Thanks, could u tell me the root cause? I will try to solve it by myself.

In Doris, we disable filesystem cache by default.
Paimon use HDFS over S3 sdk to access file on S3.
In HDFS, it will generate a file system instance for every split and will save the file system object in a global CACHE.
There maybe a policy to evit the expired filesystem in global CACHE object.
But if the query has a lot of split, it will generate a lot of file system object in short time. And every file system object will create a thread pool.

@tracy1014-hub

#65586 @tracy1014-hub we have tested this PR, it is ok. If you can compile Doris in your local machine, you can have a try.

OK, I will try to fix it, Thanks

@yiguolei

Update: PR #65586 helps but doesn't fully fix rs_normal growth — real root cause is `fs.s3a.connection.request.timeout=60s` (hadoop-aws SDK v2 timer)

@yiguolei @zhangstar333 following up on my paimon-prod (Paimon 1.4.1 on S3, Doris 4.1.1-rc01) reproduction. Compiled PR #65586 locally as suggested and tested it side by side with vanilla — TL;DR: PR #65586 fixes the S3AFileSystem cache layer (correctly and provably), but it does not stop rs_normal from growing at ~80/min in our Paimon-only workload. The real leak source is one level deeper.

Test 1 — PR #65586 alone

Built patched hadoop-deps.jar + doris-fe.jar from apache/doris#65586 head (453115a4), overlaid on apache/doris:be-4.1.1. Rolled to one BE (be-3). Observed for ~15h side-by-side against three unpatched BEs.

jmap -histo:live on the patched BE showed the FS layer is fixed:

class	vanilla BE (rs≈27k)	PR #65586 BE (rs≈27k)
`org.apache.hadoop.fs.s3a.S3AFileSystem`	5,000+	6
`org.apache.hadoop.fs.s3a.S3AInstrumentation`	5,000+	6
`software.amazon.awssdk.services.s3.DefaultS3Client`	~5,000	13

So doris.fs.cache.key is doing exactly what it should — FS instances are being reused. But rs_normal (i.e. sdk-ScheduledExecutor-* worker threads) kept climbing at the same rate as the unpatched BEs, and hit the 25k+ freeze threshold in ~4h.

The real numbers on the patched BE:

class	count
`java.util.concurrent.ScheduledThreadPoolExecutor`	5,895
`software.amazon.awssdk.utils.NamedThreadFactory`	5,536
`java.util.concurrent.ThreadPoolExecutor$Worker`	29,473
`java.lang.Thread`	29,533

MAT merge_shortest_paths on those STPEs shows the GC root is each STPE's own daemon worker Thread (sdk-ScheduledExecutor-N-M) parked on DelayedWorkQueue.take — classic strong-ref cycle.

Test 2 — JFR ThreadStart sampling: where do new STPEs come from

60-second JFR trace on be-3 captured 190 new sdk-ScheduledExecutor thread starts. Every single stack was identical:

java.util.concurrent.ThreadPoolExecutor.addWorker
  ← ScheduledThreadPoolExecutor.ensurePrestart
  ← ScheduledThreadPoolExecutor.delayedExecute
  ← ScheduledThreadPoolExecutor.schedule
  ← software.amazon.awssdk.core.internal.http.timers.TimerUtils.timeSyncTaskIfNeeded  ← source

Looking at TimerUtils.timeSyncTaskIfNeeded (SDK v2):

if (timeout <= 0) return NoOpTimeoutTracker.INSTANCE;   // ← short-circuit
// otherwise schedule the timer, which addWorker's a new daemon thread

The timeout is fs.s3a.connection.request.timeout → apiCallTimeout / apiCallAttemptTimeout on ClientOverrideConfiguration. hadoop-aws 3.4.2 default is 60 seconds (see Constants.DEFAULT_REQUEST_TIMEOUT_DURATION and AWSClientConfig.createApiConnectionSettings). Because SDK v2 uses ScheduledThreadPoolExecutor.corePoolSize=1, allowCoreThreadTimeOut=false for its internal timer STPE, every schedule() call ensures a core worker is prestarted; those workers park forever and are strongly reachable via the STPE's workQueue array.

Since Paimon HadoopFileIO issues an S3 request per split per read op, and the FS cache reuse (post-#65586) is by-bucket only, each S3 request still triggers one sdk-ScheduledExecutor daemon thread that never dies. Under our workload that's ~80/min per BE — 4h to freeze.

The one-line fix

Zero code change, just a catalog property so hadoop-aws sets apiCallTimeout=0 and TimerUtils takes the no-op path:

ALTER CATALOG paimon SET PROPERTIES ("fs.s3a.connection.request.timeout" = "0");
-- restart BE pods so hadoop Configuration picks up the new value on next S3AFileSystem.initialize

Verified on paimon-prod for the last 30+ minutes across all 4 BEs (all on the base apache/doris:be-4.1.1 image, no fscachefix):

ScheduledThreadPoolExecutor count: 0 (was 5,895)
NamedThreadFactory count: 5 stable, not growing (was 5,536 and growing)
rs_normal thread count: 48-68 stable per BE, not growing (was 20-27k and growing)
total_threads: 2058-2078 (was 24-29k)
RSS: 1.2-2.5 GB per BE (was 6-12 GB)

Side effect is minor: no more 60s API call timeout, but socket/connection timeouts (fs.s3a.connection.timeout, socket read timeout) still apply, and Doris' query-level timeout already covers the user-facing SLA.

Suggestions for 4.1.4

Merge PR branch-4.1: [fix](filesystem) Credential-aware hadoop FileSystem cache via doris.fs.cache.key #65586 (still needed for correct FS reuse and credential isolation).
Additionally, either default apiCallTimeout to 0 for the Doris-owned S3 clients, or document the interaction so users of Paimon-on-S3 know to set fs.s3a.connection.request.timeout=0 in their catalog. Right now the default is a functional footgun on any long-running Doris BE that reads S3.
Optionally file an upstream HADOOP-* ticket asking whether SDK v2's per-request timeout timer should share a bounded executor instead of relying on the caller to shutdown/reuse — the fact that a single schedule() call permanently adds a core worker is surprising and undocumented.

Happy to attach the full jmap/jstack/MAT artifacts if useful.

apache/hadoop#8426 Setting the timeout to 0 doesn’t seem like the right solution. What we really need is the fix from the upstream Hadoop PR.

BE remote scanner thread pool (rs_normal) never shrinks — thread count grows to 27k+ and BE freezes #65416

Description

BE remote scanner thread pool (rs_normal) never shrinks — thread count grows to 27k+ and BE freezes

Environment

Symptom

Misleading signals

Root cause

Configs I tried to cap the pool (all ineffective on rc01)

Current workaround

Expected behavior

Reproduction

Question

Activity

yiguolei commented on Jul 10, 2026

tracy1014-hub commented on Jul 10, 2026

1. Host CPU cores

2. BE logs from startup → capture

Snapshot at capture time (pod age 4h29m, BE still Alive=true, HeartbeatFailureCounter=0)

What I can do next

yiguolei commented on Jul 10, 2026

tracy1014-hub commented on Jul 10, 2026

Verification that the config took effect

rs_normal growth: experiment vs control (~30 min in)

rs_normal trajectory on doris-be-0 (num_cores=8) over 30 min

Observations

Cluster status

What's next?

tracy1014-hub commented on Jul 10, 2026

Experiment status (T+46 min on doris-be-0 with num_cores = 8)

Control group snapshot

Opportunity to capture freeze-time data (the "Path B" from my earlier offer)

yiguolei commented on Jul 10, 2026

tracy1014-hub commented on Jul 10, 2026

rs_normal pool metrics (the smoking gun)

What this tells us

num_cores = 8 vs default — still no difference

Live state at capture time

yiguolei commented on Jul 10, 2026

tracy1014-hub commented on Jul 10, 2026

pstack status

Freeze-time artifacts captured meanwhile (while gdb is still attached)

Note on the freeze

tracy1014-hub commented on Jul 10, 2026

Why partial

Headline finding

Diagnosis

Where to look in the code

Cluster state

yiguolei commented on Jul 10, 2026

yiguolei commented on Jul 10, 2026

yiguolei commented on Jul 10, 2026

tracy1014-hub commented on Jul 10, 2026

Thread name distribution (15,080 total Java threads)

All 7,324 parked threads have an identical stack

jmap histogram confirms the leak

Diagnosis

Where to look in the code

WeChat

Live state

yiguolei commented on Jul 10, 2026

tracy1014-hub commented on Jul 10, 2026

zhangstar333 commented on Jul 10, 2026

yiguolei commented on Jul 13, 2026

tracy1014-hub commented on Jul 14, 2026

yiguolei commented on Jul 14, 2026

yiguolei commented on Jul 15, 2026

tracy1014-hub commented on Jul 15, 2026

tracy1014-hub commented on Jul 17, 2026

Update: PR #65586 helps but doesn't fully fix rs_normal growth — real root cause is fs.s3a.connection.request.timeout=60s (hadoop-aws SDK v2 timer)

Test 1 — PR #65586 alone

Test 2 — JFR ThreadStart sampling: where do new STPEs come from

The one-line fix

Suggestions for 4.1.4

CalvinKirs commented on Jul 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

BE remote scanner thread pool (`rs_normal`) never shrinks — thread count grows to 27k+ and BE freezes

Snapshot at capture time (pod age 4h29m, BE still `Alive=true`, `HeartbeatFailureCounter=0`)

Experiment status (T+46 min on doris-be-0 with `num_cores = 8`)

Update: PR #65586 helps but doesn't fully fix rs_normal growth — real root cause is `fs.s3a.connection.request.timeout=60s` (hadoop-aws SDK v2 timer)