Repository navigation
[Bug] Historical Paimon tiering repeatedly scans manifests #4656
Description
Search before asking
- I searched in the issues and found nothing similar.
Fluss version
Observed on a Fluss 0.9-based vendor build with Flink 1.20 and Java 11. The corresponding Fluss implementation was also inspected on main.
The local Paimon cache experiment used paimon-bundle:2-ali-1.0; the exact deployed Paimon binary has not been verified.
Please describe the bug 🐞
A Fluss __historical__ bucket can contain records belonging to multiple original date partitions. During Paimon tiering, initializing their writers repeatedly plans against shared manifests, making metadata processing dominate a small split.
Observed behavior
For the same table and bucket:
| Split | Input offset span | Total duration | Planning calls | Accumulated planning |
|---|---|---|---|---|
| Historical | 1,572 | 70,358 ms | 64 | 56,971 ms |
| Normal date partition | 3,351 | 754 ms | 1 | 324 ms |
Across 9 historical and 22 normal splits on the same worker, median input offset spans were similar: 825 versus 779. Median durations were 43,650 ms versus 753 ms.
The slow historical split spent 81% of its duration in planning. It produced 575 manifest-read completion events across 7 manifest filenames, including 396 events returning zero entries. Of its 64 plans, 58 returned only one data-file metadata entry.
Selected logs from 2026-10-15, UTC+08:00; identifiers are anonymized and logger/thread prefixes omitted:
15:39:23,032 Start to tier table example_db.partitioned_pk_table with table id 1.
15:39:23,841 Read 0 manifest entries from manifest-8e458476-489b-4108-a782-534b55a4494e-0
15:39:24,259 File store scan plan completed in 993 ms. Files size : 4
15:39:24,804 Read 0 manifest entries from manifest-8e458476-489b-4108-a782-534b55a4494e-0
15:39:25,254 File store scan plan completed in 930 ms. Files size : 5
...
15:40:33,390 Finish tier bucket TableBucket{tableId=1, partitionId=100, bucket=1} ...
These are logical read events, not measured object-store request counts. Files size counts returned file metadata entries, not bytes.
Code analysis
Fluss derives the original Paimon partition from each historical record. Paimon caches writers by (partition, bucket), so repeated records for the same pair reuse their writer. Each newly encountered pair initializes a writer through:
getWriterWrapper → createWriterContainer → scanExistingFileMetas
→ FileSystemWriteRestore.restoreFiles
→ scan.withSnapshot(...).withPartitionBucket(partition, bucket).plan()
The partition/bucket filter does not guarantee separate manifests: different partitions can select the same manifest through coarse metadata pruning. In the uncached path, rows are decoded before entry filtering, so returning zero entries can still be expensive.
Paimon's existing ManifestEntryCache already indexes cached segments by partition and bucket. However, ObjectsCache.read bypasses cache admission when a manifest exceeds the single-file size threshold.
Thread samples during a later historical split on the same worker showed the source waiting in restoreFiles → plan, while manifest-reader threads performed OSS reads and Avro decoding through ObjectsCache.read:92. In the inspected local dependency, that line maps to the size-admission bypass branch.
This strongly suggests repeated uncached reads for some manifests. Live manifest sizes, effective cache limits and the exact deployed binary still need verification.
Reproduction
- Enable Paimon tiering for a partitioned primary-key table.
- Populate and tier multiple date partitions, then let their Fluss partitions expire.
- Write updates targeting those dates through the historical partition.
- Process a historical split touching multiple original partitions in one bucket.
Expected behavior
Partition writers should reuse shared manifest metadata while preserving correct file restoration and tiering compaction.
Solution
First verify manifest sizes and effective native cache settings, then evaluate cache.manifest.small-file-threshold and the cache memory budget while keeping compaction enabled.
A local 64-partition experiment using native restoration and write-only=false reduced manifest file opens from 64 to 1 when the shared manifest became eligible for caching. Planning remained 64 calls. This validates the reuse mechanism, not production performance.
If native caching removes the bottleneck, no custom Fluss restore implementation is needed. Otherwise, investigate native manifest pruning or bounded batch restoration in the existing Paimon implementation. A bucket-wide scan may load unrelated partitions and should not be assumed cheaper.
Validation should cover physical reads, accumulated planning, whole-split duration, heap/GC, updates/deletes and actual compaction behavior.
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity
beryllw commentedon Oct 16, 2026
Follow-up from parameter-only verification: native manifest caching substantially reduced historical-split planning, but removing the admission threshold alone was insufficient.
The runs used the same artifact URI and parallelism, with compaction kept enabled. Restart catch-up rounds were excluded.
“Typical plan” is the median of per-split plan-duration medians; accumulated planning is also a per-split median. GC counts are increments across the same 20 TMs within each run; “—” means not included in this comparison. Snapshots and input volumes differed, so this is an observational comparison, not an equal-load benchmark.
Unrestrictedmeanscache.manifest.small-file-threshold="9223372036854775807 b"; the budget iscache.manifest.small-file-memory. Explicit cache arguments were verified on the running jobs. Default values come from the inspected dependency;soft-valueswas not explicitly changed.Key findings:
ObjectsCache.read -> ManifestEntryCache.createSegments -> Avro/object-store reads.Ergonomicsevents, distinct from startup metadata GC.This supports native cache reuse without a custom restore path or disabling compaction. It does not establish 512 MiB as a universal/minimum budget, or distinguish capacity eviction from soft-reference collection. Physical manifest sizes, decoded cache weights and eviction causes remain unmeasured. Admission compares file size, while capacity accounts for decoded cache contents; the total budget should not simply be reused as the admission threshold.
luoyuxia commentedon Oct 17, 2026
I checked apache/paimon#8256 against this issue. It can remove the specific repeated writer-initialization scans, but only for write-only primary-key tables.
With
write.sequence-number-init-mode=snapshot, Paimon'sKeyValueFileStoreWriteskipsscanExistingFileMetas(partition, bucket)whenwrite-only=trueand the latest snapshot already containssequence.generation.max-sequence-number. That should eliminate therestoreFiles -> plan()calls caused by opening many(partition, bucket)writers in a historical split. On an existing table without the property, writers still restore old files; the first commit bootstraps the property from manifest metadata, so the benefit starts with subsequent writers after that commit.In Fluss,
PaimonLakeWritersetswrite-only = !table.datalake.auto-compaction. Therefore this optimization does not apply to the compaction-enabled runs reported above: they retain the original restore path. Native manifest caching remains the applicable mitigation for those runs. Using this PR's path would mean running tiering without automatic compaction and accepting the corresponding file-maintenance tradeoff.Fluss
maincurrently pins Paimon 2.1.0, whose release source contains #8256; the exact Paimon binary in the reported deployment is still unverified. The Paimon option can be passed through a Fluss table property aspaimon.write.sequence-number-init-mode=snapshot. I would treat #8256 as a conditional solution for write-only deployments, not as a full resolution of this issue while compaction remains enabled. A useful check would compare historical-split planning calls and end-to-end duration after the bootstrap commit, and verify update/delete reads remain correct.[lake/paimon] Optimize historical split initialization (apache#4656)
[lake/paimon] Optimize historical split initialization (apache#4656)
[lake/paimon] Optimize historical split initialization (apache#4656)