You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[lake/paimon] Allow passing an existing table to PaimonLakeSource #4957
I searched the existing issues and PRs and found nothing similar.
Motivation
Query engines such as StarRocks manage Paimon catalogs and cache loaded tables and metadata. When querying lake-tiered Fluss tables, they should be able to reuse these tables and their associated caches.
Currently, PaimonLakeSource only accepts catalog configuration and a table path. The source and its split planner create catalogs and load the table separately for filter conversion, split planning, and reader creation.
Each newly created CachingCatalog has independent caches. Consequently, repeated planning cannot reuse previously cached table, snapshot, and manifest metadata, resulting in redundant metadata loading and file reads. This also prevents Fluss from benefiting from the query engine's existing native Paimon caches.
Solution
Add a PaimonLakeSource constructor accepting an existing FileStoreTable, so callers can reuse their loaded table and its caches. Keep the existing configuration-based constructor.
Activity