Skip to content

[lake/paimon] Allow passing an existing table to PaimonLakeSource #4957

Open

Description

@bryndenZh

Search before asking

  • I searched the existing issues and PRs and found nothing similar.

Motivation

Query engines such as StarRocks manage Paimon catalogs and cache loaded tables and metadata. When querying lake-tiered Fluss tables, they should be able to reuse these tables and their associated caches.

Currently, PaimonLakeSource only accepts catalog configuration and a table path. The source and its split planner create catalogs and load the table separately for filter conversion, split planning, and reader creation.

Each newly created CachingCatalog has independent caches. Consequently, repeated planning cannot reuse previously cached table, snapshot, and manifest metadata, resulting in redundant metadata loading and file reads. This also prevents Fluss from benefiting from the query engine's existing native Paimon caches.

Solution

Add a PaimonLakeSource constructor accepting an existing FileStoreTable, so callers can reuse their loaded table and its caches. Keep the existing configuration-based constructor.

Willingness to contribute

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions