[Feature] PIP-39: Introduce lake-stream mode for second-level data freshness #10690

Search before asking

• I searched in the issues and found nothing similar.

Motivation

This is the umbrella issue for PIP-39: Introduce Lake-Stream Mode to Provide Second-Level Data Freshness for Paimon Tables.

Paimon serves reads from snapshots, so a table's freshness is measured in minutes. Lake-stream mode keeps the table's history in Paimon while a pluggable Lakestream Store serves its most recent writes, so a query can return data that is seconds old. The store's tiering service moves data into Paimon in the background, and the table keeps its name, its schema and its history.

Fluss is the first store implementation, through FIP-25: Support In-Place Promotion of Existing Paimon Tables to Datalake-Enabled Fluss Tables.

Solution

The store is resolved through each engine's own catalog and table-factory SPI, so Paimon takes on no dependency on any particular store implementation. The work therefore splits into an engine-neutral table-format contract — the options that turn a table into a lake-stream table, and the catalog operations that have to be rejected to keep the mapping to the store's table intact — plus the implementations for each engine.

Each scope is tracked as a sub-issue linked below, and is delivered by its own PR so that it stays independently reviewable.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!