[Feature] [Flink] PIP-39: Route Flink reads and regular writes through the lake-stream store #10693

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Part of #10690. Marking a table as a lake-stream table changes nothing for a Flink user today: reads still come only from Paimon snapshots, and writes still go only to Paimon. This issue is about the mode taking effect for Flink reads and regular writes.

Solution

Required behaviour:

• A Flink catalog configured with a Lakestream Store can reach that store's tables. Paimon must not gain a dependency on any store implementation: a store becomes available by putting its jar on the classpath and naming it in the catalog configuration, with its own options passed through to it.
• Reading <table>$rt returns the freshest data for that table, served by the store. Reading the table by its own name keeps returning Paimon snapshots, so existing pipelines are unaffected.
• The $rt alias is read-only and does not take a time-travel timestamp: it exists to serve the freshest data, not a historical point.
• A regular write to a lake-stream table reaches the store rather than Paimon. Writes to tables that are not lake-stream tables are unchanged.
• Misconfiguration is reported as an error that names the store, rather than as a lookup failure or a silent fallback: when the store cannot be resolved, when an option the store requires is missing, when the store side of a lake-stream table does not exist, and when a write is attempted through the read-only alias.
• The behaviour is verified end to end against a real Lakestream Store cluster, not only against a test double.
• The user-facing behaviour is documented.

Anything else?

Out of scope: INSERT OVERWRITE on a lake-stream table. Such a statement is rejected rather than executed, because the write is served by the store and the store cannot execute an overwrite. Also out of scope: the CDC ingestion action, which reaches the table through its own path rather than through a SQL statement.

Are you willing to submit a PR?

• I'm willing to submit a PR!