Repository navigation
[Bug] [Python] Batch reads ignore the table option scan.fallback-branch #10884
Description
Search before asking
- I searched in the issues and found nothing similar.
Paimon version
pypaimon 2.1.0, and master, which prints the same.
Compute Engine
Python API (pypaimon 2.1.0, Python 3.12.16, pyarrow 19.0.1) on a local filesystem catalog. Spark 3.5.6 with paimon-spark-3.5_2.12 2.1.0 for the comparison.
Minimal reproduce step
Currently, pypaimon checks scan.fallback-branch only in the catalog options and never reads it from the table, so a batch read leaves out the partitions that exist only in the fallback branch. Spark's SELECT k, p FROM paimon.db.t on the same table returns (1, 'a') and (2, 'b').
import pyarrow as pa
from pypaimon import CatalogFactory, Schema
from pypaimon.schema.schema_change import SchemaChange
cat = CatalogFactory.create({"warehouse": "wh"})
cat.create_database("db", True)
sch = pa.schema([("k", pa.int64()), ("p", pa.string())])
cat.create_table("db.t", Schema.from_pyarrow_schema(sch, partition_keys=["p"]), False)
cat.create_branch("db.t", "b")
def write(name, k, p):
wb = cat.get_table(name).new_batch_write_builder()
w, c = wb.new_write(), wb.new_commit()
w.write_arrow(pa.table({"k": [k], "p": [p]}, schema=sch))
c.commit(w.prepare_commit())
write("db.t", 1, "a")
write("db.t$branch_b", 2, "b")
cat.alter_table("db.t", [SchemaChange.set_option("scan.fallback-branch", "b")])
rb = cat.get_table("db.t").new_read_builder()
print(rb.new_read().to_arrow(rb.new_scan().plan().splits()).to_pylist())
# [{'k': 1, 'p': 'a'}], should also have {'k': 2, 'p': 'b'}
# partition p=b does not exist on main, so it should be read from the fallback branch bWhat doesn't meet your expectations?
The read returns (1, 'a') from main and (2, 'b') from the fallback branch b, as Spark does.
Anything else?
No response
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity