Expressions evaluated in the Parquet scan lose field metadata (extension types) of columns replaced with literals #26855

Describe the bug

When a Parquet table has field metadata in its schema, such as an Arrow extension type (ARROW:extension:name), expressions evaluated inside the Parquet scan can see the column without that metadata. Expressions that read the field metadata of their arguments then see an empty map.

For example, arrow_metadata(u) returns {} for a Parquet column u, although the table schema has ARROW:extension:name on u. The same query over an in-memory table returns the metadata. Extension-aware UDFs that dispatch on the argument Field at execution time are affected the same way.

To Reproduce

With datafusion 55.0.0 from PyPI. skip_metadata=False makes schema inference keep the field metadata; with the default skip_metadata=True the table schema has no field metadata at all, see #26851.

import datafusion
import pyarrow as pa
import pyarrow.parquet as pq

# An extension type unknown to pyarrow, so that pyarrow keeps it as field metadata
field = pa.field("u", pa.binary(16), metadata={"ARROW:extension:name": "example.ext"})
table = pa.table({"u": pa.array([b"\x00" * 16], pa.binary(16))}, schema=pa.schema([field]))
pq.write_table(table, "ext.parquet")

ctx = datafusion.SessionContext()
ctx.register_parquet("t", "ext.parquet", skip_metadata=False)
print("table schema:", ctx.table("t").schema().field("u").metadata)

query = "SELECT arrow_metadata(u) AS m FROM t"
print("arrow_metadata(u):", ctx.sql(query).to_pylist())
print(ctx.sql(f"EXPLAIN {query}").to_pylist()[-1]["plan"])

Output:

table schema: {b'ARROW:extension:name': b'example.ext'}
arrow_metadata(u): [{'m': []}]
DataSourceExec: file_groups={1 group: [[.../ext.parquet]]}, projection=[arrow_metadata(u@0) as m], file_type=parquet

Controls on the same version:

So the scan itself returns u with its metadata. Only expressions pushed into DataSourceExec (projection=[arrow_metadata(u@0) as m]) see it without metadata.

Cause. The Parquet opener replaces column references with literals for partition columns and for columns that the file statistics prove constant (min == max). The literals are created without field metadata (replace_columns_with_literals in datafusion/physical-expr-adapter/src/schema_rewriter.rs, called from datafusion/datasource-parquet/src/opener/mod.rs). In the example, the file has one row, so u is constant and arrow_metadata(u) is evaluated on a literal without metadata.

The extension name is unknown to pyarrow (example.ext) only so that pyarrow keeps it as field metadata. A file written by DataFusion itself with an arrow.uuid column behaves the same way.

Expected behavior

arrow_metadata(u) returns {ARROW:extension:name: example.ext}, as for the in-memory table: the scan produces fields with the metadata of the table schema.

Additional context

Related: #26851 (schema inference drops extension types when skip_metadata = true) and #24721 (closed; projection field metadata lost when physical optimization removes a ProjectionExec).

Willingness to contribute

I plan to contribute a fix for this bug independently