[Bug] Variant extraction into MAP<STRING NOT NULL, ...> returns NULL #10875

Search before asking

• I searched in the issues and found nothing similar.

Paimon version
master @ 5c82868 / 2.2-SNAPSHOT

Compute Engine
Spark 4.1 (variant pushdown); the defect itself is in paimon-common.

Minimal reproduce step
On Spark 4.1 with default settings (spark.sql.variant.pushVariantIntoScan defaults to true), run SELECT try_variant_get(v, '$.m', 'map<string,int>') on a Paimon table whose v holds {"m":{"a":1}}. It returns NULL; with spark.sql.variant.pushVariantIntoScan=false it returns {a -> 1}. Verified by code trace and paimon-common unit tests, not by an end-to-end Spark run.

What doesn't meet your expectations?
A map with a STRING key should be extracted regardless of the key's nullability.

Anything else?
SparkTypeUtils always converts map keys to NOT NULL, so the pushed-down target is MAP<STRING NOT NULL, INT>. VariantGet.cast and BaseVariantReader.create check the key with equals(DataTypes.STRING()), which compares nullability, and treat it as an invalid cast. The checks came from #6652, which ported Spark's MapType(_: StringType, _, _) pattern. Only try_variant_get/try_cast with MAP<STRING, ...> targets (also nested in arrays) are affected; variant_get with a map target is not pushed down.

Are you willing to submit a PR?

• I'm willing to submit a PR!