intersect_metadata_for_union() is too strict for extension types with JSON metadata serialization #26810

Describe the bug

The intersect_metadata_for_union() function, which determines what metadata is propagated through a UNION for aligned fields, is slightly too strict: it discards metadata that doesn't agree, which strips metadata if two ARROW:extension:metadatas are, say, {"width":5} and "{ "width" : 5 } (despite the types being identical. We came across this problem because GeoArrow's metadata includes a "crs" key where one "crs" can be represented a few ways.

A note that this can be worked around with an optimizer rule (e.g., apache/sedona-db#1520 ).

To Reproduce

use std::collections::HashMap;

use datafusion_expr::expr::intersect_metadata_for_union;

fn main() {
    let metadata = |json: &str| {
        HashMap::from([
            (
                "ARROW:extension:name".to_string(),
                "example.width".to_string(),
            ),
            ("ARROW:extension:metadata".to_string(), json.to_string()),
        ])
    };

    let left = metadata(r#"{"width":5}"#);
    let right = metadata(r#"{      "width" : 5 }"#);

    let merged = intersect_metadata_for_union([&left, &right]);
    println!("{merged:?}");
}

Expected behavior

The current behaviour is not bad because we can work around it (i.e., it doesn't error); however, it would be nice for DFExtensionType to be able to handle this (e.g., handle type equality).

For what it's worth, R handles this extensibly with vec_ptype2() and vec_cast() generics: it reduces the list of types pairwise to find a type that all inputs agree they can be cast into, and then vec_cast dispatches and performs the final cast.

Additional context

No response

Willingness to contribute

None