You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
date_part declares a non-nullable result for non-null input, but can return NULL #26685
date_part fails when a non-null input gives a NULL result:
SELECT extract(second from ts)
FROM (VALUES (arrow_cast(9223372036854775807, 'Timestamp(Second, None)'))) AS t(ts);
-- Arrow error: Invalid argument error: Column 'date_part(Utf8("SECOND"),t.ts)' is declared as non-nullable but contains null valuesSELECT extract(hour from d)
FROM (VALUES (arrow_cast(9223372036854775807, 'Duration(Second)'))) AS t(d);
-- Arrow error: Invalid argument error: Column 'date_part(Utf8("HOUR"),t.d)' is declared as non-nullable but contains null values
To Reproduce
Run the queries above on main (dad9063be).
Expected behavior
Both return NULL.
Additional context
return_field_from_args takes the result's nullability from the input. Arrow's date_part returns NULL for some non-null inputs, such as timestamps it can't convert and values that overflow Int32 (Duration(s) hour).
The result was always nullable until #19823 (53.0.0). That PR added the Spark date_part, which simplifies to this function, and changed the nullability so the two fields would match (review thread). The Spark wrapper still takes its nullability from the inputs, so a fix here should change both to keep them matching. The optimizer's schema check (logically_equivalent_names_and_types) compares names and types but not nullability, so a mismatch between them would go unnoticed.
Activity
fix: date_part sub-second parts and result nullability, reusing arrow…