Repository navigation
Description
Describe the bug
Since #19078, the to_timestamp / to_timestamp_seconds / to_timestamp_millis / to_timestamp_micros / to_timestamp_nanos functions override the timezone of an input that is already timezone-aware with the session's datafusion.execution.time_zone.
When that config is unset (the default None), an input of type Timestamp(_, Some(tz)) is returned as Timestamp(_, None). The zone annotation is dropped while the underlying epoch is kept, which silently changes the wall-clock meaning of the value.
Previously, these functions had a dedicated match arm that preserved a zoned input's timezone:
// before #19078
Timestamp(_, Some(tz)) => args[0].cast_to(&Timestamp(Millisecond, Some(tz)), None),#19078 collapsed that arm into the generic one, which casts to the execution timezone:
// after #19078 (current main)
Null | Int32 | Int64 | Timestamp(_, _) => args[0].cast_to(&Timestamp(Millisecond, self.timezone.clone()), None),The PR's rationale when I read it, only concerns naive timestamp strings being interpreted in the execution timezone, which is reasonable. Dropping the timezone of inputs that are already zoned appears to be an unintended side effectas it makes a normalisation function silently discard a caller's explicit timezone.
To Reproduce
Run in datafusion-cli with default settings (no SET datafusion.execution.time_zone):
-- input epoch = 2024-01-01T00:00:00Z, annotated +05:00 => local wall clock 05:00
SELECT
arrow_typeof(to_timestamp_millis(arrow_cast(1704067200000, 'Timestamp(Millisecond, Some("+05:00"))'))) AS result_type,
date_part('hour', arrow_cast(1704067200000, 'Timestamp(Millisecond, Some("+05:00"))')) AS hour_direct,
date_part('hour', to_timestamp_millis(arrow_cast(1704067200000, 'Timestamp(Millisecond, Some("+05:00"))'))) AS hour_via_to_timestamp;Actual
| result_type | hour_direct | hour_via_to_timestamp |
|---|---|---|
Timestamp(Millisecond, None) |
5 | 0 |
Befor the change
| result_type | hour_direct | hour_via_to_timestamp |
|---|---|---|
Timestamp(Millisecond, Some("+05:00")) |
5 | 5 |
Expected behavior
to_timestamp_millis (and the rest of the family) should not silently drop the timezone of an already-zoned input. Passing a Timestamp(_, Some("+05:00")) through the function should either:
- preserve the input zone, or
- if honoring
datafusion.execution.time_zone, convert the timezone but never relabel to a naive type, which changes the wall-clock value.
In particular, when datafusion.execution.time_zone is unset (None), a zoned input should keep its zone rather than becoming naive. As it stands, date_part('hour', to_timestamp_millis(<zoned column>)) returns the UTC hour instead of the local hour, which corrupts any hour-of-day / day-of-month bucketing over zoned columns.
Additional context
Relevant commit: ada0923a3927d16dc5d810637cfbea4146c54f54 (PR #19078 that closed #17998)
Activity
nevi-me commentedon Jul 23, 2026
I'm attempting a fix
AmoghAtreya commentedon Jul 25, 2026
@nevi-me are you still working on this? if not, i can pick it up
nevi-me commentedon Jul 25, 2026
@AmoghAtreya I have been testing a fix in our fork, I'll submit the PR this weekend