Skip to content

to_timestamp_* discards the timezone of already-timezone-aware inputs #23841

Description

@nevi-me

Describe the bug

Since #19078, the to_timestamp / to_timestamp_seconds / to_timestamp_millis / to_timestamp_micros / to_timestamp_nanos functions override the timezone of an input that is already timezone-aware with the session's datafusion.execution.time_zone.

When that config is unset (the default None), an input of type Timestamp(_, Some(tz)) is returned as Timestamp(_, None). The zone annotation is dropped while the underlying epoch is kept, which silently changes the wall-clock meaning of the value.

Previously, these functions had a dedicated match arm that preserved a zoned input's timezone:

// before #19078
Timestamp(_, Some(tz)) => args[0].cast_to(&Timestamp(Millisecond, Some(tz)), None),

#19078 collapsed that arm into the generic one, which casts to the execution timezone:

// after #19078 (current main)
Null | Int32 | Int64 | Timestamp(_, _) => args[0].cast_to(&Timestamp(Millisecond, self.timezone.clone()), None),

The PR's rationale when I read it, only concerns naive timestamp strings being interpreted in the execution timezone, which is reasonable. Dropping the timezone of inputs that are already zoned appears to be an unintended side effectas it makes a normalisation function silently discard a caller's explicit timezone.

To Reproduce

Run in datafusion-cli with default settings (no SET datafusion.execution.time_zone):

-- input epoch = 2024-01-01T00:00:00Z, annotated +05:00 => local wall clock 05:00
SELECT
  arrow_typeof(to_timestamp_millis(arrow_cast(1704067200000, 'Timestamp(Millisecond, Some("+05:00"))'))) AS result_type,
  date_part('hour', arrow_cast(1704067200000, 'Timestamp(Millisecond, Some("+05:00"))')) AS hour_direct,
  date_part('hour', to_timestamp_millis(arrow_cast(1704067200000, 'Timestamp(Millisecond, Some("+05:00"))'))) AS hour_via_to_timestamp;

Actual

result_type hour_direct hour_via_to_timestamp
Timestamp(Millisecond, None) 5 0

Befor the change

result_type hour_direct hour_via_to_timestamp
Timestamp(Millisecond, Some("+05:00")) 5 5

Expected behavior

to_timestamp_millis (and the rest of the family) should not silently drop the timezone of an already-zoned input. Passing a Timestamp(_, Some("+05:00")) through the function should either:

  1. preserve the input zone, or
  2. if honoring datafusion.execution.time_zone, convert the timezone but never relabel to a naive type, which changes the wall-clock value.

In particular, when datafusion.execution.time_zone is unset (None), a zoned input should keep its zone rather than becoming naive. As it stands, date_part('hour', to_timestamp_millis(<zoned column>)) returns the UTC hour instead of the local hour, which corrupts any hour-of-day / day-of-month bucketing over zoned columns.

Additional context

Relevant commit: ada0923a3927d16dc5d810637cfbea4146c54f54 (PR #19078 that closed #17998)

Activity

  1. added theissue type on Jul 23, 2026
  2. nevi-me commented on Jul 23, 2026

    @nevi-me
    ContributorAuthor

    I'm attempting a fix

  3. AmoghAtreya commented on Jul 25, 2026

    @AmoghAtreya

    @nevi-me are you still working on this? if not, i can pick it up

  4. nevi-me commented on Jul 25, 2026

    @nevi-me
    ContributorAuthor

    @AmoghAtreya I have been testing a fix in our fork, I'll submit the PR this weekend

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions