Type Coercion fails for List with inner type struct which has large/view types

Describe the bug

A LargeList(Struct({"foo": LargeUtf8})) cannot be coerced to List(Struct({"foo": Utf8})). It however it works fine for LargeList(LargeUtf8) -> List(Utf8) and Struct({"foo": LargeUtf8}) -> Struct({"foo": Utf8}).

To Reproduce

import polars as pl
from deltalake import DeltaTable

tmp_path = "test_table__"
df = pl.DataFrame({"foo": [1], "bar": [[{"foo": "!"}]]})
df.write_delta(tmp_path, mode="overwrite", overwrite_schema=True)

DeltaTable(tmp_path).merge(
    df.to_arrow(compat_level=1),
    predicate="s.foo = t.foo",
    source_alias="s",
    target_alias="t",
    large_dtypes=None,
).when_matched_update_all().execute()

DeltaError: Generic DeltaTable error: type_coercion
caused by
Error during planning: Failed to coerce then ([LargeList(Field { name: "item", data_type: Struct([Field { name: "foo", data_type: Utf8View, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "element", data_type: Struct([Field { name: "foo", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "element", data_type: Struct([Field { name: "foo", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "element", data_type: Struct([Field { name: "foo", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })]) and else (None) to common types in CASE WHEN expression

Expected behavior

Be able to coerce Large/view and normal arrow types in deeply nested types.

Additional context

Luckly we still can downcast in python using the large_dtypes=False, but datafusion should be able to coerce any deeply nested dtype.

I tested this in a fork of the deltalake repo but could not reproduce the error:

DeltaLake Version: 0.23.0
Polars Version: 1.20.0

@kosiew

@kosiew you are testing it against an older version of deltalake from what I can see in the commit: https://github.com/delta-io/delta-rs/blob/d8080b13f5724aa09fb268b17f507dfd8559255f/python/pyproject.toml

In that commit you can see it was Datafusion v43, we are now at datafusion v44.

@ion-elgreco

Good catch @ion-elgreco ☝!
I managed to reproduce the error after fetching the latest main from the upstream.

I investigated the delta-rs repo, because I earlier tested the coercion in datafusion v44, but could not trigger the error:

https://github.com/kosiew/datafusion/blob/f845a233cb90758b42c0ab2d3ea1c1e02da36aa3/datafusion/optimizer/src/analyzer/type_coercion.rs#L2140-L2199

Given the description on this PR it appears it is a regression (something that used to work, but now does not). Is this the case?

I'll put it on the list of things to investigate before the 45 relese

@alamb

@alamb I'll have to double check, but I think it worked in DF43

@alamb

@alamb yes this indeed worked fine on DF43. I tested it against deltalake v0.23.3

Possibly related:

Support List type coercion for CASE-WHEN-THEN expression #12490 (released in DataFusion 44.0.0)
Coerce Array inner types #13452 (also released in DataFusion 44.0.0)

Ao the next step for this PR is to find a DataFusion only reproducer that works in DF 43 but not in DF 44. I will try to do so tomorrow

I tried to make a datafusion only reproducer but it turns out I can't create LargeLists of structs via SQL 🤔 I'll have to think about how to do so a bit more tomorrow...

Well, I found another bug when working on this one

Case coercion of Structs loses field names #14383

I still haven't found a reproducer for this one. But I have another idea

Ok, I have a datafusion only reproducer:

create or replace table t as values
(
 100,                                         -- column1 int (so the case isn't constant folded)
 [{ 'foo': arrow_cast('baz', 'Utf8View') }],  -- column2 has List of Struct w/ Utf8View
 [{ 'foo': 'bar' }],                          -- column3 has List of Struct w/ Utf8
 [{ 'foo': 'blarg' }]                         -- column4 has List of Struct w/ Utf8
);

SELECT column2, column3, column4  FROM t;

SELECT
  case
    when column1 > 0 then column2
    when column1 < 0 then column3
    else column4
  end
FROM t;

Fails with

Error during planning: Failed to coerce then ([List(Field { name: "item", data_type: Struct([Field { name: "foo", data_type: Utf8View, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Struct([Field { name: "foo", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })]) and else (Some(List(Field { name: "item", data_type: Struct([Field { name: "foo", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }))) to common types in CASE WHEN expression

It works with datafusion-cli on 43:

> SELECT
  case
    when column1 > 0 then column2
    when column1 < 0 then column3
    else column4
  end
FROM t;
+-----------------------------------------------------------------------------------------------------------+
| CASE WHEN t.column1 > Int64(0) THEN t.column2 WHEN t.column1 < Int64(0) THEN t.column3 ELSE t.column4 END |
+-----------------------------------------------------------------------------------------------------------+
| [{foo: baz}]                                                                                              |
+-----------------------------------------------------------------------------------------------------------+
1 row(s) fetched.

This works fine with only structs, just not with structs in lists 🤔

I think this PR fixes the issue (it is related, but not the same as #14383)

Fix regression list Type Coercion List with inner type struct which has large/view types #14385

Navigation Menu

Repository navigation

Type Coercion fails for List with inner type struct which has large/view types #14154

Describe the bug

To Reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Type Coercion fails for List with inner type struct which has large/view types #14154

Description

Describe the bug

To Reproduce

Expected behavior

Additional context

Activity

kosiew commented on Jan 17, 2025

ion-elgreco commented on Jan 17, 2025

kosiew commented on Jan 17, 2025

alamb commented on Jan 27, 2025

ion-elgreco commented on Jan 27, 2025

ion-elgreco commented on Jan 28, 2025

alamb commented on Jan 29, 2025

alamb commented on Jan 29, 2025

alamb commented on Jan 31, 2025

alamb commented on Jan 31, 2025

alamb commented on Jan 31, 2025

alamb commented on Jan 31, 2025

alamb commented on Jan 31, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions