Repository navigation
Feature is not implemeneted: Unsupported cast with list of structs #15338
Description
Describe the bug
This bug for me originated when encountering schema evolutions on Delta tables using the delta-rs library. Whenever a schema evolution occurred on my table that contains a field with a list of structs, Datafusion is returning this error:
This feature is not implemented: Unsupported CAST from Struct([Field { name: "properties", data_type: Struct([Field { name: "someNewField", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "fields", data_type: List(Field { name: "item", data_type: Struct([Field { name: "messageId", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) to Struct([Field { name: "properties", data_type: Struct([Field { name: "fields", data_type: List(Field { name: "element", data_type: Struct([Field { name: "messageId", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "someNewField", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])
It appears that the issue isn't necessarily the exact casting itself, but the ORDER of the new field that's added someNewField. If it's put at the end, then it works, but inserting the field in another position yields this error.
To Reproduce
Below is the python code using delta-rs (which is currently on Datafusion 46) that shows this error
import pyarrow as pa
from deltalake import write_deltalake
# Define the path for the Delta table
delta_table_path = "./datafusion-repro-test-table"
# Define the data for the first write
data_first_write = [
{
"uid": "ws_2",
"event": {
"properties": {
"fields": [
{
"messageId": "veniam sed et elit adipisicing"
}
],
},
}
}
]
schema = pa.schema([
pa.field("uid", pa.string()),
pa.field("event", pa.struct([
pa.field("properties", pa.struct([
pa.field("fields", pa.list_(pa.struct([
pa.field("messageId", pa.string()),
]))),
])),
])),
])
print(schema)
first_write = pa.Table.from_pylist(data_first_write, schema=schema)
# Write data to Delta table for the first write
write_deltalake(delta_table_path, first_write, mode="append", engine="rust", schema_mode="merge")
#### NOW FOR THE SECOND WRITE THAT BREAKS ####
data_second_write = [
{
"uid": "ws_2",
"event": {
"properties": {
"someNewField": "test-value", # New field
"fields": [
{
"messageId": "veniam sed et elit adipisicing"
}
],
},
}
}
]
second_schema = pa.schema([
pa.field("uid", pa.string()),
pa.field("event", pa.struct([
pa.field("properties", pa.struct([
pa.field("someNewField", pa.string()), # New field
pa.field("fields", pa.list_(pa.struct([
pa.field("messageId", pa.string()),
]))),
])),
])),
])
second_write = pa.Table.from_pylist(data_second_write, schema=second_schema)
# Write data to Delta table for the second write
write_deltalake(delta_table_path, second_write, mode="append", engine="rust", schema_mode="merge")Expected behavior
Datafusion would support casting a schema when said schema contains a list of structs, no matter where in the order a new field is added to the schema.
Additional context
Originating bug report in delta-rs: delta-io/delta-rs#3339
Activity
bodubas commentedon Mar 20, 2025
Got exactly the same issue. Would be great if datafusion could sort the input schema to alight with the target schema.
alamb commentedon Mar 25, 2025
I think @kosiew may have a PR up that is related
ion-elgreco commentedon Mar 28, 2025
I don't think the PR solves it for us, to my understanding that PR helps when you read parquet data.
But we are evolving the schema within a dataframe cast
dabljues commentedon Jul 14, 2025
Same issue for me, explained here: delta-io/delta-rs#3595
alamb commentedon Jul 18, 2025
@kosiew filed a ticket that I think describes a solution to this problem:
cast_columnto enable struct → struct casting in more contexts #16579adriangb commentedon Aug 5, 2026
This appears fixed on
main(verified againstdatafusion-cli 54.1.0). Name-based nested struct casting now handles field reordering, so the specific complaint here (that insertingsomeNewFieldbeforefieldsfails while appending it at the end work) no longer reproduces.Repro, using a Parquet file whose physical struct order differs from the declared table order, which is the path the delta-rs failure took:
(The quoted identifiers matter — unquoted ones are lowercased and then genuinely won't match by name.)
Schema evolution across two files in one directory works too, with the older file's missing field null-filled:
The
itemvselementlist-item name difference visible in the original error is also handled:cast_columnmatches names only insideStruct, and rebuilds list types from the target's item field.Closing as fixed. @ion-elgreco @dabljues @bodubas if you still hit this through delta-rs please let us know and we can re-open.