Repository navigation
Spark map_from_arrays / map_from_entries return wrong results for sliced lists #25419
Description
Describe the bug
When the input list array is sliced, map_from_arrays and map_from_entries in datafusion-spark pair the correct values with keys taken from rows outside the slice.
To Reproduce
datafusion-cli --spark:
> CREATE TABLE t AS VALUES ([99, 98], [990, 980]), ([1, 2], [10, 20]);
> SELECT map_from_arrays(column1, column2)
FROM (SELECT * FROM t ORDER BY column2 DESC LIMIT 1 OFFSET 1) s;
+--------------------------------------+
| map_from_arrays(s.column1,s.column2) |
+--------------------------------------+
| {99: 10, 98: 20} |
+--------------------------------------+
> CREATE TABLE e AS VALUES
([struct(99, 990), struct(98, 980)], 1),
([struct(1, 10), struct(2, 20)], 2);
> SELECT map_from_entries(column1)
FROM (SELECT * FROM e ORDER BY column2 LIMIT 1 OFFSET 1) s;
+-----------------------------+
| map_from_entries(s.column1) |
+-----------------------------+
| {99: 10, 98: 20} |
+-----------------------------+
Expected behavior
Both queries should return {1: 10, 2: 20}. Without the OFFSET, both queries return the correct results.
Additional context
No response
Activity
Himanshu-2005-code commentedon Sep 17, 2026
Would love to work on this can you assign me this
fix: map_from_arrays/map_from_entries pair wrong keys and values for …
neilconway commentedon Sep 17, 2026
@Himanshu-2005-code Thanks! I actually already had a fix for this in-progress, but I would love your review of the change if you get a moment: #25431
Himanshu-2005-code commentedon Sep 17, 2026
Sorry Sir, I hadn't seen you'd already opened a fix when I asked to be assigned. Reviewed #25431: pulled the branch, built it, and ran the datafusion-spark test suite plus the two updated .slt files — all pass, and cargo clippy is clean. The fix looks correct to me — it slices flat_keys down to the current row range before filtering, so the mask lines up with the values it's applied against. I also checked the FixedSizeList/LargeList cases specifically (arrow-rs re-bases FixedSizeListArray::values() on slice already, so the fix's assumptions hold there too), and the new unit tests cover the LAST_WIN and null-row interactions well. Nothing I'd change.
neilconway commentedon Sep 17, 2026
@Himanshu-2005-code thank you!
fix: Spark `map_from_arrays`, `map_from_entries` for sliced lists (ap…
fix: Spark `map_from_arrays`, `map_from_entries` for sliced lists (ap…
fix: report map builder errors in Spark's row order and trim the per-…
fix: Spark `map_from_arrays`, `map_from_entries` for sliced lists (ap…