Repository navigation
Wrong query results for filters that involve partition columns and data file columns and pushdown_filters is enabled #15912
Description
Describe the bug
Filters such as partition_col = col_from_file are never applied if datafusion.execution.parquet.pushdown_filters = true
To Reproduce
With datafusion-cli:
COPY (
SELECT arrow_cast('a', 'Utf8') AS val
) TO 'test_files/scratch/test/part=a/123.parquet'
STORED AS PARQUET;
COPY (
SELECT arrow_cast('b', 'Utf8') AS val
) TO 'test_files/scratch/test/part=b/123.parquet'
STORED AS PARQUET;
COPY (
SELECT arrow_cast('xyz', 'Utf8') AS val
) TO 'test_files/scratch/test/part=c/123.parquet'
STORED AS PARQUET;
set datafusion.execution.parquet.pushdown_filters = true;
CREATE EXTERNAL TABLE test(part text, val text)
STORED AS PARQUET
PARTITIONED BY (part)
LOCATION 'test_files/scratch/test/';
SELECT * FROM test;
explain analyze
select * from test
where part != val;> select * from test
where part != val;
+-----+------+
| val | part |
+-----+------+
| a | a |
| xyz | c |
| b | b |
+-----+------+
3 row(s) fetched.
Which is clearly wrong.
Expected behavior
> select * from test
where part != val;
+-----+------+
| val | part |
+-----+------+
| xyz | c |
+-----+------+
Additional context
No response
Activity
alamb commentedon May 2, 2025
sounds serious
[-]Wrong query results for filters that involve partition columns and data file columns[/-][+]Wrong query results for filters that involve partition columns and data file columns and `pushdown_filters` is enabled[/+]alamb commentedon May 2, 2025
This is all the more reason I think to avoid adding more complexity to ListingTable as we are disucssing in
location,size,last_modified) inListingTableProvider#15173adriangb commentedon May 2, 2025
I think the fix is relatively simple though: any filters that reference both partition columns and data columns need to be marked as Inexact. I'm traveling so don't know that I can make a PR soon but it's a couple LOC.
adriangb commentedon May 2, 2025
Also although yes this is serious I suspect is pretty rare to have a filter that depends on both a partition column and data column. It hasn't been reported for years...
alamb commentedon May 2, 2025
I also think it is not that common to turn on filter_pushdown and use it with ListingTable - e.g. InfluxData and Comet both do not use ListingTable 🤔
adriangb commentedon May 2, 2025
Good point
adriangb commentedon May 3, 2025
Looking at the git blame... it looks like I introduced the bug in #15263.
Feeling guilty so here is the fix: #15935