Incorrect results with parquet filtering pushdown enabled

Describe the bug
DataFusion gets different answers when parquet pushdown is enabled

NOTE that pushdown filtering is not enabled by default (as we are still working on it) so this issue will not likely affect users:

To Reproduce

Download data from
repro.zip
Run datafusion CLI

The query run is

select count(*) from foo where container = 'backend_container_0' OR pod = 'aqcathnxqsphdhgjtgvxsfyiwbmhlmg';

Expected behavior
Same answer should be produced with and without page index filtering enabled. However, the answers are different

Without filter pushdown 39982 rows are produced

$ DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=false datafusion-cli -f script.sql
...
+-----------------+
| COUNT(UInt8(1)) |
+-----------------+
| 39982           |
+-----------------+

With it enabled:

DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=true datafusion-cli -f script.sql
+-----------------+
| COUNT(UInt8(1)) |
+-----------------+
| 0               |
+-----------------+
1 row in set. Query took 0.004 seconds.

Additional context
Found by the test here #3976

I wonder if this relates to pushing down an empty projection, it is possible there is a bug there, I don't remember seeing a test in arrow-rs for this.

Edit: Sadly not 😢

Edit Edit: I think I have found it

Ok so the bug is that DatafusionArrowPredicate is assuming that ProjectionMask is order preserving, and so is not remapping the columns before passing them to PhysicalExpr. In reality ProjectionMask is not order preserving, and so the batch passed to ArrowPredicate::evaluate has columns in the order of the file's schema. In the case of this query, this results in it evaluating the predicates against the wrong columns, resulting in no rows that pass the predicate.

You can see this, by reordering the predicates in the query

select count(*) from foo where pod = 'aqcathnxqsphdhgjtgvxsfyiwbmhlmg' OR container = 'backend_container_0';
+-----------------+
| COUNT(UInt8(1)) |
+-----------------+
| 39982           |
+-----------------+
1 row in set. Query took 0.107 seconds.

This is likely also the issue behind #4006

Project columns within DatafusionArrowPredicate (apache#4005) (apache…

Fix predicate pushdown bugs: project columns within DatafusionArrowPr…

@tustvold

Thank you for debugging / fixing this @tustvold

Fix predicate pushdown bugs: project columns within DatafusionArrowPr…

Navigation Menu

Repository navigation

Incorrect results with parquet filtering pushdown enabled #4005

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Incorrect results with parquet filtering pushdown enabled #4005

Description

Activity

tustvold commented on Oct 29, 2022

tustvold commented on Oct 29, 2022

alamb commented on Oct 31, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions