Skip to content

Internal error when parquet predicate pushdown is enabled "Error evaluating filter predicate:" #4006

Closed
#4021
Closed
Internal error when parquet predicate pushdown is enabled "Error evaluating filter predicate:"#4006
#4021
Labels
bugSomething isn't working

Description

@alamb
alamb
opened on Oct 28, 2022

Describe the bug
DataFusion generates an error for some predicates when predicate pushdown is enabled

NOTE that pushdown filtering is not enabled by default (as we are still working on it) so this issue will not likely affect users:

To Reproduce

  1. Download data from
    repro.zip
  2. Run datafusion CLI

The query run is

select count(*) from foo where request_duration_ns > 791684060 OR client_addr NOT in ('213.120.214.213');

Expected behavior
Same answer should be produced with and without row filtering enabled. However, with row filtering an error is produced

datafusion-cli -f script.sql 
+-----------------+
| COUNT(UInt8(1)) |
+-----------------+
| 53819           |
+-----------------+
1 row in set. Query took 0.006 seconds.

With it enabled:

DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=true datafusion-cli -f script.sql 
...
1 row in set. Query took 0.002 seconds.
ArrowError(ExternalError(Execution("Arrow error: External error: Arrow: underlying Arrow error: Compute error: Error evaluating filter predicate: Internal(\"Cannot evaluate binary expression Gt with types Utf8 and Int32\")")))

Additional context
Found by the test here #3976

Activity

  1. alamb
    added
    bugSomething isn't working
    on Oct 28, 2022

  2. tustvold commented on Oct 29, 2022

    @tustvold
    tustvold
    on Oct 29, 2022
    Contributor

    It looks like this has the same underlying cause as #4005 (comment)

    Reordering the predicates works

    ❯ select count(*) from foo where client_addr NOT in ('213.120.214.213') OR request_duration_ns > 791684060;
+-----------------+
| COUNT(UInt8(1)) |
+-----------------+
| 53819           |
+-----------------+
1 row in set. Query took 0.247 seconds.
  3. tustvold
    added a commit that references this issue on Oct 29, 2022

    Project columns within DatafusionArrowPredicate (apache#4005) (apache…

    34229e8
  4. tustvold
    closed this as completedin #4021on Oct 30, 2022
  5. tustvold
    added a commit that references this issue on Oct 30, 2022

    Fix predicate pushdown bugs: project columns within DatafusionArrowPr…

    35f786b
  6. jimexist
    added a commit that references this issue on Oct 31, 2022

    Fix predicate pushdown bugs: project columns within DatafusionArrowPr…

    ed27e31
  7. Dandandan
    added a commit that references this issue on Nov 5, 2022

    Fix predicate pushdown bugs: project columns within DatafusionArrowPr…

    f9bff57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions