Skip to content

Wrong results when pushdown_filters is enabled (starting in 48.0.0) #16588

New issue
New issue
Closed
#16646
Closed
Wrong results when pushdown_filters is enabled (starting in 48.0.0)#16588
#16646
Labels
bugSomething isn't workingregressionSomething that used to work no longer does

Description

@alamb
alamb
opened on Jun 27, 2025

Describe the bug

There are a lot of linked issues and discussions here, so it's quite possible that I missed an existing bug report, but I wanted to report an issue I encountered when enabling datafusion.execution.parquet.pushdown_filters.

Originally posted by @ianthetechie in #3463

To Reproduce

I've put together an MRE that demonstrates it with what seems like a trivial query (I initially had something MUCH more complex in a real app, but even this gets wrong results): https://github.com/ianthetechie/datafusion-predicate-pushdown-mre.

Expected behavior

I am expecting the number to be 72, but the count comes to 1245 with datafusion.execution.parquet.pushdown_filters set to true.

Additional context

No response

Activity

  1. alamb
    added
    bugSomething isn't working
    on Jun 27, 2025

  2. ianthetechie commented on Jun 28, 2025

    @ianthetechie
    ianthetechie
    on Jun 28, 2025
    Contributor

    @xudong963 asked in the other issue whether the issue existed in 47.0. I just tested and 47.0 gives the correct output. So the issue is somewhere between the 47.0 and 48.0 releases.

  3. alamb
    added
    regressionSomething that used to work no longer does
    on Jun 28, 2025
  4. alamb
    changed the title [-]Wrong results when `pushdown_filters` is enabled[/-] [+]Wrong results when `pushdown_filters` is enabled (starting in 48.0.0)[/+] on Jun 28, 2025

  5. alamb commented on Jun 28, 2025

    @alamb
    alamb
    on Jun 28, 2025
    ContributorAuthor

    Thanks @ianthetechie -- I have marked this issue as a regression in 48

  6. adriangb commented on Jul 1, 2025

    @adriangb
    adriangb
    on Jul 1, 2025
    Contributor

    @ianthetechie I looked at your example repo (thanks so much for providing an MRE) but I don't see the data anywhere. Maybe I'm missing something but is categories a known dataset? Without that it may be hard to repro. The query looks relatively simple and I feel like similar queries are extensively tested.

  7. ianthetechie commented on Jul 1, 2025

    @ianthetechie
    ianthetechie
    on Jul 1, 2025
    Contributor

    @adriangb The dataset is loaded here: https://github.com/ianthetechie/datafusion-predicate-pushdown-mre/blob/0abc24cd2f728a88e7496f0294554f0858d7c79b/src/context.rs#L12. It's coming from a public S3 bucket, and is a bit large to put in a git repo (though not excessive; the query is very fast), so I didn't bother committing the data. The example should run as-is though. Let me know if you have any issues / find otherwise!

  8. adriangb commented on Jul 1, 2025

    @adriangb
    adriangb
    on Jul 1, 2025
    Contributor

    Sorry I missed that!

  9. ianthetechie commented on Jul 1, 2025

    @ianthetechie
    ianthetechie
    on Jul 1, 2025
    Contributor

    No worries; I thought I was doing a favor trying to separate things into different files, but obviously that didn't work out ;)

  10. adriangb commented on Jul 1, 2025

    @adriangb
    adriangb
    on Jul 1, 2025
    Contributor

    I can reproduce running this SQL in datafusion-cli:

    -- Create external table pointing to the provided parquet file
CREATE EXTERNAL TABLE categories 
STORED AS PARQUET
LOCATION 's3://fsq-os-places-us-east-1/release/dt=2025-06-10/categories/parquet/categories.zstd.parquet';

-- First, inspect the data structure
SELECT * FROM categories LIMIT 5;

-- Test with pushdown DISABLED
SET datafusion.execution.parquet.pushdown_filters = false;

-- Query with pushdown DISABLED - establish baseline
SELECT COUNT(*)
FROM categories
WHERE level1_category_name = 'Arts and Entertainment';

-- Enable pushdown
SET datafusion.execution.parquet.pushdown_filters = true;

-- Same query with pushdown ENABLED - should match baseline
SELECT COUNT(*)
FROM categories
WHERE level1_category_name = 'Arts and Entertainment';

-- Cleanup
DROP TABLE categories;

  11. adriangb commented on Jul 1, 2025

    @adriangb
    adriangb
    on Jul 1, 2025
    Contributor

    A bisect revealed the offending commit is #15769. I am investigating further.

  12. adriangb commented on Jul 1, 2025

    @adriangb
    adriangb
    on Jul 1, 2025
    Contributor

    This small change to the query fixes it:

    -- Enable pushdown
SET datafusion.execution.parquet.pushdown_filters = true;

-- Create external table pointing to the provided parquet file
CREATE EXTERNAL TABLE categories 
STORED AS PARQUET
LOCATION 's3://fsq-os-places-us-east-1/release/dt=2025-06-10/categories/parquet/categories.zstd.parquet';

-- Same query with pushdown ENABLED - should match baseline
SELECT COUNT(*)
FROM categories
WHERE level1_category_name = 'Arts and Entertainment';

-- Cleanup
DROP TABLE categories;

  13. alamb commented on Jul 1, 2025

    @alamb
    alamb
    on Jul 1, 2025
    ContributorAuthor

    Thank you for looking into this @adriangb

  14. adriangb commented on Jul 1, 2025

    @adriangb
    adriangb
    on Jul 1, 2025
    Contributor

    I have a fix

  15. adriangb commented on Jul 1, 2025

    @adriangb
    adriangb
    on Jul 1, 2025
    Contributor

    #16646

  16. alamb
    closed this as completedin #16646on Jul 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingregressionSomething that used to work no longer does

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions