You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Incorrect results: Bloom filters on UInt8, Int8, UInt16 and Int16 columns always return false negatives #9779
changed the title [-]Bloom filters on Int8 and Int16 columns always return false negatives[/-][+]Bloom filters on `UInt8`, `Int8`, `UInt16` and `Int16` columns always return false negatives[/+]on Apr 1, 2024
Not exactly: as long as #9770 is not merged, bloom filters are not used on UInt8 and UInt16.
Now that I say it, I realize that I should probably amend that PR (and the existing code) to disable bloom filters entirely on these types; so Datafusion is slow instead of incorrect.
Activity
progval commentedon Mar 24, 2024
I just reproduced the bug in the
parquetcrate, so this isn't an issue in Datafusion: apache/arrow-rs#5550[-]Bloom filters on Int8 and Int16 columns always return false negatives[/-][+]Bloom filters on `UInt8`, `Int8`, `UInt16` and `Int16` columns always return false negatives[/+]alamb commentedon Apr 1, 2024
It turns out that #9770 demonstrates that the unsigned variants are incorrect as well so I updated the title of this ticket
progval commentedon Apr 1, 2024
Not exactly: as long as #9770 is not merged, bloom filters are not used on
UInt8andUInt16.Now that I say it, I realize that I should probably amend that PR (and the existing code) to disable bloom filters entirely on these types; so Datafusion is slow instead of incorrect.
[-]Bloom filters on `UInt8`, `Int8`, `UInt16` and `Int16` columns always return false negatives[/-][+]Incorrect results: Bloom filters on `UInt8`, `Int8`, `UInt16` and `Int16` columns always return false negatives[/+]alamb commentedon Apr 2, 2024
It seems like #9770 was just merged .
I filed #9914 to fix the problem by disabling this feature
alamb commentedon Apr 7, 2024
@edmondop disabled this code #9969 🙏
alamb commentedon Apr 18, 2024
This issue came up in the context of 37.1.0 release: #9904 and I wanted to cross post here
Specifically, versions 34.0.0 through 37.0.0 have a bug where
int8/int16bloom filters can incorrectly filter out incorrect answers.The int8/int16 bloom filter support was added in #7821 / shipped as part of https://github.com/apache/arrow-datafusion/blob/main/dev/changelog/33.0.0.md
We have disabled using bloom filters for int8/int16 columns as of datafusion 38.0.0 (until we fix the underlying issue)
alamb commentedon Nov 3, 2026
There is an upstream API that would help here from @bhaskargurram-ai
ArrowSbbfto check Arrow arrays against Parquet bloom filters arrow-rs#11549