Improve performance of extracting statistics from parquet files

@Lordworms

Is your feature request related to a problem or challenge?

@Lordworms added a benchmark for extracting statistics from parquet files in #10610

As this code can be used to extract statistics from parquet files, we would like to make sure it is efficient (especially if we are going to extract statistics for many files at once)

The idea here is to improve the speed of the statistics extraction

Describe the solution you'd like

Make this go faster

cargo bench --bench parquet_statistic

Describe alternatives you've considered

I did some brief profiling:

I think they key would be to change these loops so they built the required Arrow Arrays directly from primitive values rather than from ScalarValue:

datafusion/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs

Lines 183 to 189 in 1bf7112

    
           pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>( 
        
               data_type: &DataType, 
        
               iterator: I, 
        
           ) -> Result<ArrayRef> { 
        
               let scalars = iterator 
        
                   .map(|x| x.and_then(|s| get_statistic!(s, min, min_bytes, Some(data_type)))); 
        
               collect_scalars(data_type, scalars)

Additional context

No response

I was thinking about this last night -- what I would suggest for this is to

make functions like this for each type of https://docs.rs/parquet/latest/parquet/file/statistics/struct.ValueStatistics.html

/// Returns an iterator over min values stored in `ValueStatistics<i32>`
fn extract_i32_mins(stats: impl IntoIterator<&Statistics>) -> impl Iterator<Item = Option<i32>> {
...
}

And then with those iterators, we can make the arrays directly

something like

let Int32ArrayMins = Int32Aray::from_iter(extract_i32_mins(stats));

I think that would be both simple and fast.

Navigation Menu

Repository navigation

Improve performance of extracting statistics from parquet files #10626

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>(
	data_type: &DataType,
	iterator: I,
	) -> Result<ArrayRef> {
	let scalars = iterator
	.map(\|x\| x.and_then(\|s\| get_statistic!(s, min, min_bytes, Some(data_type))));
	collect_scalars(data_type, scalars)

Improve performance of extracting statistics from parquet files #10626

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

alamb commented on May 23, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions