Skip to content

[Question] Are composite types supported ? #2179

Closed

Description

@kamilkonior

Hi, I have seen in unit tests that It is possible to store arrow data types in parquet using ArrowWriter. I have created composite type like UUID to check if query against such data will work, but it fails. It looks like as if schema couldn't be read correctly or simply understood.

This page mentions that nested types are not supported https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html,
but there is a way to serialize them via ArrowWriter.

Is something wrong with my approach(code), or this feature is not ready yet ?

const TABLE_ABS_PATH: &str = "/abc/table/...";
const SAMPLE_PATH: &str = "/abc/table/sample1.parquet";

#[tokio::main]
async fn main() -> Result<()> {
    write();
    println!();
    read();
    println!();
    sql().await?;

    Ok(())
}

fn write() {
    let file = std::fs::File::create(Path::new(SAMPLE_PATH)).unwrap();

    let uuid_structure = DataType::Struct(vec![
        Field::new("most", DataType::Int64, false),
        Field::new("least", DataType::Int64, false),
    ]);

    let uuid = Field::new("UUID", uuid_structure.clone(), false);

    let schema = Arc::new(Schema::new(vec![uuid]));

    let mut writer = ArrowWriter::try_new(file, schema.clone(), None).expect("...");

    let data = StructArray::from(vec![
        (
            Field::new("most", DataType::Int64, false),
            Arc::new(Int64Array::from(vec![1, 3])) as ArrayRef,
        ),
        (
            Field::new("least", DataType::Int64, false),
            Arc::new(Int64Array::from(vec![2, 4])) as ArrayRef,
        ),
    ]);

    let rd =
        datafusion::arrow::record_batch::RecordBatch::try_new(schema.clone(), vec![Arc::new(data)])
            .expect("...");

    writer.write(&rd);

    writer.close();
}

fn read() {
    use parquet::file::reader::{FileReader, SerializedFileReader};
    use std::{fs::File, path::Path};

    let f = File::open(Path::new(SAMPLE_PATH)).unwrap();
    let reader1 = SerializedFileReader::new(f).expect("...");
    let mut pqrd = ParquetFileArrowReader::new(Arc::new(reader1));

    let result = pqrd.get_record_reader(60).expect("...");
    for batch in result {
        let batch = batch.unwrap();
        println!("{:?}", batch);
    }
}

async fn sql() -> Result<()> {
    let mut ctx = ExecutionContext::new();

    let file_format = ParquetFormat::default().with_enable_pruning(false);
    let listing_options = ListingOptions {
        file_extension: DEFAULT_PARQUET_EXTENSION.to_owned(),
        format: Arc::new(file_format),
        table_partition_cols: vec![],
        collect_stat: false,
        target_partitions: 1,
    };

    let uuid_structure = DataType::Struct(vec![
        Field::new("most", DataType::Int64, false),
        Field::new("least", DataType::Int64, false),
    ]);

    let uuid = Field::new("UUID", uuid_structure.clone(), false);

    let schema = Arc::new(Schema::new(vec![uuid]));

    ctx.register_listing_table(
        "FANCY_TABLE",
        &format!("file://{}", TABLE_ABS_PATH),
        listing_options,
        Some(schema),
    )
    .await
    .unwrap();

    let df = ctx.sql("SELECT * FROM FANCY_TABLE").await?;

    df.show().await?;

    Ok(())
}

Error:

Invalid argument error: column types must match schema types, expected Struct([Field { name: \"most\", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"least\", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }]) but found Struct([Field { name: \"most\", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }]) at column index 0")

Content:

RecordBatch { schema: Schema { fields: [Field { name: "UUID", data_type: Struct([Field { name: "most", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: "least", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }], metadata: {} }, columns: [StructArray
[
-- child 0: "most" (Int64)
PrimitiveArray<Int64>
[
  1,
  3,
]
-- child 1: "least" (Int64)
PrimitiveArray<Int64>
[
  2,
  4,
]
]] }

Activity

  1. kamilkonior commented on Apr 15, 2022

    @kamilkonior
    ContributorAuthor

    It would be great to receive some feedback, I am not sure but described issue might be a bug living somewhere in arrow reader

  2. alamb commented on Apr 17, 2022

    @alamb
    Contributor

    Sorry @Cheappie -- I think the answer to your question is that this feature is not ready yet. We have some support like #119 but perhaps there is more needed as well.

    I wonder if you have tried the following formulations:

    SELECT UUID.most  FROM FANCY_TABLE
    SELECT UUID["most"]  FROM FANCY_TABLE
  3. kamilkonior commented on Apr 19, 2022

    @kamilkonior
    ContributorAuthor

    @alamb well in my case It is impossible to run query because datafusion(ArrowReader) fails to load parquet file that was serialized via ArrowWriter.

    From what we can see in error below, somehow schema is missing field. But actually ArrowReader reads schema correctly. Just a bit later one field is lost somewhere or maybe struct is incorrectly interpreted somewhere in datafusion.

    expected: Struct([Field { name: \"most\"}, Field { name: \"least\"}]) 
    but found: Struct([Field { name: \"most\"]) at column index 0")
    

    Whats even more interesting using same ArrowReader(ParquetFileArrowReader) as datafusion uses internally, I was able to read this parquet file without issues and access both columns of struct using below snippet.

        let rd = SerializedFileReader::new(file).expect("...");
        let mut pqrd = ParquetFileArrowReader::new(Arc::new(rd));
    
        for batch in pqrd.get_record_reader(60).expect("...") {
            let batch = batch.unwrap();
            let col = batch.column(0);
            let child_data = col.data().child_data();
            println!("{:?}", child_data[0].buffers().get(0));
            println!("{:?}", child_data[1].buffers().get(0));
        }
    
  4. alamb commented on Apr 24, 2022

    @alamb
    Contributor

    Thanks @Cheappie -- I agree that the support in DataFusion is not complete. I have filed #2326 to start tracking what else is needed. Thanks again.

  5. kamilkonior commented on Apr 28, 2022

    @kamilkonior
    ContributorAuthor

    thanks @alamb for taking over on that issue, I will keep my fingers crossed that this feature will arrive in near future

  6. alamb commented on Apr 24, 2023

    @alamb
    Contributor

    Tracking in #2326

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions