Support FixedSizeList in array methods #6560

Activity

1. izveigor

   added

   enhancementNew feature or requestNew feature or request

   on Jun 6, 2023
2. jayzhan211 commented on Jun 15, 2023

   

   jayzhan211

   on Jun 15, 2023

   Contributor

   More actions

   We should do coercion in datafusion/optimizer/src/analyzer/type_coercion.rs, right?
3. alamb commented on Jun 15, 2023

   

   alamb

   on Jun 15, 2023

   Last edited by alamb

   Contributor

   More actions

   We should do coercion in datafusion/optimizer/src/analyzer/type_coercion.rs, right?

   Yes, that is the preferred location (as it happens before any optimization passes) -- the reason is that type coercion can change the semantics of the plan, but the optimizer passes should not

   https://docs.rs/datafusion/latest/datafusion/index.html#planning has more information

   👍React with 👍1Reacted by Jay Zhan
4. jayzhan211 commented on Jun 16, 2023

   

   jayzhan211

   on Jun 16, 2023

   Last edited by jayzhan211

   Contributor

   More actions

   My first thought is to add casting from FixedSizeList to List in coerce_arguments_for_signature.

   However, I fail to construct a test.
   I don't know how to create a test case with DataType::FixedSizeList in sqllogictests array.slt. make_array([1,2]) gives us ScalarValue::List which converted to DataType::List, and since the args in ScalarFuncion for TypeCoercionRewriter is Vec<Expr>, I could only construct Expr::Literal(ScalarValue::List), no idea to construct FixedSizeList from Expr and ScalarValue in a unit test, any idea about this?
   https://github.com/apache/arrow-datafusion/blob/84e49771b7403b3d313d8493b61d2d58dcdd7514/datafusion/optimizer/src/analyzer/type_coercion.rs#L383
5. alamb commented on Jun 16, 2023

   

   alamb

   on Jun 16, 2023

   Contributor

   More actions

   Since FixedSizeList is a different Arrow type, we probably need a new ScalarValue variant for it. Maybe ScalarValue::FixedSizedList 🤔

   https://docs.rs/datafusion/latest/datafusion/scalar/enum.ScalarValue.html#variant.List
6. alamb commented on Jun 16, 2023

   

   alamb

   on Jun 16, 2023

   Contributor

   More actions

   I wonder if a reasonable approach would be to start with the code to cast to/from List and FixedSizedList -- that way there would at least be some path for working with data that came in as FixedSizedList

   So like

   SELECT 
     arrow_cast(my_column, 'List(Utf8)') -- would call the `cast` kernel to convert `my_column` to `LIstArray`
   FROM
     my_parquet_table_with_fixed_size_lists

   👍React with 👍1Reacted by Jay Zhan
7. jayzhan211 commented on Apr 23, 2024

   

   jayzhan211

   on Apr 23, 2024

   Contributor

   More actions

   I think we have support FixedSizeList already. Thanks for @Weijun-H

   👍React with 👍1Reacted by Andrew Lamb
8. jayzhan211

   closed this as completedon Apr 23, 2024