Invalid argument error: the data type binary has no natural order #7343

Activity

1. JayjeetAtGithub

   added

   bugSomething isn't workingSomething isn't working

   on Aug 21, 2023
2. alamb commented on Aug 21, 2023

   

   alamb

   on Aug 21, 2023

   Contributor

   More actions

   This looks similar to #7039 which @jonahgao fixed by adding a coercion from binary --> UTF8 for comparison. I think we could do something similar here.
3. alamb

   added

   good first issueGood for newcomersGood for newcomers

   on Aug 21, 2023
4. alamb commented on Aug 21, 2023

   

   alamb

   on Aug 21, 2023

   Contributor

   More actions

   Marking as a good first issue as there is a reproducer and I think the fix should be relatively straightforward
5. tustvold commented on Aug 21, 2023

   

   tustvold

   on Aug 21, 2023

   Contributor

   More actions

   This may be fixed in the next arrow release, which adds Binary support to lexsort

   👍React with 👍4Reacted by Jayjeet Chakraborty, Daniël Heres, Andrew Lamb and Alex Huang
6. alamb

   removed

   good first issueGood for newcomersGood for newcomers

   on Aug 21, 2023
7. yjshen commented on Aug 27, 2023

   

   yjshen

   on Aug 27, 2023

   Member

   More actions

   A slightly simpler way to reproduce without the need for datafusion-cli:

   #[tokio::test]
   async fn binary_order() -> Result<()> {
       let schema = Arc::new(Schema::new(vec![
           Field::new("a", DataType::Binary, false),
           Field::new("b", DataType::Int32, false),
       ]));
       let batch = RecordBatch::try_new(
           schema.clone(),
           vec![Arc::new(BinaryArray::from(vec![
               "a".as_bytes(),
               "b".as_bytes(),
               "c".as_bytes(),
           ])),
           Arc::new(Int32Array::from(vec![1, 2, 3]))],
       )?;
   
       let ctx = SessionContext::new();
       ctx.register_batch("aa", batch)?;
       let result = ctx.sql("select a from aa where a <> '' order by b, a").await?;
       let result = result.collect().await?;
       let expected = vec![
           "+----+",
           "| a  |",
           "+----+",
           "| 61 |",
           "| 62 |",
           "| 63 |",
           "+----+",
       ];
       assert_batches_eq!(expected, &result);
       Ok(())
   }
8. alamb commented on Oct 16, 2023

   

   alamb

   on Oct 16, 2023

   Contributor

   More actions

   I have verified that this now works in datafusion 34. Tests added in #7839
9. alamb

   closed this as completedon Oct 16, 2023