Flaky fuzz tests for filtered outer SortMergeJoin #12359

Activity

1. korowa

   added

   bugSomething isn't workingSomething isn't working

   on Sep 6, 2024
2. comphead commented on Sep 8, 2024

   

   comphead

   on Sep 8, 2024

   Last edited by comphead

   Contributor

   More actions

   left_semi_join was fixed sometime ago. Need to check whats happening with

   test_left_join_1k_filtered
   test_right_join_1k_filtered
   test_full_join_1k_filtered
   

   The repro case is to run the same test 1000 times and depending on data distribution between batches the problem can arise

   👍React with 👍1Reacted by Eduard Karacharov
3. comphead commented on Sep 10, 2024

   

   comphead

   on Sep 10, 2024

   Contributor

   More actions

   fyi @viirya
4. comphead commented on Sep 12, 2024

   

   comphead

   on Sep 12, 2024

   Last edited by comphead

   Contributor

   More actions

   I believe all of them flaky because of flaky LeftAnti join, all Left, Right, Full depends on LeftAnti starting from #10892 PR.
   Checking if thats the case
5. comphead commented on Sep 12, 2024

   

   comphead

   on Sep 12, 2024

   Last edited by comphead

   Contributor

   More actions

   Simple test case for LeftOuter

   #[tokio::test]
   async fn test_left_outer() {
       let left: Vec<RecordBatch> = make_staggered_batches(1);
   
       let left = vec![
           RecordBatch::try_new(
               left[0].schema().clone(),
               vec![
                   Arc::new(Int32Array::from(vec![1])),
                   Arc::new(Int32Array::from(vec![2])),
                   Arc::new(Int32Array::from(vec![10])),
                   Arc::new(Int32Array::from(vec![20])),
               ],
           ).unwrap()
       ];
   
       let right = vec![
           RecordBatch::try_new(
               left[0].schema().clone(),
               vec![
                   Arc::new(Int32Array::from(vec![1, 1])),
                   Arc::new(Int32Array::from(vec![2, 2])),
                   Arc::new(Int32Array::from(vec![0, 30])),
                   Arc::new(Int32Array::from(vec![0, 40])),
               ],
           ).unwrap()
       ];
   
       JoinFuzzTestCase::new(
           left,
           right,
           JoinType::Left,
           Some(Box::new(col_lt_col_filter)),
       )
           .run_test(&[JoinTestType::HjSmj], false)
           .await;
   }
   

   @viirya cc
6. comphead commented on Sep 13, 2024

   

   comphead

   on Sep 13, 2024

   Contributor

   More actions

   For LeftOuter/RightOuter and FullJoin the rootcause is the same. SMJ emits a null row if the filtered match is not found.
   But if the filtered match row is found it should emit the joined row instead of null row. And here we come to the same issue as for LeftAnti, so we need to know when all the right rows processed for the given left row.

   Given that SMJ emits rows in freeze_streamed which can be called anytime once the output size is equal to batch size there is 2 possible options:

   • Try to calculate if the SMJ is about to switch to ordering == Less which moves left pointer down, meaning everything is processed for the row
   • Move emit to some other place, for example we can fill the buffer with rows and filtered flags until the ordering switch, and only then make a final emit based on the data in the buffer + filtered
7. comphead commented on Nov 5, 2024

   

   comphead

   on Nov 5, 2024

   Contributor

   More actions

   Related apache/datafusion-comet#398
8. comphead

   self-assigned this

   on Nov 5, 2024
9. alamb

   closed this as completedin #13369on Nov 13, 2024