Skip to content

Similar to the "count-bug" case that produces incorrect results #15032

Closed

Description

@suibianwanwank

Describe the bug

For cases similar to the well-known "count-bug", the correlated subquery returns the incorrect result.

To Reproduce

> select * from 'data.csv';
+------+---+
| a    | b |
+------+---+
| 1    | 2 |
| NULL | 0 |
+------+---+
2 row(s) fetched. 
Elapsed 0.003 seconds.

> select e.b ,(select case when max(e2.a) > 10 then 'a' else 'b' end from 'data.csv' e2 where e2.b = e.b+1 ) from  'data.csv' e;
+---+-------------------------------------------------------------------+
| b | CASE WHEN max(e2.a) > Int64(10) THEN Utf8("a") ELSE Utf8("b") END |
+---+-------------------------------------------------------------------+
| 0 | NULL                                                              |
| 2 | NULL                                                              |
+---+-------------------------------------------------------------------+
2 row(s) fetched. 
Elapsed 0.004 seconds.

Expected behavior

Subqueries with no matches should return 'b' instead of null.

Additional context

What is count-bug: Optimization of Nested SQL Queries Revisited
A paper describing this issue: Parameterized ueries and Nesting Equivalences

The well-known "count-bug" is not specific to the count aggregate, and outer-join does not solve it.
The anomaly can occur on any aggregate function; aggregates need modification to distiguish empty set from null values; and optimizing out the outerjoin depends on utilization context

Activity

  1. suibianwanwank commented on Mar 5, 2025

    @suibianwanwank
    ContributorAuthor

    take

  2. suibianwanwank commented on Apr 2, 2025

    @suibianwanwank
    ContributorAuthor

    Additionally, I tried this query in several other databases—here are some execution plans.
    SQL server execution plan:

    |--Compute Scalar(DEFINE:([Expr1006]=CASE WHEN [Expr1004]=(1) THEN 'A' ELSE 'B' END))
    	|--Hash Match(Right Outer Join, HASH:([B].[b])=([A].[b]), RESIDUAL:([test].[test].[B].[b] as [A].[b]=[test].[test].[B].[b] as [B].[b]))
    		|--Stream Aggregate(GROUP BY:([B].[b]) DEFINE:([Expr1004]=MAX([test].[test].[B].[a] as [B].[a])))
    		|	|--Sort(ORDER BY:([B].[b] ASC))
                    |   	|--Table Scan(OBJECT:([test].[test].[B] AS [B]))
    		|--Table Scan(OBJECT:([test].[test].[B] AS [A]))
    

    Presto execution plan:

    Presto Plan:
    - Output[PlanNodeId 50][b, _col1] => [field_0:integer, expr_35:varchar(1)]
            Estimates: {source: CostBasedSourceInfo, rows: 3 (180B), cpu: 558.00, memory: 91.00, network: 0.00}
            b := field_0 (9:5)
            _col1 := expr_35 (10:5)
        - Project[PlanNodeId 406][projectLocality = LOCAL] => [field_0:integer, expr_35:varchar(1)]
                Estimates: {source: CostBasedSourceInfo, rows: 3 (180B), cpu: 558.00, memory: 91.00, network: 0.00}
                expr_35 := SWITCH(BOOLEAN'true', WHEN((max) > (INTEGER'10'), VARCHAR'a'), VARCHAR'b') (12:22)
            - Aggregate(STREAMING)[field_0, unique][PlanNodeId 405] => [field_0:integer, unique:bigint, max:integer]
                    Estimates: {source: CostBasedSourceInfo, rows: 3 (180B), cpu: 378.00, memory: 91.00, network: 0.00}
                    max := "presto.default.max"((field_16)) (11:22)
                - LeftJoin[PlanNodeId 404][("add" = "field_17")][$hashvalue, $hashvalue_47] => [field_0:integer, unique:bigint, field_16:integer]
                        Estimates: {source: CostBasedSourceInfo, rows: 3 (180B), cpu: 329.00, memory: 34.00, network: 0.00}
                        Distribution: REPLICATED
                    - Project[PlanNodeId 816][projectLocality = LOCAL] => [field_0:integer, unique:bigint, add:integer, $hashvalue:bigint]
                            Estimates: {source: CostBasedSourceInfo, rows: 2 (120B), cpu: 122.00, memory: 0.00, network: 0.00}
                            $hashvalue := combine_hash(BIGINT'0', COALESCE($operator$hash_code(add), BIGINT'0')) (2:6)
                        - Project[PlanNodeId 489][projectLocality = LOCAL] => [field_0:integer, unique:bigint, add:integer]
                                Estimates: {source: CostBasedSourceInfo, rows: 2 (120B), cpu: 66.00, memory: 0.00, network: 0.00}
                                add := (field_0) + (INTEGER'1') (2:6)
                            - AssignUniqueId[PlanNodeId 403] => [field_0:integer, unique:bigint]
                                    Estimates: {source: CostBasedSourceInfo, rows: 2 (120B), cpu: 28.00, memory: 0.00, network: 0.00}
                                - LocalExchange[PlanNodeId 778][ROUND_ROBIN] () => [field_0:integer]
                                        Estimates: {source: CostBasedSourceInfo, rows: 2 (120B), cpu: 10.00, memory: 0.00, network: 0.00}
                                    - Values[PlanNodeId 0] => [field_0:integer]
                                            Estimates: {source: CostBasedSourceInfo, rows: 2 (120B), cpu: 0.00, memory: 0.00, network: 0.00}
                                            (INTEGER'2')
                                            (INTEGER'0')
                    - LocalExchange[PlanNodeId 779][HASH][$hashvalue_47] (field_17) => [field_16:integer, field_17:integer, $hashvalue_47:bigint]
                            Estimates: {source: CostBasedSourceInfo, rows: 2 (120B), cpu: 68.00, memory: 0.00, network: 0.00}
                        - Project[PlanNodeId 817][projectLocality = LOCAL] => [field_16:integer, field_17:integer, $hashvalue_48:bigint]
                                Estimates: {source: CostBasedSourceInfo, rows: 2 (120B), cpu: 34.00, memory: 0.00, network: 0.00}
                                $hashvalue_48 := combine_hash(BIGINT'0', COALESCE($operator$hash_code(field_17), BIGINT'0')) (3:9)
                            - Values[PlanNodeId 28] => [field_16:integer, field_17:integer]
                                    Estimates: {source: CostBasedSourceInfo, rows: 2 (120B), cpu: 0.00, memory: 0.00, network: 0.00}
                                    (INTEGER'1', INTEGER'2')
                                    (null, INTEGER'0')
    

    BTW, I ran this query in DuckDB and found that DuckDB also has this issue.
    FYI, @xudong963 @jayzhan211

  3. jayzhan211 commented on Apr 2, 2025

    @jayzhan211
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions