Skip to content

Leaf expression pushdown leaves an unqualified column in an Aggregate's output name, so the plan fails a datafusion-proto round trip #26144

Open
batmnnn/datafusion
#1

Description

@clintonjrobinson

Describe the bug

With datafusion.optimizer.enable_leaf_expression_pushdown = true (the default), ExtractLeafExpressions moves a get_field out of an aggregate argument into an extraction projection, and PushDownLeafProjections moves that projection below a SubqueryAlias. The aggregate keeps an unqualified Column("__datafusion_extracted_N"), so its output field is named max(... __datafusion_extracted_N ...). The optimized plan executes correctly.

logical_plan_to_bytes encodes it, but logical_plan_from_bytes fails: decoding rebuilds the aggregate through LogicalPlanBuilder, which qualifies the column as hdr.__datafusion_extracted_N. The aggregate's output name changes, and the projection above it, which references the old name, no longer resolves.

To Reproduce

With datafusion and datafusion-proto 53.1.0 or 55.2.0 (the same failure with a Parquet table in place of the literal):

use datafusion::prelude::SessionContext;
use datafusion_proto::bytes::{logical_plan_from_bytes, logical_plan_to_bytes};

let ctx = SessionContext::new();
let plan = ctx.sql(
    "WITH t AS (SELECT 'm1' AS id, [named_struct('name', 'Subject', 'value', 'Hello')] AS headers), \
     hdr AS (SELECT id, unnest(headers) AS h FROM t) \
     SELECT id, MAX(CASE WHEN lower(hdr.h['name']) = 'subject' THEN hdr.h['value'] END) AS subject \
     FROM hdr GROUP BY id",
).await?.into_optimized_plan()?;
let bytes = logical_plan_to_bytes(&plan)?;
logical_plan_from_bytes(&bytes, &ctx.task_ctx())?; // fails

On 53.1.0 (55.2.0 adds a list of valid fields):

Schema error: No field named "max(CASE WHEN lower(__datafusion_extracted_1) = Utf8(""subject"") THEN __datafusion_extracted_2 END)". Did you mean 'max(CASE WHEN lower(hdr.__datafusion_extracted_1) = Utf8("subject") THEN hdr.__datafusion_extracted_2 END)'?

The same SQL round-trips with enable_leaf_expression_pushdown = false, and when the struct fields are projected as columns before the aggregate.

Expected behavior

The optimized plan round-trips through datafusion-proto: either the extracted column reference in the aggregate is qualified the way the input schema holds it, or the aggregate expression keeps its original output name with an alias.

Additional context

Related to the name-resolution class in #25459 and #25448 (resolve columns by schema index, not by name).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions