You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
lpad / rpad with a non-literal pad argument falls back to Spark #5579
lpad and rpad decline two shapes, and both fall the whole projection back to Spark:
if (expr.str.isInstanceOf[Literal]) {
returnUnsupported(Some(PadReasons.literalStrReason))
}
if (!expr.pad.isInstanceOf[Literal]) {
returnUnsupported(Some(PadReasons.nonLiteralPadReason))
}
spark/src/main/scala/org/apache/comet/serde/strings.scala:413-419 (CometStringRPad) and :441-447 (CometStringLPad)
The non-literal pad restriction is the one that matters. lpad(name, 10, pad_col) — padding with a value that varies per row — is a perfectly ordinary query shape, and there is nothing about it that Spark's generated code cannot do. StringRPad and StringLPad are plain codegen expressions over StringType and IntegerType, all of which CometBatchKernelCodegen.isSupportedDataType admits.
The literal-str restriction is lower value (ConstantFolding handles most of it) but costs nothing extra to cover.
Fix: mix CodegenDispatchFallback into both serdes.
Worth checking while in here: docs/source/user-guide/latest/expressions.md renders lpad and rpad with an em-dash in the Implementation column, meaning GenerateDocs found no serde registered under those function names. Either the registry name differs from the function name or the rows need fixing — either way the doc currently says nothing useful about how these execute.
Activity
Satyr09 commentedon Sep 7, 2026
take