Repository navigation
[datafusion-spark] Implement size function #5338
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
There is a user on Discord that wants size akin to the PySpark function with the same name to be implemented.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Implement size as a BuiltinScalarFunction for ListArray, LargeListArray and MapArray.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Activity
comphead commentedon Feb 20, 2023
btw, there is no
sizefunction in PG. There arecardinalityfunction orarray_length.Its again should we support both spark/PG syntax ?
alamb commentedon Feb 20, 2023
I would personally suggest supporting postgres syntax unless postgres doesn't support the feature -- and in that case we could follow spark
iajoiner commentedon Feb 20, 2023
@alamb Sure! So I guess
array_lengthshould be the correct name here.iajoiner commentedon Feb 20, 2023
@alamb Wait. In Postgres https://www.postgresql.org/docs/15/functions-array.html it is necessary to have compatibility to have nested arrays so it does make sense to have
array_length(array, dimension). For us this makes less sense since we can have arrays that look like[[2, 3], [4, 5, 6]]and even lists of maps.cardinalitydoes make more sense but it doesn't suit our current use case.sizefrom Spark probably makes the most sense here.[-]Add `size` as a `BuiltinScalarFunction`[/-][+][datafusion-spark] Implement `size` function[/+]Jefffrey commentedon Oct 20, 2025
I'll rebrand this as a Spark function issue.
Reference: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.size.html
We want to support
size()function on array + map types. Though we do havearray_lengthalready, this is meant for arrays so can be confusing to extend it to support maps.Jefffrey commentedon Oct 20, 2025
I'll mark as good first issue, some pointers:
array_lengthimplementation in DataFusion for reference as I mentioned above; can ignore all the code around handlingdimensions though since Sparksizedoesn't do thatCuteChuanChuan commentedon Oct 27, 2025
take
Yuvraj-cyborg commentedon Dec 11, 2025
@Jefffrey is this issue resolved yet or can I work on it ??
CuteChuanChuan commentedon Dec 11, 2025
Hi @Yuvraj-cyborg,
I am currently working on this. Will submit a PR soon.
feat: implement Spark size function for arrays and maps (#19592)
feat: implement Spark size function for arrays and maps (apache#19592)