Repository navigation
Update the CONCAT scalar function to support Utf8View #11836
Description
Related: #11766
Currently, a call to CONCAT with a Utf8View datatypes induces a cast. After the change that fixes this issue, it should not.
query TT
EXPLAIN SELECT
concat(column1_utf8view, column2_utf8view) as c
FROM test;
----
logical_plan
01)Projection: concat(CAST(test.column1_utf8view AS Utf8), CAST(test.column2_utf8view AS Utf8)) AS c
02)--TableScan: test projection=[column1_utf8view, column2_utf8view]Concat is defined here: https://github.com/apache/datafusion/blob/main/datafusion/functions/src/string/concat.rs
Is your feature request related to a problem or challenge?
We are working to add complete StringView support in DataFusion, which permits potentially much faster processing of string data. See #10918 for more background.
Today, most DataFusion string functions support DataType::Utf8 and DataType::LargeUtf8 and when called with a StringView argument DataFusion will cast the argument back to DataType::Utf8 which is expensive.
To realize the full speed of StringView, we need to ensure that all string functions support the DataType::Utf8View directly.
Describe the solution you'd like
Update the function to support DataType::Utf8View directly
Describe alternatives you've considered
The typical steps are:
- Write some tests showing the function doesn't support Utf8View (see the tests in
string_view.sltto ensure the arguments are not being cast - Change the
Signatureof the function to acceptUtf8Viewin addition toUtf8/LargeUtf8 - Update the implementation of the function to operate on
Utf8View
Example PRs
- Update to use an arrow kernel that already supports StringView: feat: support
Utf8Viewtype instarts_withfunction #11787 - Change the implementation to support StringView directly: Implement native support StringView for character length #11676
- Change implementation (option 2): Initial support for regex_replace on
StringViewArray#11556
Additional context
The documentation of string functions can be found here: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#string-functions
To test a function with StringView with datafusion-cli you can use an example like this (replacing starts_with with the relevant function)
> create table foo as values (arrow_cast('foo', 'Utf8View'), arrow_cast('bar', 'Utf8View'));
0 row(s) fetched.
Elapsed 0.043 seconds.
> select starts_with(column1, column2) from foo;
+--------------------------------------+
| starts_with(foo.column1,foo.column2) |
+--------------------------------------+
| false |
+--------------------------------------+
1 row(s) fetched.
Elapsed 0.015 seconds.To see if it is using utf8 view, use EXPLAIN to see the plan and verify there is no CAST. In this example the CAST(column1@0 AS Utf8) indicates that the function is not using Utf8View natively
> explain select starts_with(column1, column2) from foo;
+---------------+------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+------------------------------------------------------------------------------------------------------------------------------+
| logical_plan | Projection: starts_with(CAST(foo.column1 AS Utf8), CAST(foo.column2 AS Utf8)) |
| | TableScan: foo projection=[column1, column2] |
| physical_plan | ProjectionExec: expr=[starts_with(CAST(column1@0 AS Utf8), CAST(column2@1 AS Utf8)) as starts_with(foo.column1,foo.column2)] |
| | MemoryExec: partitions=1, partition_sizes=[1] |
| | |
+---------------+------------------------------------------------------------------------------------------------------------------------------+
2 row(s) fetched.
Elapsed 0.006 seconds.It is also often good to test with a constant as well (likewise there should be no cast):
> explain select starts_with(column1, 'foo') from foo;
+---------------+----------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+----------------------------------------------------------------------------------------------------------+
| logical_plan | Projection: starts_with(CAST(foo.column1 AS Utf8), Utf8("foo")) |
| | TableScan: foo projection=[column1] |
| physical_plan | ProjectionExec: expr=[starts_with(CAST(column1@0 AS Utf8), foo) as starts_with(foo.column1,Utf8("foo"))] |
| | MemoryExec: partitions=1, partition_sizes=[1] |
| | |
+---------------+----------------------------------------------------------------------------------------------------------+
2 row(s) fetched.
Elapsed 0.002 seconds.
### Additional context
_No response_
Activity
[-]Update the CONCAT scalar function to support Utf8View[/-][+]Update the `CONCAT` scalar function to support Utf8View[/+][-]Update the `CONCAT` scalar function to support Utf8View[/-][+]Update the `CONCAT` scalar function to support `Utf8View`[/+]dharanad commentedon Aug 6, 2024
take
dharanad commentedon Aug 7, 2024
Maybe i can reference this PR: #11851
tshauck commentedon Aug 7, 2024
That's a good one to reference... You also might consider looking at #11787, which similar to this
concatfunc in that you have to consider multiple arguments and their types (e.g. concat'ing a UTF8View with a UTF8). It also has sqllogictests (slt), which it'd be nice to new PRs to have.alamb commentedon Aug 13, 2024
There are a bunch of PRs now that port functions to Utf8View which can probably be used as examples
tshauck commentedon Aug 28, 2024
@dharanad -- hey, are you still working on this? If not, can I take a look. This is one of the three issues still open on #11790 -- we're really close.
dharanad commentedon Aug 28, 2024
Hey @tshauck , please feel free to work on it
devanbenz commentedon Aug 28, 2024
take