Repository navigation
Update REGEXP_MATCH scalar function to support Utf8View #11911
Description
Currently, a call to REGEXP_MATCH with a Utf8View datatypes induces a cast. After the change that fixes this issue, it should not.
query TT
EXPLAIN SELECT
REGEXP_MATCH(column1_utf8view, '^https?://(?:www\.)?([^/]+)/.*$') AS k
FROM test;
----
logical_plan
01)Projection: regexp_match(CAST(test.column1_utf8view AS Utf8), Utf8("^https?://(?:www\.)?([^/]+)/.*$")) AS k
02)--TableScan: test projection=[column1_utf8view]REGEXP_MATCH is defined here: https://github.com/apache/datafusion/blob/main/datafusion/functions/src/regex/regexpmatch.rs
casting tests are in: https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/test_files/string_view.slt
Is your feature request related to a problem or challenge?
We are working to add complete StringView support in DataFusion, which permits potentially much faster processing of string data. See #10918 for more background.
Today, most DataFusion string functions support DataType::Utf8 and DataType::LargeUtf8 and when called with a StringView argument DataFusion will cast the argument back to DataType::Utf8 which is expensive.
To realize the full speed of StringView, we need to ensure that all string functions support the DataType::Utf8View directly.
Describe the solution you'd like
Update the function to support DataType::Utf8View directly
Describe alternatives you've considered
The typical steps are:
- Write some tests showing the function doesn't support Utf8View (see the tests in
string_view.sltto ensure the arguments are not being cast - Change the
Signatureof the function to acceptUtf8Viewin addition toUtf8/LargeUtf8 - Update the implementation of the function to operate on
Utf8View
Example PRs
- Update to use an arrow kernel that already supports StringView: feat: support
Utf8Viewtype instarts_withfunction #11787 - Change the implementation to support StringView directly: Implement native support StringView for character length #11676
- Change implementation (option 2): Initial support for regex_replace on
StringViewArray#11556
Additional context
The documentation of string functions can be found here: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#string-functions
To test a function with StringView with datafusion-cli you can use an example like this (replacing starts_with with the relevant function)
> create table foo as values (arrow_cast('foo', 'Utf8View'), arrow_cast('bar', 'Utf8View'));
0 row(s) fetched.
Elapsed 0.043 seconds.
> select starts_with(column1, column2) from foo;
+--------------------------------------+
| starts_with(foo.column1,foo.column2) |
+--------------------------------------+
| false |
+--------------------------------------+
1 row(s) fetched.
Elapsed 0.015 seconds.To see if it is using utf8 view, use EXPLAIN to see the plan and verify there is no CAST. In this example the CAST(column1@0 AS Utf8) indicates that the function is not using Utf8View natively
> explain select starts_with(column1, column2) from foo;
+---------------+------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+------------------------------------------------------------------------------------------------------------------------------+
| logical_plan | Projection: starts_with(CAST(foo.column1 AS Utf8), CAST(foo.column2 AS Utf8)) |
| | TableScan: foo projection=[column1, column2] |
| physical_plan | ProjectionExec: expr=[starts_with(CAST(column1@0 AS Utf8), CAST(column2@1 AS Utf8)) as starts_with(foo.column1,foo.column2)] |
| | MemoryExec: partitions=1, partition_sizes=[1] |
| | |
+---------------+------------------------------------------------------------------------------------------------------------------------------+
2 row(s) fetched.
Elapsed 0.006 seconds.It is also often good to test with a constant as well (likewise there should be no cast):
> explain select starts_with(column1, 'foo') from foo;
+---------------+----------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+----------------------------------------------------------------------------------------------------------+
| logical_plan | Projection: starts_with(CAST(foo.column1 AS Utf8), Utf8("foo")) |
| | TableScan: foo projection=[column1] |
| physical_plan | ProjectionExec: expr=[starts_with(CAST(column1@0 AS Utf8), foo) as starts_with(foo.column1,Utf8("foo"))] |
| | MemoryExec: partitions=1, partition_sizes=[1] |
| | |
+---------------+----------------------------------------------------------------------------------------------------------+
2 row(s) fetched.
Elapsed 0.002 seconds.
### Additional context
_No response_
Activity
PsiACE commentedon Aug 12, 2024
take
Omega359 commentedon Sep 29, 2024
Note that I just saw that apache/arrow-rs#6370 was merged into arrow-rs which may be relevant to this work
Omega359 commentedon Nov 11, 2024
Filed upstream ticket apache/arrow-rs#6717 to have regexp_match updated to support StringViewArray's. In the meantime we either wait for a release with that update or we implement it in DF and remove it once it's implemented upstream.
alamb commentedon Jan 27, 2025
I think we now have the needed upstream support in arrow-rs for this but it will not be available until arrow 54.1.0 -- slated to be released this week
See commit apache/arrow-rs@5249f99
alamb commentedon Feb 3, 2025
@Weijun-H updated us
54.1.0today inarrow/parquet54.1.0and fix clippy/ci #14415We are ready to rock on this PR
Omega359 commentedon Feb 3, 2025
I am working on this now. Code changes are pretty simple, the .slt test needs to be refactored to reflect the changes made to the strings slt tests for improved readability
alamb commentedon Feb 3, 2025
Thanks @Omega359 -- I will review it with urgency!
You probably already found it but if you put the tests in here:
https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest/test_files/string
You'll end up with coverage for all the different string types with a single set of tests
Omega359 commentedon Feb 3, 2025
Yup. I'm actually just going to put them in regexp.slt for now and file a PR to refactor it instead just to get this in as quick as possible.