[Epic] Native StringView support for string functions

@tshauck

Is your feature request related to a problem or challenge?

We are working to add complete StringView support in DataFusion, which permits potentially much faster processing of string data. See #10918 for more background.

Today, most DataFusion string functions support DataType::Utf8 and DataType::LargeUtf8 and when called with a StringView argument DataFusion will cast the argument back to DataType::Utf8 which is expensive.

To realize the full speed of StringView, we need to ensure that all string functions support the DataType::Utf8View directly.

Describe the solution you'd like

Port all string functions

Describe alternatives you've considered

No response

Additional context

See coordination plan with @tshauck and myself here: #11787 (comment)

@Kev1n8

One thing I have noticed during implementations is that some functions such as ltrim/rtrim/btrim could be more efficient if they produced Utf8View as output in addition to accepting them as input

For example, in #11920 (comment) from @Kev1n8 it is actually probably a good idea to always generate StringView as output (rather than StringArray) as it could avoid a copy.

I am thinking once we get the string functions so they can support StringView as input then we can do a second pass and optimize some functions so they produce StringView as output

@Omega359

Inspired by @Omega359 's great PR #11941, I have some suggestion on testing Utf8View support for functions:

Although most implementation is adapted from existing implementation, but the execution takes another path, so I think comprehensive end-to-end tests are still needed.
The good news is there already exists sqllogictests for original string functions, the only thing to do is to duplicate existing testings with Utf8View

Here are the examples on how to adapt existing test cases for Utf8View input

For functions takes scalar value, use arrow_cast() like https://github.com/apache/datafusion/pull/11941/files#diff-51757b2b1d0a07b88551d88eabeba7f74e11b5217e44203ac7c6f613c0221196

For functions read from a table, string column can be converted to Utf8View column like

datafusion/datafusion/sqllogictest/test_files/string_view.slt

Lines 30 to 42 in 2cf0956

    
           # Table with the different combination of column types 
        
           statement ok 
        
           create table test as 
        
           SELECT 
        
             arrow_cast(column1, 'Utf8') as column1_utf8, 
        
             arrow_cast(column2, 'Utf8') as column2_utf8, 
        
             arrow_cast(column1, 'LargeUtf8') as column1_large_utf8, 
        
             arrow_cast(column2, 'LargeUtf8') as column2_large_utf8, 
        
             arrow_cast(column1, 'Utf8View') as column1_utf8view, 
        
             arrow_cast(column2, 'Utf8View') as column2_utf8view, 
        
             arrow_cast(column1, 'Dictionary(Int32, Utf8)') as column1_dict, 
        
             arrow_cast(column2, 'Dictionary(Int32, Utf8)') as column2_dict 
        
           FROM test_source;

We are making pretty good progress here -- just a few more functions left 🚀

I think we can claim this is completed. Follow on work is tracked in

[Epic] A Collection of Additional UTF8View support tickets #13504

🎉

Navigation Menu

Repository navigation

[Epic] Native `StringView` support for string functions #11790

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	# Table with the different combination of column types
	statement ok
	create table test as
	SELECT
	arrow_cast(column1, 'Utf8') as column1_utf8,
	arrow_cast(column2, 'Utf8') as column2_utf8,
	arrow_cast(column1, 'LargeUtf8') as column1_large_utf8,
	arrow_cast(column2, 'LargeUtf8') as column2_large_utf8,
	arrow_cast(column1, 'Utf8View') as column1_utf8view,
	arrow_cast(column2, 'Utf8View') as column2_utf8view,
	arrow_cast(column1, 'Dictionary(Int32, Utf8)') as column1_dict,
	arrow_cast(column2, 'Dictionary(Int32, Utf8)') as column2_dict
	FROM test_source;

[Epic] Native StringView support for string functions #11790

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

alamb commented on Aug 12, 2024

2010YOUY01 commented on Aug 12, 2024

alamb commented on Aug 21, 2024

alamb commented on Nov 21, 2024

alamb commented on Nov 21, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[Epic] Native `StringView` support for string functions #11790