You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Improve performance of SUBSTR for StringViewArray #12031
However, the initial implementation returns an output StringArray when the input is a StringViewArray, which means all the strings are copied
In some functions, such as substr, this extra copy is unnecessary and only the views (aka the i128s that make up the pointers). See GenericByteViewArray for more details
Describe the solution you'd like
I think we can avoid the copy when the input uses StringViewArray and thus make substr faster
Describe alternatives you've considered
The idea would be to
Create a benchmark for the substring function for StringArray, LargeStringArray and StringViewArray
Optimize the implementation of substr
The optimization would likely look like:
Change the signature of substr so it produces a StringViewArray when its first argument is a StringViewArray (at the moment it produces StringArray when its argument is a StringViewArray)
Make a function that took StringViewArray as input and produced another StringViewArray as output
@XiangpengHao can you remember any example functions / kernels in arrow-rs that only manipulate the views in this way that we can point to as an example for this function?
Alternatively, we can also create a builder, then use append_block to add all blocks from old array (array.data_buffers()) to the builder, and iterate the views, if the len is smaller than 12 bytes, we can directly call append_value, if it is larger than 12 bytes, then we call append_view_unchecked. (It is simpler than it looks)
Activity
alamb commentedon Aug 16, 2024
@XiangpengHao can you remember any example functions / kernels in arrow-rs that only manipulate the views in this way that we can point to as an example for this function?
XiangpengHao commentedon Aug 16, 2024
I don't have an exact match, but the
takekernel can be a good candidate: https://github.com/apache/arrow-rs/blob/042d725888358c73cd2a0d58868ea5c4bad778f7/arrow-select/src/take.rs#L481-L491It basically breaks the string view array into smaller pieces and then assemble them together.
Alternatively, we can also create a builder, then use
append_blockto add all blocks from old array (array.data_buffers()) to the builder, and iterate the views, if the len is smaller than 12 bytes, we can directly callappend_value, if it is larger than 12 bytes, then we callappend_view_unchecked. (It is simpler than it looks)Kev1n8 commentedon Aug 16, 2024
take