perf: improve calculating length performance for nested arrays in row conversion#9079
Merged
Dandandan merged 4 commits intoapache:mainfrom Jan 14, 2026
Conversation
Member
Author
|
run benchmark row_format |
This was referenced Dec 31, 2025
|
🤖 |
|
🤖: Benchmark completed Details
|
…culating-rows-length
Contributor
|
run benchmark row_format |
|
🤖 |
|
🤖: Benchmark completed Details
|
alamb
added a commit
that referenced
this pull request
Jan 13, 2026
…ow conversion (#9080) # Which issue does this PR close? N/A # Rationale for this change Making the row length calculation faster which result in faster row conversion # What changes are included in this PR? 1. Instead of iterating over the bytes and getting the length from the byte slice, we use the offsets directly, this is faster as it saves us going to the buffer 2. Added new API for `GenericByteViewArray` (explained below) # Are these changes tested? Yes # Are there any user-facing changes? Yes, added `lengths` function to `GenericByteViewArray` to get an iterator over the lengths of the items in the array ----- Related to: - #9078 - #9079 --------- Co-authored-by: Andrew Lamb <[email protected]>
…culating-rows-length # Conflicts: # arrow-row/src/lib.rs
Member
Author
|
@alamb can you please review and hopefully merge if approved with no comments |
alamb
reviewed
Jan 13, 2026
Contributor
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thank you for this PR @rluvaton
I have some concerns about the introduction of unsafe APIs and it wasn't clear how this would improve performance (and it doesn't seem like the benchmarks show any improvements)
Did you see improvements in any of your internal benchmarks?
arrow-row/src/lib.rs
Outdated
| } | ||
| } | ||
|
|
||
| /// Returns the length of the row at index `row` in bytes |
Contributor
There was a problem hiding this comment.
Can you please document that "length" means bytes here (not, for example, columns)
arrow-row/src/lib.rs
Outdated
| /// | ||
| /// # Safety | ||
| /// Caller must ensure that `index` is less than the number of offsets (#rows + 1) | ||
| pub unsafe fn row_len_unchecked(&self, index: usize) -> usize { |
Contributor
There was a problem hiding this comment.
Why do we need this new pub unsafe API?
in terms of pub it doesn't seem to be used anywhere other than row_len so we could at least make it non pul
In terms of unsafe, Given that row_len does an assert, what is the rationale for using unsafe rather than just normal array access?
As in, why not
pub fn row_len(&self, row: usize) -> usize {
self.offsets[row+1] - self.offsets[row]
}That would be simpler and not use unsafe
It in theory may have one extra bounds check, but unless the performance gains are compelling I think we should avoid adding new unsafe when possible
Member
Author
There was a problem hiding this comment.
removed the unchecked version and only kept row_len
Member
Author
Although the benchmarks show no improvement, there is no really a benchmark that go through that path extensively. This should improve the performance nonetheless as we are now doing less computation than before. |
Member
Author
|
show benchmark queue |
Dandandan
pushed a commit
that referenced
this pull request
Jan 14, 2026
…n row conversion (#9078) # Which issue does this PR close? N/A # Rationale for this change Making the row length calculation faster which result in faster row conversion # What changes are included in this PR? Instead of iterating over the items in the array and getting the length from the byte slice, we use the offsets directly and zip with nulls if necessary # Are these changes tested? Existing tests # Are there any user-facing changes? Faster encoding ------ Split to 2 more PRs as the other 2 add a change to the public API Related to: - #9079 - #9080 --------- Co-authored-by: Andrew Lamb <[email protected]>
Dandandan
approved these changes
Jan 14, 2026
Contributor
I see, perhaps we can add a benchmark case? |
rluvaton
deleted the
improve-performance-of-rows-encoding-for-calculating-rows-length
branch
Contributor
BTW the benchmark runner was rebooted for some reason. I have fixed it |
Dandandan
pushed a commit
to Dandandan/arrow-rs
that referenced
this pull request
Jan 15, 2026
…ow conversion (apache#9080) # Which issue does this PR close? N/A # Rationale for this change Making the row length calculation faster which result in faster row conversion # What changes are included in this PR? 1. Instead of iterating over the bytes and getting the length from the byte slice, we use the offsets directly, this is faster as it saves us going to the buffer 2. Added new API for `GenericByteViewArray` (explained below) # Are these changes tested? Yes # Are there any user-facing changes? Yes, added `lengths` function to `GenericByteViewArray` to get an iterator over the lengths of the items in the array ----- Related to: - apache#9078 - apache#9079 --------- Co-authored-by: Andrew Lamb <[email protected]>
Dandandan
pushed a commit
to Dandandan/arrow-rs
that referenced
this pull request
Jan 15, 2026
…n row conversion (apache#9078) # Which issue does this PR close? N/A # Rationale for this change Making the row length calculation faster which result in faster row conversion # What changes are included in this PR? Instead of iterating over the items in the array and getting the length from the byte slice, we use the offsets directly and zip with nulls if necessary # Are these changes tested? Existing tests # Are there any user-facing changes? Faster encoding ------ Split to 2 more PRs as the other 2 add a change to the public API Related to: - apache#9079 - apache#9080 --------- Co-authored-by: Andrew Lamb <[email protected]>
Dandandan
pushed a commit
to Dandandan/arrow-rs
that referenced
this pull request
Jan 15, 2026
… conversion (apache#9079) # Which issue does this PR close? N/A # Rationale for this change Making the row length calculation faster which result in faster row conversion # What changes are included in this PR? 1. Instead of iterating over the rows and getting the length from the byte slice, we use the offsets directly, this 2. Added 3 new APIs for `Rows` (explained below) # Are these changes tested? Yes # Are there any user-facing changes? Yes, added 3 functions to `Rows`: - `row_len` - get the row length at index - `row_len_unchecked` - get the row length at index without bound checks - `lengths` - get iterator over the lengths of the rows ----- Related to: - apache#9078 - apache#9080 --------- Co-authored-by: Andrew Lamb <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
Rationale for this change
Making the row length calculation faster which result in faster row conversion
What changes are included in this PR?
Rows(explained below)Are these changes tested?
Yes
Are there any user-facing changes?
Yes, added 3 functions to
Rows:row_len- get the row length at indexrow_len_unchecked- get the row length at index without bound checkslengths- get iterator over the lengths of the rowsRelated to:
GenericByteArrayin row conversion #9078