Less copying, more batch optimisation#820
Merged
phillip-stephens merged 2 commits intozmap:mainfrom Mar 14, 2024
Merged
Conversation
Introduce new prepare_packet callback for the initialization of send buffers; contrary to the per-thread initialization callback where this was done previously, prepare_packet can be called multiple times, once for each send buffer. Make use of this to prepare packets to send directly in the batch buffers instead of copying them over.
The ip field was only still used in send-bsd and only on a failure path for logging, which does not seem like a strong justification for keeping it around, especially given that it can always be read from packet data. While removing the ip field, align buf such that the IP header is going to start at a 32 bit aligned address, improving perf of IP header field access.
zakird
approved these changes
Mar 13, 2024
Member
zakird
left a comment
zakird
left a comment
There was a problem hiding this comment.
Looks sane to me. @phillip-stephens can you take a deeper pass?
phillip-stephens
approved these changes
Mar 14, 2024
Contributor
phillip-stephens
left a comment
phillip-stephens
left a comment
There was a problem hiding this comment.
LGTM! I wasn't able to test the --iplayer option but that's due to issue #821 , so I'm good to merge.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
thread_initializecallback intothread_initializeandprepare_packet, allowing the latter to be called on each buffer in a batch.ipfield frombatch->packets[], as it was only used on an error path of BSD's send code, and the IP can also be read from the packet data. While removing theipfield, align the packet data in batch such that the IP headers start at a 32-bit boundary, speeding up 32-bit header field access.These combined get me ~ 2.9 % send rate improvement, two thirds of which coming from the first change. Again, I realize some or all of this may be arguable.
The change in probe module interface could also be a step towards potential future work of letting probe modules write directly into mapped NIC memory in netmap mode (tho I suspect that might be more intrusive than it's worth).
This is a revised version of #818, now without breaking
--dryrun. Tested on macOS Sonoma, FreeBSD 14 and Ubuntu 23.10; smoke tested various probe modules;test-shard.shandtest_big_group.shsucceed.