You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Source's networking currently is quite slow, but with some increases it seemingly can reach speeds of 4MB possibly +100MB/s & higher in my tests without anything breaking.
This would definitely require net compat breaking changes.
Changes:
-> Implemented a dynamic net buffer that can allocate up to ~8MB if needed for a transmit.
- -> NET_MAX_PAYLOAD is now used as the stackalloc limit for the net buffer- any size above will be allocated using malloc, which will be slower.
- -> It calculates how big the buffer needs to be to allocate properly handling cases where, for example a full update may require 1MB to be sent.
-> Raised the nMaxQueuedPackets limit inside CQueuedPacketSender::QueuePacket to 63 * 1024 (from 1024)
- -> This is due to far more split packets being sent/created.
-> Optimized CRC creation to use _mm_crc32_u64 (CPUs since ~2009 should support it?)
- -> These are two new seperate methods but the one to use is CRC32_ProcessSingleBufferFast which is used inside BufferToShortChecksum (net_chan.cpp)
- -> The result differs from the normal CRC method! But this should be no issue if both the server & clients use the same method.
-> Added bf_write::WriteBytesAligned (bf_read::ReadBytesAligned counterpart) as a way faster way to write bytes
- -> Previously the engine used bf_write::WriteBytes which is utterly slow, as it actually uses WriteBits soo writing 1MB of bits is shit.
- -> This will align the buffer to a byte and then memcpy the data into the buffer.
-> Made the IFileSystem Interface properly return/handle > 2GB sizes
- -> The FileSystem (atleast in my build) wasn't really ready for +2GB files, so the commit includes changes to handle that.
-> Changed how MAX_SUBCHANNELS works
- -> Implemented SUBCHANNEL_BITS which is calculated at compile time
- -> Allowed it to be raised to any number by using CBitVec's properly.
- -> For this, new methods CBitVec::Flip & CBitVec::CompareBit were added.
-> Changed how CNetChan::UpdateSubChannels determines how many fragments to send.
- -> Previously it used the m_MaxReliablePayloadSize / FRAGMENT_SIZE but that has been switched out.
- -> The count of remaining bytes is passed to it now for it to determine how many fragments it can shove in there.
- -> The unreliable data size has been accounted for in the passed remaining buffer size so that it wouldn't fill it completely so that no snapshot could be sent.
-> Inside CNetChan::SendSubChannelData changed how it reads & writes files to the buffer
- -> The previous implementation would try to stack allocate the entire length, which at 8MB resulted in a stack overflow.
-> Inside NET_FindOrCreateSplitPacketEntry changed how it allocates a new CSplitPacketEntry
- -> Previously it would FIRST stackallocate an CSplitPacketEntry and then move that into the splitPacketEntries list, which internally allocates it and copies it...
- -> The issue is that this was able to stack overflow as the CSplitPacketEntry became too large with our changes.
- -> The new approach directly calls splitPacketEntries.AddToTailGetPtr(); avoiding the whole stackalloc & copying while having the same result and probably better performance.
-> Raised how often NET_ReceiveValidDatagram loops as far more split packets may be sent.
-> Raised NET_COMPRESSION_STACKBUF_SIZE from 4096 to 16384 should be fine?
-> Raised UDP_SO_RCVBUF_SIZE from 131072 to 16777216 (with actual testing in GMod this could probably be lowered later on)
-> Raised max value of net_udp_rcvbuf from 128KB to 32MB
-> Changed how packetID of the SPLITPACKET is networked
- -> If one packet had to be split a lot, it would result in it overflowing the numbers. (This apparently was once a Source Engine exploit?)
- -> It should be able to handle up to ~80MB big transmits/packets, though with this current setup a transmit will be 8MB at max.
-> Raised MAX_RATE from 1MB to 1GB
-> Raised DEFAULT_RATE from 80KB to 1MB
-> Raised net_maxfilesize from max 64 to max 16384(16GB) for testing, should probably be lowered to 4GB or such.
-> Added NUM_FRAGMENT_BITS & raised it from 3 to 6
- -> Controls how many bits are used for the count of fragments that can be sent in a transmit.
-> Added MAX_ADDITIONAL_FRAGMENTS
- -> Controls how many fragments can be sent in a transmit additionally
- -> This means our dynamic buffer will intentionally allocate space for the fragments to be sent.
- -> If this value is, for example 0, it would mean our dynamic buffer would ignore fragments, and the fragments would only be networked if the leftover buffer space is enough to fit one inside.
-> Added net_compresspackets_maxsize to limit at which size packets are compressed (at some sizes it's simply not worth it)
I've spent a bit testing it locally using net_usesocketsforloopback 1 && cl_download 10GB.bin which worked fine, though I am fairly certain there will at minimum be some issues if it's implemented into GMod, though it would need to actually be done for it to be able to be tested.
Note
I couldn't test this with an actual server / do not have any srcds compile.
Having the limits raised would be very useful & would reduce loading times (especially when downloading a map/Lua files from the server) and it shouldn't cause any issues.
There hopefully shouldn't be any security issues with it, and using sv_minrate, sv_maxrate & the added net_ convars, server owners can limit the networking if they wanted to.
Details
Source's networking currently is quite slow, but with some increases it seemingly can reach speeds of
4MBpossibly +100MB/s & higher in my tests without anything breaking.This would definitely require net compat breaking changes.
New commits:
-> Sub channel changes: RaphaelIT7/obsolete-source-engine@2ef348a
-> Networking limit changes: RaphaelIT7/obsolete-source-engine@dbd150b
Changes:
-> Implemented a dynamic net buffer that can allocate up to ~8MB if needed for a transmit.
- ->
NET_MAX_PAYLOADis now used as the stackalloc limit for the net buffer- any size above will be allocated using malloc, which will be slower.- -> It calculates how big the buffer needs to be to allocate properly handling cases where, for example a full update may require 1MB to be sent.
-> Raised the
nMaxQueuedPacketslimit insideCQueuedPacketSender::QueuePacketto63 * 1024(from1024)- -> This is due to far more split packets being sent/created.
-> Optimized CRC creation to use _mm_crc32_u64 (CPUs since ~2009 should support it?)
- -> These are two new seperate methods but the one to use is
CRC32_ProcessSingleBufferFastwhich is used insideBufferToShortChecksum(net_chan.cpp)- -> The result differs from the normal CRC method! But this should be no issue if both the server & clients use the same method.
-> Added
bf_write::WriteBytesAligned(bf_read::ReadBytesAlignedcounterpart) as a way faster way to write bytes- -> Previously the engine used
bf_write::WriteByteswhich is utterly slow, as it actually usesWriteBitssoo writing 1MB of bits is shit.- -> This will align the buffer to a byte and then memcpy the data into the buffer.
-> Made the IFileSystem Interface properly return/handle > 2GB sizes
- -> The FileSystem (atleast in my build) wasn't really ready for +2GB files, so the commit includes changes to handle that.
-> Changed how
MAX_SUBCHANNELSworks- -> Implemented
SUBCHANNEL_BITSwhich is calculated at compile time- -> Allowed it to be raised to any number by using
CBitVec's properly.- -> For this, new methods
CBitVec::Flip&CBitVec::CompareBitwere added.-> Changed how
CNetChan::UpdateSubChannelsdetermines how many fragments to send.- -> Previously it used the
m_MaxReliablePayloadSize / FRAGMENT_SIZEbut that has been switched out.- -> The count of remaining bytes is passed to it now for it to determine how many fragments it can shove in there.
- -> The unreliable data size has been accounted for in the passed remaining buffer size so that it wouldn't fill it completely so that no snapshot could be sent.
-> Inside
CNetChan::SendSubChannelDatachanged how it reads & writes files to the buffer- -> The previous implementation would try to stack allocate the entire length, which at 8MB resulted in a stack overflow.
-> Inside
NET_FindOrCreateSplitPacketEntrychanged how it allocates a newCSplitPacketEntry- -> Previously it would FIRST stackallocate an
CSplitPacketEntryand then move that into thesplitPacketEntrieslist, which internally allocates it and copies it...- -> The issue is that this was able to stack overflow as the
CSplitPacketEntrybecame too large with our changes.- -> The new approach directly calls
splitPacketEntries.AddToTailGetPtr();avoiding the whole stackalloc & copying while having the same result and probably better performance.-> Raised how often
NET_ReceiveValidDatagramloops as far more split packets may be sent.-> Raised
NET_COMPRESSION_STACKBUF_SIZEfrom4096to16384should be fine?-> Raised
UDP_SO_RCVBUF_SIZEfrom131072to16777216(with actual testing in GMod this could probably be lowered later on)-> Raised max value of
net_udp_rcvbuffrom128KBto32MB-> Changed how
packetIDof theSPLITPACKETis networked- -> If one packet had to be split a lot, it would result in it overflowing the numbers. (This apparently was once a Source Engine exploit?)
- -> It should be able to handle up to ~80MB big transmits/packets, though with this current setup a transmit will be 8MB at max.
-> Raised
MAX_RATEfrom1MBto1GB-> Raised
DEFAULT_RATEfrom80KBto1MB-> Raised
net_maxfilesizefrom max64to max16384(16GB) for testing, should probably be lowered to 4GB or such.-> Added
NUM_FRAGMENT_BITS& raised it from3to6- -> Controls how many bits are used for the count of fragments that can be sent in a transmit.
-> Added
MAX_ADDITIONAL_FRAGMENTS- -> Controls how many fragments can be sent in a transmit additionally
- -> This means our dynamic buffer will intentionally allocate space for the fragments to be sent.
- -> If this value is, for example 0, it would mean our dynamic buffer would ignore fragments, and the fragments would only be networked if the leftover buffer space is enough to fit one inside.
-> Added
net_compresspackets_maxsizeto limit at which size packets are compressed (at some sizes it's simply not worth it)I've spent a bit testing it locally using
net_usesocketsforloopback 1&&cl_download 10GB.binwhich worked fine, though I am fairly certain there will at minimum be some issues if it's implemented into GMod, though it would need to actually be done for it to be able to be tested.Note
I couldn't test this with an actual server / do not have any srcds compile.
Having the limits raised would be very useful & would reduce loading times (especially when downloading a map/Lua files from the server) and it shouldn't cause any issues.
There hopefully shouldn't be any security issues with it, and using
sv_minrate,sv_maxrate& the addednet_convars, server owners can limit the networking if they wanted to.