Skip to content

Increase networking limits #2632

@RaphaelIT7

Description

@RaphaelIT7

Details

Source's networking currently is quite slow, but with some increases it seemingly can reach speeds of 4MB possibly +100MB/s & higher in my tests without anything breaking.
This would definitely require net compat breaking changes.

New commits:
-> Sub channel changes: RaphaelIT7/obsolete-source-engine@2ef348a
-> Networking limit changes: RaphaelIT7/obsolete-source-engine@dbd150b

Changes:
-> Implemented a dynamic net buffer that can allocate up to ~8MB if needed for a transmit.
- -> NET_MAX_PAYLOAD is now used as the stackalloc limit for the net buffer- any size above will be allocated using malloc, which will be slower.
- -> It calculates how big the buffer needs to be to allocate properly handling cases where, for example a full update may require 1MB to be sent.

-> Raised the nMaxQueuedPackets limit inside CQueuedPacketSender::QueuePacket to 63 * 1024 (from 1024)
- -> This is due to far more split packets being sent/created.

-> Optimized CRC creation to use _mm_crc32_u64 (CPUs since ~2009 should support it?)
- -> These are two new seperate methods but the one to use is CRC32_ProcessSingleBufferFast which is used inside BufferToShortChecksum (net_chan.cpp)
- -> The result differs from the normal CRC method! But this should be no issue if both the server & clients use the same method.

-> Added bf_write::WriteBytesAligned (bf_read::ReadBytesAligned counterpart) as a way faster way to write bytes
- -> Previously the engine used bf_write::WriteBytes which is utterly slow, as it actually uses WriteBits soo writing 1MB of bits is shit.
- -> This will align the buffer to a byte and then memcpy the data into the buffer.

-> Made the IFileSystem Interface properly return/handle > 2GB sizes
- -> The FileSystem (atleast in my build) wasn't really ready for +2GB files, so the commit includes changes to handle that.

-> Changed how MAX_SUBCHANNELS works
- -> Implemented SUBCHANNEL_BITS which is calculated at compile time
- -> Allowed it to be raised to any number by using CBitVec's properly.
- -> For this, new methods CBitVec::Flip & CBitVec::CompareBit were added.

-> Changed how CNetChan::UpdateSubChannels determines how many fragments to send.
- -> Previously it used the m_MaxReliablePayloadSize / FRAGMENT_SIZE but that has been switched out.
- -> The count of remaining bytes is passed to it now for it to determine how many fragments it can shove in there.
- -> The unreliable data size has been accounted for in the passed remaining buffer size so that it wouldn't fill it completely so that no snapshot could be sent.

-> Inside CNetChan::SendSubChannelData changed how it reads & writes files to the buffer
- -> The previous implementation would try to stack allocate the entire length, which at 8MB resulted in a stack overflow.

-> Inside NET_FindOrCreateSplitPacketEntry changed how it allocates a new CSplitPacketEntry
- -> Previously it would FIRST stackallocate an CSplitPacketEntry and then move that into the splitPacketEntries list, which internally allocates it and copies it...
- -> The issue is that this was able to stack overflow as the CSplitPacketEntry became too large with our changes.
- -> The new approach directly calls splitPacketEntries.AddToTailGetPtr(); avoiding the whole stackalloc & copying while having the same result and probably better performance.

-> Raised how often NET_ReceiveValidDatagram loops as far more split packets may be sent.
-> Raised NET_COMPRESSION_STACKBUF_SIZE from 4096 to 16384 should be fine?
-> Raised UDP_SO_RCVBUF_SIZE from 131072 to 16777216 (with actual testing in GMod this could probably be lowered later on)
-> Raised max value of net_udp_rcvbuf from 128KB to 32MB

-> Changed how packetID of the SPLITPACKET is networked
- -> If one packet had to be split a lot, it would result in it overflowing the numbers. (This apparently was once a Source Engine exploit?)
- -> It should be able to handle up to ~80MB big transmits/packets, though with this current setup a transmit will be 8MB at max.

-> Raised MAX_RATE from 1MB to 1GB
-> Raised DEFAULT_RATE from 80KB to 1MB
-> Raised net_maxfilesize from max 64 to max 16384(16GB) for testing, should probably be lowered to 4GB or such.

-> Added NUM_FRAGMENT_BITS & raised it from 3 to 6
- -> Controls how many bits are used for the count of fragments that can be sent in a transmit.

-> Added MAX_ADDITIONAL_FRAGMENTS
- -> Controls how many fragments can be sent in a transmit additionally
- -> This means our dynamic buffer will intentionally allocate space for the fragments to be sent.
- -> If this value is, for example 0, it would mean our dynamic buffer would ignore fragments, and the fragments would only be networked if the leftover buffer space is enough to fit one inside.

-> Added net_compresspackets_maxsize to limit at which size packets are compressed (at some sizes it's simply not worth it)

I've spent a bit testing it locally using net_usesocketsforloopback 1 && cl_download 10GB.bin which worked fine, though I am fairly certain there will at minimum be some issues if it's implemented into GMod, though it would need to actually be done for it to be able to be tested.

Note

I couldn't test this with an actual server / do not have any srcds compile.

Having the limits raised would be very useful & would reduce loading times (especially when downloading a map/Lua files from the server) and it shouldn't cause any issues.
There hopefully shouldn't be any security issues with it, and using sv_minrate, sv_maxrate & the added net_ convars, server owners can limit the networking if they wanted to.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Net-Compat-BreakingThis issue requires breaking network compatibility with existing servers to solve.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions