At production server, I have a simple code

mSocket.ReceiveFrom(mBuffer, SocketFlags.None, ref ep);

I call this code every millisecond. Recently I found out that almost 40% of allocations from my application are from this code. And all of the allocations are only 3 primitives: SocketAddress, IPEndPoint, IPAdress. It seems wasteful. Maybe I do something wrong? Can I read data from the UDP socket without this allocations? Can I have managed UDP transport with zero allocations?
Simple reproduction https://github.com/kripergvg/UdpAllocations
dotmemory after 4 minutes