@ethanaobrien has covered most of what I would have commented on in this review - but I'll add the following:

Please add JSDoc comments. That way if anyone needs to work with this code we all know what it's doing.

While I do know that much of the rest of the code doesn't have comments - this is something I'm trying to change. It makes it far easier for everyone to know what your intentions around a function are.

To note ai is pretty good at writing jsdoc comments if you feed it the entire project

Alright, @ethanaobrien and @michael-j-green I'm confident in this updated code. It works great and I edited the code directly on Github so there shouldn't be any reformatting. defineNetplayFunctions is broken down and I had AI write JSdoc comments, which is much more wordy than I am, so it should be plenty detailed :)

@S-Coding23 Could you please pull in this commit from upstream?

9b0e804

@S-Coding23 Could you please pull in this commit from upstream?

9b0e804

Done

@S-Coding23 I have been contributing to romm a little bit, and I am building out my own EmulatorJS based Netplay fork I work on, with different goals in mind. I built a mediasoup relay server or "SFU" as an alternative to the P2P netplay framework, which requires hosting, but can offer a more feature rich netplay experience than P2P can, and supports 4P (or more) gameplay much better.

Throughout my own work for the past few months on this, I've come across some optimizations that might help you with the P2P netplay here. Yui suggested I could share my ideas here with you.

One idea, and maybe you've implemented this cause it's a lot of code to check through right now... Use separate channels for the video and the audio streams - this divorces the video timings from the audio timings, so that latency or microstutters in the audio do not force the video feed to frameskip or lockup. It can potentially lead to desyncing the audio but if the host has half decent hardware and your logic handles the audio stream well, that shouldn't be an issue. This results normally in much much smoother video streams for clients.

Second, is that I do see you are still using VP8. VP9 is actually much better for this application. It offers superior video compression, for faster streams, better quality, and reduced bitrates. All EmulatorJS users are running in browser webkits anyways, so device compatibility is not an issue, I've tested on firefox, chrome, android, mobile browsers, and a few different romm-frontend android apps and never had an issue caused by VP9.

VP9 supports "SVC" - Scalable Video Coding, which VP8 and other codecs available to us for use in this project do not.

SVC is a way we can implement multiple streams of video for clients to automatically switch between based on network performance (or manually, via 'preferred' settings or forced settings, if you code in user options that way). You can create multiple spatial layers, or temporal layers, at your discretion. Example; L2T3 = 2 spatial layers, 3 temporal layers.

A spatial layer has pretty high overhead for the host and requires a good strong CPU, because it is rendering separate video streams at different resolutions - you have full control over which resolutions you want, and you can code this as a relative reference (for example, L1 is 'full resolution' and L2 is 'half resolution' based on whatever native resolution is set to in host's EmulatorJS). I don't actually use spatial layers because of the overhead on the CPU, I did not like the impact it had in my tests.

A temporal layer has very low CPU overhead however, and what temporal layers do is they stream multiple framerates for the same video feed. So it is much safer to implement with minimal impact on host performance, and it uses the same idea - clients can be automatically switching based on performance or optionally use a netplay setting to set a preferred-or forced-choice of temporal layers.

You can also use temporal layers as a sort of latency hack to reduce video latency and the perception of input delay for clients. I have a netplay options menu where I let the host select L1T1 or L1T2 only for their stream (60fps vs 120fps) and I set the bitrate really low, like 4 or 5 mbps.

What happens here when I do this, is the client when using 120fps is forcing their application to stream 2 duplicate frames from the 60fps source in each timing window. This can either help or not depending on network performance so I make it optional to the users - but when it works, it results in lower video latency because frame pacing is artificially doubled and so corrections to video latency happen twice as fast.

My work is on hiatus for a little bit while I focus on other things for now, and my project is going to be run on it's own dedicated romm frontend application to integrate with my relay server so it's not part of this work for many reasons, but I think these optimizations can greatly improve the netplay performance for the core EmulatorJS project with P2P netplay.

When I implemented the lower bitrates, and the VP9 even at 60fps only (L1T1), I basically took Super Mario World & Super Smash Bros 64 from unplayable to playable, even with my friend's client app being on the other side of Canada from me. All of these features I've described are independent of the netplay architecture, what I mean by that is, they work as expected with either P2P or relay server based networking, because all the logic and behavior is dictated by the applications themselves and the codecs, and not a feature of my relay server.

@S-Coding23 I have been contributing to romm a little bit, and I am building out my own EmulatorJS based Netplay fork I work on, with different goals in mind. I built a mediasoup relay server or "SFU" as an alternative to the P2P netplay framework, which requires hosting, but can offer a more feature rich netplay experience than P2P can, and supports 4P (or more) gameplay much better.

Throughout my own work for the past few months on this, I've come across some optimizations that might help you with the P2P netplay here. Yui suggested I could share my ideas here with you.

One idea, and maybe you've implemented this cause it's a lot of code to check through right now... Use separate channels for the video and the audio streams - this divorces the video timings from the audio timings, so that latency or microstutters in the audio do not force the video feed to frameskip or lockup. It can potentially lead to desyncing the audio but if the host has half decent hardware and your logic handles the audio stream well, that shouldn't be an issue. This results normally in much much smoother video streams for clients.

Second, is that I do see you are still using VP8. VP9 is actually much better for this application. It offers superior video compression, for faster streams, better quality, and reduced bitrates. All EmulatorJS users are running in browser webkits anyways, so device compatibility is not an issue, I've tested on firefox, chrome, android, mobile browsers, and a few different romm-frontend android apps and never had an issue caused by VP9.

VP9 supports "SVC" - Scalable Video Coding, which VP8 and other codecs available to us for use in this project do not.

SVC is a way we can implement multiple streams of video for clients to automatically switch between based on network performance (or manually, via 'preferred' settings or forced settings, if you code in user options that way). You can create multiple spatial layers, or temporal layers, at your discretion. Example; L2T3 = 2 spatial layers, 3 temporal layers.

A spatial layer has pretty high overhead for the host and requires a good strong CPU, because it is rendering separate video streams at different resolutions - you have full control over which resolutions you want, and you can code this as a relative reference (for example, L1 is 'full resolution' and L2 is 'half resolution' based on whatever native resolution is set to in host's EmulatorJS). I don't actually use spatial layers because of the overhead on the CPU, I did not like the impact it had in my tests.

A temporal layer has very low CPU overhead however, and what temporal layers do is they stream multiple framerates for the same video feed. So it is much safer to implement with minimal impact on host performance, and it uses the same idea - clients can be automatically switching based on performance or optionally use a netplay setting to set a preferred-or forced-choice of temporal layers.

You can also use temporal layers as a sort of latency hack to reduce video latency and the perception of input delay for clients. I have a netplay options menu where I let the host select L1T1 or L1T2 only for their stream (60fps vs 120fps) and I set the bitrate really low, like 4 or 5 mbps.

What happens here when I do this, is the client when using 120fps is forcing their application to stream 2 duplicate frames from the 60fps source in each timing window. This can either help or not depending on network performance so I make it optional to the users - but when it works, it results in lower video latency because frame pacing is artificially doubled and so corrections to video latency happen twice as fast.

My work is on hiatus for a little bit while I focus on other things for now, and my project is going to be run on it's own dedicated romm frontend application to integrate with my relay server so it's not part of this work for many reasons, but I think these optimizations can greatly improve the netplay performance for the core EmulatorJS project with P2P netplay.

When I implemented the lower bitrates, and the VP9 even at 60fps only (L1T1), I basically took Super Mario World & Super Smash Bros 64 from unplayable to playable, even with my friend's client app being on the other side of Canada from me. All of these features I've described are independent of the netplay architecture, what I mean by that is, they work as expected with either P2P or relay server based networking, because all the logic and behavior is dictated by the applications themselves and the codecs, and not a feature of my relay server.

Thank you so much for taking the time to share all this, I really appreciate you passing along what you've learned from your work. That sounds like a cool project and I'd love to check it out when you're ready.

I went through your notes and ended up making some changes to my code based on them, though I landed in a slightly different spot than you suggested on a few things.

On the codec stuff, I'm actually already preferring H.264 over VP8, so I'm not on VP8 anymore. I looked into switching to VP9 but for real-time emulator streaming I think H.264 is still the better call, mainly because hardware encoder support is way more consistent across devices. VP9 software encoding can actually add CPU load and latency on the host side, which is the opposite of what we want.

I dug into splitting video and audio into separate channels a bit more and I still believe in WebRTC they're already traveling as separate streams with their own timing, even when bundled together. So I don't think that one buys us as much as it sounds like it would, and it would add a lot of complexity for setup and reconnection.

The SVC/temporal layers idea, I took a conservative version of this. I added a temporal scalability hint (L1T2) so guests on weaker connections can gracefully drop to half framerate without the stream dying. Full spatial SVC I passed on since it mostly helps SFU setups like yours rather than P2P.

The frame pacing trick I didn't implement.I couldn't convince myself it actually reduces latency (duplicating frames doesn't make data arrive faster), but I suspect what was really helping you there was the low bitrate forcing the encoder into a tighter mode.

Which brings me to the thing that is your biggest insight for me, the bitrate tuning. I dropped my defaults across the board (relay path especially), switched the degradation preference to favor framerate over resolution, and set the content hint to motion instead of detail. I also added a playout delay hint of 0 on the receiver side, which should shave off a chunk of perceived input lag from the jitter buffer, and bumped the input data channel to high priority so inputs beat video under congestion.

Alot of what you suggested pointed me in good directions even where I didn't take the exact path, and the bitrate tip in particular was really valuable. Thanks again for sharing, and good luck with the rest of your project when you pick it back up!

I'm not sure H.264 can take advantage of hardware encoding because it is bound to the browser, and only use the CPU - that is one of the reasons I initially chose VP9 in the first place when researching my options. I could be wrong though, and I never tried to really push that or anything. I'll revisit it and test it out later. I still have a few months of extra work probably to do, as I have to build out a frontend application for my project now still, that works with the romm client API. And then refactor my AAA framework...

It doesn't make frames arrive faster no, you are correct, it's a hack, and only smooths out video latency in situations where there is already fast transmission. I'm wondering if I didn't hallucinate the latency improvements I saw when testing it now.

I also added a playout delay hint of 0 on the receiver side

That's useful to me to learn now thanks. I'll remember this when I go back to work on my application this summer.

Thanks I'm glad I was helpful, and you were also helpful to me here with your response. I think you've got this under control it sounds like, so good luck wrapping up the refactor.

Which brings me to the thing that is your biggest insight for me, the bitrate tuning. I dropped my defaults across the board (relay path especially), switched the degradation preference to favor framerate over resolution, and set the content hint to motion instead of detail. I also added a playout delay hint of 0 on the receiver side, which should shave off a chunk of perceived input lag from the jitter buffer, and bumped the input data channel to high priority so inputs beat video under congestion.

There's actually a lot in there I'm going to have to come back to this and double check what I've got, I could probably make some minor improvements also with these notes.

https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats/Video_codecs#avc_h.264

In HTML content for web browsers, AVC is broadly compatible and many platforms support hardware encoding and decoding of AVC media

https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats/WebRTC_codecs

Another factor to consider—especially on mobile platforms—is the impact a codec may have on battery life. If a codec is handled in hardware on a given platform, that codec is likely to allow for much better battery life and less heat production.
For example, Safari for iOS and iPadOS introduced WebRTC with AVC as the only supported video codec. AVC has the advantage, on iOS and iPadOS, of being able to be encoded and decoded in hardware. Safari 12.1 introduced support for VP8 within IRC, which improves interoperability, but at a cost—VP8 has no hardware support on iOS devices, so using it causes increased processor impact and reduced battery life.