This could probably land for 10.0.3.0 but we need it green. @matthias-fratz-bsz Could you rebase on current master and re-push to see how it shakes out?

So... rebasing doesn't seem to have helped. There's still a massive number of failures (15 failing, 55 successful, and 3 that still haven't finished after an hour).

The theme with these failures seems to be:

27)
IO#wait [events, timeout] passed raises ArgumentError when events is not positive ERROR
Expected ArgumentError (Events must be positive integer!)
but got: ArgumentError (unsupported mode: Integer)

(...plus a lot of other failures that I don't understand.)

To me this looks like I've been trying to merge into the wrong branch. I was trying to merge into 10-dev, but jruby-10.0.0 looks much more recent. Unfortunately, I realized that the fix is already in master only after I was done rebasing and pushing.

@headius Should I change the target branch to jruby-10.0.0 and push a version rebased onto that? Or should I just close the pull request? It looks mostly obsolete to me now that the 9.4 variant of the fix is in both master and 10.1-dev.

@matthias-fratz-bsz @NC-piercej This is not going to make 10.0.3.0 but I want to work with the two of you to identify and fix all embedding leaks for 10.0.4.0. Perhaps you could stop by the JRuby Matrix chat and we can figure out a plan?

https://www.jruby.org/chat

Seems like 10.0.0.3 is imminent, which should include the non-cleaner upstream change that hopefully resolves the thread memory leak. Once released, we'll try to update ASAP and monitor our memory usage in ECS. For reference, here is our existing memory chart:

The rapid drop is a deploy, after which memory usage slowly creeps up. We are using JRuby in server mode with -J-Xmx6594m (75% of 8GB), so I think it's expected that the JVM will be reticent to free back up RAM to the OS. However, we do see tasks eventually become unhealthly and cycle, so we think we are affected by a (quite slow) memory leak somewhere. We (and Claude) haven't been able to identify any memory leaks in our actual Ruby code, and our code does do ~8 threads per request, so we think the now-hopefully-fixed thread memory leak is the culprit.

I doubt we'll be approved to actually deploy a non-release version of JRuby, however.

An update: we've had 10.0.3.0 / Java 25 deployed for a while now, and we are no longer seeing tasks recycle for memory issues. So, there is a very good chance that the underlying issue may have been fixed.

@NC-piercej Ok thank you for that update. Please let us know if you see anything else that looks like it may be leaking memory.

@matthias-fratz-bsz I realized I missed a question from you above:

To me this looks like I've been trying to merge into the wrong branch. I was trying to merge into 10-dev, but jruby-10.0.0 looks much more recent.

Sorry, we don't have very good documentation about what lives where...

You want to use master right now, which always represents the current release line (10.0 in this case). When we start preparing to release 10.1, the 10.1-dev branch will merge to master and the 10.0 maintenance will move to a jruby-10.0 branch.

10-dev is the branch we used while doing development of JRuby 10.0.0.0. It should have been deleted once we merged to master. Same will go for 10.1-dev once that merges.

If you can try to rebase on top of master I think these issues will clear up. I'll wait to wipe out the 10-dev branch because I think doing so right now will permanently close this PR.

@matthias-fratz-bsz I attempted to do a rebase myself but it seems like the commit disappears. I suspect because we have the Cleaner commit followed by another that removes it (for Java 8 etc) git believes it has already been applied.

So I think the right move would be for me to push a revert of the Cleaner removal and then we'll be back to the Cleaner version. I'll give that a try and push a replacement PR if that appears to work right.

@matthias-fratz-bsz I have pushed #9260 with the revert of the de-Cleaner-ification commit, so 10.0.4.0 forward will use the Cleaner API again. Functionality-wise I don't believe there's any other changes.

If you and @NC-piercej have any input there let me know, but otherwise I will merge once it runs green.

I'll close this and clean up the confusing branches.