Currently JRuby seems to have at least 3 major modes to run:

• jruby myscript.rb, the default.
• jruby --dev ... for improving startup, at the cost of degrading peak performance.
• jruby -Xcompile.invokedynamic=true ... which uses invokedynamic.

That's I think more modes than any other Ruby implementation.
MRI has default and --jit, TruffleRuby has default and --jvm (in GraalVM, otherwise only the default mode).

When I (or someone else) benchmark JRuby, let's say with a benchmark-ips benchmark so startup is not the biggest concern, I would like to only run JRuby in just one mode and not have to guess the right one or run in different modes and need to compare.
So which mode should I choose?

I think the default mode should be the one producing good benchmark results.
Otherwise, any numbers reported should specify the alternative mode used and also risks being very far from what people use in production.
-Xcompile.invokedynamic=true is so long and so hard to find that I guess very few JRuby users use it in production, it looks more like a performance debugging option to me (much like the other 23 -X options with invokedynamic in the name).

In fact, I think many people don't even know -Xcompile.invokedynamic=true, which seems a pretty obscure option, by being not mentioned in jruby -h, jruby --help nor jruby -X.
For comparison, --jit is listed by ruby -h/--help and --jvm is listed by truffleruby --help.
The only way I found to list this option is jruby '-X?' | grep invoke, so I expect very few will manage to even find the option or even know about it.

One example to illustrate what I'm saying is https://twitter.com/headius/status/1230915989786648576:

JRuby's now pushing close to 100FPS for me locally on the Ruby "optcarrot" benchmark, with CRuby around 35FPS

That I think is misleading, jruby bin/optcarrot ... is about 27 FPS.
jruby -Xcompile.invokedynamic=true bin/optcarrot ... is about 86 FPS, and about the same as ruby --jit.
The reason I think it's misleading is:

• People can't reproduce these numbers without guessing and searching hard for that flag
• People are unlikely to experience such performance when running jruby ....

I think there are two ways forward:

• Make -Xcompile.invokedynamic=true the default. Then it's actually true jruby ... has that performance, and everyone experiences it.
• Have an easier-to-find, better documented flag like --indy, and mention it whenever benchmark results depend on it.

I think the second is not nearly as good as the first one, because it will again mean people will most likely observe fairly different performance in their production applications running on JRuby with default flags. But it'd be an improvement over the current state.