lgtm, let's wait for confirmation from @mstud?

As already mentioned in #2182, it is unfortunately not sufficient to fix the issue. While using --job does not fail anymore, it now ignores optarch completely inside of a job, because the same mechanism that lead to the dict getting converted to a string before the second pass apparently now leads to the string getting converted to a string encapsulated in single quotes, which messes with the parsing.
If I also change
optarch_parts = self.options.optarch.split(OPTARCH_SEP)
to
optarch_parts = self.options.optarch[1:-1].split(OPTARCH_SEP)
stripping off the quotes, it works just fine, but this seems a bit like a hack.

Argh. @boegel why/where is it that happening? Shouldn't the options be keep verbatim, if they are going to be "reprocessed"?

@damianam https://github.com/hpcugent/easybuild-framework/blob/master/easybuild/tools/parallelbuild.py#L136

It quotes the arguments to make sure the shell doesn't trip over them, but it seems like something is going wrong in this case, eb should never actually see the quotes...

@mstud Can you look up the log line that says Command template for jobs in the log where you use --job? You can collect it either via eb -ld or using --disable-cleanup-tmpdir (see log file mentioned at very beginning).

In [8]: subprocess.list2cmdline(['mystuff', 'more stuff', 'something is off', "--optarch='something:is weird,here:I believe'"]).replace('%', '%%')
Out[8]: 'mystuff "more stuff" "something is off" "--optarch=\'something:is weird,here:I believe\'"'

It seems like list2cmdline doesn't do what you expected it to do. I guess the same issue would appear in any option that accepts spaces.

@damianam We could probably fix this by using the (monkey-patched) shell_quote, see tools/options.py

I guess it's not relevant anymore after damianam's discovery, but in the Command template for jobs line the parameter shows up as:
eb [...] "--optarch='Intel:xSSE4.2 -axCORE-AVX2;GCC:march=westmere -mtune=haswell'" [...]

@boegel that will result in a token like this: '--optarch=\'Intel:xSSE4.2 -axCORE-AVX2;GCC:march=westmere -mtune=haswell\''. That's precisely what we don't want, right?

Looks to me like we need to write a method to append options to a string, and wrap them around in quotes just if there are spaces outside of any quoted area. Something like this maybe?

import re

tokens=['software.eb', '-r path', "--optarch='Intel:xSSE4.2 -axCORE-AVX2;GCC:march=westmere -mtune=haswell'"]

blank_checker = re.compile(r".*('.*\s.*')|(\".*\s.*\").*")

tokens_str=""

for token in tokens:
    if " " in token:
        # Wrap in double quotes
        if not blank_checker.match(token) and "'" in token:
            tokens_str+=' '+'"'+token+'"'
        # Wrap in single quotes
        elif not blank_checker.match(token) and '"' in token:
            tokens_str+=" "+"'"+token+"'"
        # Wrap in single quotes
        elif not blank_checker.match(token):
            tokens_str+=" "+"'"+token+"'"
        else:
            tokens_str+=" "+token
    else:
        tokens_str+=" "+token

print(tokens_str)

It is ugly, I know, I don't like it either.......

The problem is in generate_cmd_line, not in the use of subprocess.list2cmdline.

Here's what subprocess.list2cmdline spits out for options that include --optarch="GCC:O3 -mtune=generic;Intel:O3 -xHost":

[..., '--optarch=\'"GCC:O3 -mtune=generic;Intel:O3 -xHost"\'', ...]

So, we'll need to fix generate_cmd_line, probably via eb_shell_quote in options.py, I'm looking into it.

Options values that include spaces are quite rare, so this hasn't popped up before.

@damianam problems should be fixed with additional changes in damianam#2, confirmed by enhanced tests

@mstud Please verify whether the additional changes fix your problem?

@boegel Yes, it works properly now both with and without --job. Thank you very much.

Good to go to be included in EasyBuild v3.2.0, thanks for the (partial ;)) fix @damianam and the feedback @mstud!