more generally, some semantics are needed to drive build constraints, such as:

• avoid building 2 intel compilers concurrently (some license file issue is going on under /tmp)
• build ATLAS on dedicated node only (otherwise its heuristics are distorted)
• avoid building NCBI-Toolkit & BLAST on the same node concurrently (don't recall what, something under /tmp)
  ... there are other issues of same family, a generic build campaign across many sites will make them show up.

EasyBuild never hardcodes /tmp, but simply uses the Python module tempfile, and the functions it provides, e.g., tempfile.mkdtemp, tempfile.mkstemp, ...
By default, those functions returns a (unique) temporary subdir in /tmp.

Do we need to provide a wrapper function in the EasyBuild framework that checks whether the path obtained can be used, and try other options if it detects that the obtained path may be troublesome?
If so, what should the other options be?

A wrapper is not needed. tempfile looks for a tmp directory in the following env vars: $TMPDIR, $TEMP, $TMP. I suggest the following:

• Call tempfile.gettempdir() to find the current tmp directory.
• Check the mount options on the tmp directory.
  • If noexec is present, use $EASYBUILDPREFIX/tmp (set with tempfile.tempdir and set those env vars for any other program)
  • if noexec is not present, continue as usual

I would not use /var/tmp as an alternative because that is for persistent temp storage. I think $EASYBUILDPREFIX/tmp is the next best thing.

This should be done in the main() of the program, I think? I shall have a look at it when I find the time.

We can also make it a config variable? So People can choose them self.

Just for the record, the noexec policy in /tmp some people use, stems from experience with botnets like this:
http://bsdly.blogspot.ca/2013/10/the-hail-mary-cloud-and-lessons-learned.html
(you need not convince us that being lean and clean on sshd policies is the higher quality approach)

Back on the subject now,
I'd prefer that EASYBUILDBUILDPATH remains the authoritative definition about where to run the builds,
because fi. in our case /tmp corresponds to the local SSD disk, which makes a lot of sense to use w. EB.
I would dislike somebody to take the carpet under my feet, with such automation around mktemp();
I'd rather prefer to receive a warning about the particular package, so that I have a chance to intervene...

So, the config variable appeals to me as the most community-friendly approach.

Your takes?

I agree, a config variable seems a nice way of doing it. We just have to set $TMPDIR, $TEMP, $TMP to that location and everything should work as before.

I agree that a specific config variable should do the trick. Something like eb --tmpdir, and thus $EASYBUILD_TMPDIR. If it is set, then set $TMPDIR to that, and EasyBuild should pick it up (we take care not to hardcode /tmp anywhere).

Who's up for implementing that?
Just find a sensible location in easybuild/tools/options.py to add it, and then check it and set $TMPDIR somewhere in easybuild/main.py (fairly early, I guess, right behind where parse_options is called)

I will have a look at it.

Automatic reply from Jenkins: Can I test this?

I have added a --tmpdir / EASYBUILD_TMPDIR option that sets the path that the tempfile python module uses. I set the TMP, TMPDIR and TEMP env vars. Most program uses one of those vars to find a location for temp storage.

In the second commits, I've added a check to see if we can execute files in the temp storage. I've put it in the main but maybe it's better to split it off in a seperate function?

looks good to me
Jenkins: test this please

Please review again?

Jenkins: ok to test

Thanks a lot for initiating this and working together on the fiddly bits @wpoely86, merging it in!