We had a somewhat complicated case where a .so as used by Python was having an rpath where the icc path is listed before the imkl path. The .so is then linked against (among others):

$EBROOTICC/lib/intel64/libiomp5.so
$EBROOTIMKL/mkl/lib/intel64/libmkl_rt.so

but libmkl_rt.so at runtime also tries to load libiomp5.so, probably via dlopen and friends. But instead it tries to load $EBROOTIMKL/lib/intel64/libiomp5.so which then results in this error:

OMP: Error #15: Initializing libiomp5.so, but found libiomp5.so already initialized.
OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.
Aborted

When we compiled the software manually without easybuild this problem did not occur, because in LIBRARY_PATH the imkl path has precedence, but in EB LDFLAGS, the icc path has precedence (note that we are not using LD_LIBRARY_PATH!!). In order to match LIBRARY_PATH's order the LDFLAGS need to be reversed wholesale (and also #2930 is necessary to add the imkl path to LDFLAGS)

I thought about simply converting LIBRARY_PATH (as obtained via os.getenv after loading modules) to LDFLAGS. But that then disregards the 32bit option flexibility. On the other hand 32bit looks pretty bit-rotted to me, so perhaps 32bit support could be removed for EB 4.0 and then LDFLAGS could be constructed from LIBRARY_PATH @boegel ?

@bartoldeman I'm not sure what impact this change may have elsewhere, so I'm conservatively tagging this for 4.0 rather than for the next 3.9.x release.

Getting rid of the 32-bit stuff for EasyBuild v4.0 makes sense to me.

Defining $LDFLAGS based on $LIBRARY_PATH could make sense I guess...
But also there we'd need to be careful to not break anything?

Doing so for EasyBuild v4.0 is OK though, that gives us some credit for making "breaking" changes (and then dealing with the fallout afterwards).

Will close and reopen this since develop and 4.0 are now one and the same

Once the tests pass I'll merge