fix logic in make_module_dep w.r.t. excluding loads for toolchain & toolchain components#2140
Conversation
|
This works like a charm :-) |
|
lgtm |
|
This certainly a more elegant solution! lgtm2 |
…in make_module_extra that was defined in configure_step
|
Thanks for the feedback everyone, going in to be included in EasyBuild v3.1.1. |
|
I did not check properly... This fix is good for the compiler-only toolchain case where the toolchain used equals the MODULEPATH extender (in fact my hook for ec in expand_toolchain_load(ec=self.cfg) was another way to fix this because our module naming scheme uses that to check for compiler-only toolchains, but your fix is much better than this). However the original problem fixed in #2091 is still there. |
|
How does that work: HPL using intel-2016b expands the dependencies from intel-2016b.eb: The modules icc, ifort, and impi are eliminated since they expand $MODULEPATH. |
|
@bartoldeman Hmm, ok, I indeed overlooked that... And since I also changed to test case, the tests didn't complain either. Can you open a new PR to current Do also include test case from #2091 in the existing Do make sure that the original test from #2091 highlights the reintroduced bug; if not, we need a better test case. |
|
I am a little worried about reintroducing #2091 because that also means adding #2135 again. since However if I rewrite deps in intel-2016b.eb as follows: then I trigger an exception in I think I will try to experiment with increasing depth in the call in toolchain.py since that way the icc and ifort dependencies are found (because they are deps of iccifort). |
No, it doesn't, #2135 was a workaround for modules that include So, as long as your modules have been regenerating with the fix included in #2091, you don't need to avoid reloading modules anymore like we do in #2135 (other than maybe to optimise things, but you'll gain very little from it). |
|
fix was (re)implemented by @bartoldeman in #2152 |
This provides a proper fix for the problem that #2135 dances around, and basically undoes @bartoldeman's PR #2091 since it implements a better solution that also does the right thing when
GCC(i.e. the bundle ofGCCcoreandbinutils) is used as a toolchain.Thanks to @rtmclay for helping out with figuring out the actual problem by looking at the Lmod debug output provided by @akesandgren where reloading the
OpenMPImodule failed because thebinutilsmodule couldn't be found anymore.The problem was that
loadstatements were still being inserted forGCCcoreandbinutilswhen something was being built with theGCCtoolchain (e.g.OpenMPI) when a module naming scheme that enablesexpand_toolchain_loadwas being used (for exampleHierarchicalMNS).The existing code that excludes dependencies that extend
$MODULEPATHfailed becauseGCC(i.e. the toolchain being used) was expanded intoGCCcore+binutilsbefore the list of dependencies that extend$MODULEPATHare determined viapath_to_top_of_module_tree, and henceexcluded_depswas an empty list while it should have includedGCC.This is only a problem when the
GCCtoolchain was being used, since bothGCCand itsGCCcorecomponent extend$MODULEPATH.By now passing both the toolchain (e.g.
GCC) and the toolchain components (GCCcore+binutils) topath_to_top_of_module_tree, theGCCtoolchain itself is correctly excluded as well (and hence so are the toolchain components).The additional loop that excludes dependencies of dependencies that was added in #2091 by @bartoldeman is no longer needed now.
cc @geimer, @ocaisa, @damianam