Skip to content

list build dependencies *before* runtime dependencies, so dependencies listed via multi_deps are loaded first in toolchain environment#2839

Merged
bartoldeman merged 2 commits intoeasybuilders:developfrom
boegel:build_deps_first
Apr 24, 2019
Merged

list build dependencies *before* runtime dependencies, so dependencies listed via multi_deps are loaded first in toolchain environment#2839
bartoldeman merged 2 commits intoeasybuilders:developfrom
boegel:build_deps_first

Conversation

@boegel
Copy link
Copy Markdown
Member

@boegel boegel commented Apr 24, 2019

While working on an easyconfig file that required SciPy-bundle as a dependency and where multiple Python versions were specified via multi_deps, I noticed a problem we have overlooked in #2828.

When Python is included in multi_deps, the installation procedure is performed iteratively, once for every listed Python version. Internally, this is done via builddependencies (to ensure no (strict) runtime dependency on a particular Python version is introduced in the generated module file).

If this is combined with a dependency like SciPy-bundle by which a Python version gets loaded by default if none is loaded yet (since multi_deps_load_default was enabled when installing SciPy-bundle), a version conflict arises when the SciPy-bundle module (which in turns loads the module for Python 3.7.2 (for example)) was loaded first, and a(nother) Python version was loaded afterwards...

For example, with:

toolchain = {'name': 'foss', 'version': '2019a'}

multi_deps = {'Python': ['3.7.2', '2.7.15']}

dependencies = [('SciPy-bundle', '2019.03')]

the problem occurs when the Python/2.7.15-GCCcore-8.2.0 module is being loaded for the 2nd iteration of the installation, since at that point the SciPy-bundle/2019.03-foss-2019a module was already loaded, which in turns resulted in loading Python/3.7.2-GCCcore-8.2.0 by default since no Python module was loaded yet at this point...

The problem is easily fixed by simply switching the order of build dependencies vs (runtime) dependencies to ensure that build dependencies are loaded first (after the toolchain is loaded)...

In practice this most likely rarely matters in contexts where multi_deps is not involved, so this change should be safe to make.

We ran into a similar issue with the sanity check step while implementing the support for multi_deps_load_default in #2828, see https://github.com/easybuilders/easybuild-framework/pull/2828/files#diff-a1bcbbc1aab6b59688b219f298fddb56R1344

Copy link
Copy Markdown
Contributor

@bartoldeman bartoldeman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@bartoldeman bartoldeman merged commit ff6b0aa into easybuilders:develop Apr 24, 2019
@boegel boegel deleted the build_deps_first branch April 25, 2019 06:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants