add support for --try-update-deps, to upgrade dependencies based on available easyconfigs by ocaisa · Pull Request #2599 · easybuilders/easybuild-framework

ocaisa · 2018-09-29T14:20:40Z

An example of how this can be used:

eb --dry-run --robot --try-toolchain=foss,2018b --minimal-toolchains --try-update-deps Keras-2.1.3-intel-2017b-Python-3.6.3.eb

It currently does the less invasive update of the dependencies, i.e., if we upgrade software with version 1.2.3, it would chose 1.2.4 over 1.3.0 (if both existed in the target toolchain).

ocaisa · 2018-10-01T10:38:25Z

@boegel This is ready for a look and initial comments.

I already see a case where it won't work as intended, Java/1.8 with the new approach there, since it relies on LooseVersion to do the comparison. We may need to make this expand the logic there to deal with this.

ocaisa · 2018-10-01T11:03:29Z

This works but is quite aggressive

Not needed since this will be created if required

ocaisa · 2020-02-27T12:45:21Z

@boegel Ok, this can now also update software versions as well as dependencies even in the case where the software version appears in exts_list (like it might for a PythonBundle), so your (updated) command line

eb Python-3.7.4-GCCcore-8.3.0.eb --try-software-version 3.8.1 --try-toolchain-version 9.2.0 --try-update-deps --ignore-osdeps -D --experimental --disable-cleanup-tmpdir

returns

== temporary log file in case of crash /tmp/eb-RraSlk/easybuild-FFh1z5.log
Dry run: printing build status of easyconfigs and dependencies
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/m/M4/M4-1.4.18.eb (module: Core | M4/1.4.18)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/b/Bison/Bison-3.3.2.eb (module: Core | Bison/3.3.2)
 * [x] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/z/zlib/zlib-1.2.11.eb (module: Core | zlib/1.2.11)
 * [x] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/h/help2man/help2man-1.47.4.eb (module: Core | help2man/1.47.4)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/m/M4/M4-1.4.17.eb (module: Core | M4/1.4.17)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/b/Bison/Bison-3.0.4.eb (module: Core | Bison/3.0.4)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/f/flex/flex-2.6.4.eb (module: Core | flex/2.6.4)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/b/binutils/binutils-2.32.eb (module: Core | binutils/2.32)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/g/GCCcore/GCCcore-9.2.0.eb (module: Core | GCCcore/9.2.0)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/h/help2man/help2man-1.47.10-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | help2man/1.47.10)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/m/M4/M4-1.4.18-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | M4/1.4.18)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/z/zlib/zlib-1.2.11-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | zlib/1.2.11)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/b/Bison/Bison-3.3.2-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | Bison/3.3.2)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/f/flex/flex-2.6.4-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | flex/2.6.4)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/b/binutils/binutils-2.32-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | binutils/2.32)
 * [ ] /tmp/eb-RraSlk/tweaked_dep_easyconfigs/bzip2-1.0.8-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | bzip2/1.0.8)
 * [ ] /tmp/eb-RraSlk/tweaked_dep_easyconfigs/ncurses-6.1-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | ncurses/6.1)
 * [ ] /tmp/eb-RraSlk/tweaked_dep_easyconfigs/Tcl-8.6.9-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | Tcl/8.6.9)
 * [ ] /tmp/eb-RraSlk/tweaked_dep_easyconfigs/libreadline-8.0-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | libreadline/8.0)
 * [ ] /tmp/eb-RraSlk/tweaked_dep_easyconfigs/SQLite-3.29.0-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | SQLite/3.29.0)
 * [ ] /tmp/eb-RraSlk/tweaked_dep_easyconfigs/libffi-3.2.1-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | libffi/3.2.1)
 * [x] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/n/ncurses/ncurses-6.1.eb (module: Core | ncurses/6.1)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/g/gettext/gettext-0.20.1.eb (module: Core | gettext/0.20.1)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/x/XZ/XZ-5.2.4-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | XZ/5.2.4)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/a/Autoconf/Autoconf-2.69-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | Autoconf/2.69)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/a/Automake/Automake-1.16.1-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | Automake/1.16.1)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/l/libtool/libtool-2.4.6-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | libtool/2.4.6)
 * [ ] /home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs/a/Autotools/Autotools-20180311-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | Autotools/20180311)
 * [ ] /tmp/eb-RraSlk/tweaked_dep_easyconfigs/GMP-6.1.2-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | GMP/6.1.2)
 * [ ] /tmp/eb-RraSlk/tweaked_easyconfigs/Python-3.8.1-GCCcore-9.2.0.eb (module: Compiler/GCCcore/9.2.0 | Python/3.8.1)
== Keeping temporary log file(s) /tmp/eb-RraSlk/easybuild-FFh1z5.log* and directory /tmp/eb-RraSlk.

ocaisa · 2020-03-04T16:26:36Z

@Micket made a valid comment that highlights the naivity of the software version upgrading in this approach. The problem is that the %(version)s template could be used anywhere in the easyconfig, but since this approach only works on the fully parsed easyconfig, that templating is lost...which could be really bad in the case where %(version)s is used in source_urls (for example).

To do this properly I would need to know the places where %(version)s appears, and update these accordingly. Another problem would be that for patches you can't do this (since the patch wouldn't exist).

Micket · 2020-03-04T17:37:06Z

@ocaisa What about all the other fields, like %(pyver)s; does it all get expanded in the generated easyconfigs with the current approach?

ocaisa · 2020-03-05T14:01:17Z

@Micket I just tested that with

eb Boost-1.55.0-foss-2016a-Python-2.7.11.eb --experimental --try-software-version 3.8.1 --try-toolchain-version 2018b --try-update-deps --ignore-osdeps -D --disable-cleanup-tmpdir

and the pyver template survived.

For "other fields", I don't know, I think we just have to try it out. I also checked what happens when the templates like %(version)s are used in used source_urls and this also survives:

eb Perl-5.28.1-GCCcore-8.2.0.eb --experimental --try-software-version 6.30.2 --try-toolchain-version 9.2.0 --try-update-deps --ignore-osdeps -D --disable-cleanup-tmpdir

So, as far as I can tell, there are still no issues to fix in this PR.

boegel · 2020-03-08T13:27:51Z

@Micket made a valid comment that highlights the naivity of the software version upgrading in this approach. The problem is that the %(version)s template could be used anywhere in the easyconfig, but since this approach only works on the fully parsed easyconfig, that templating is lost...which could be really bad in the case where %(version)s is used in source_urls (for example).

To do this properly I would need to know the places where %(version)s appears, and update these accordingly. Another problem would be that for patches you can't do this (since the patch wouldn't exist).

This goes way beyond the %(version)s template too. If there's a list comprehension in sanity_check_paths for example, it will get expanded in the resulting easyconfig, which is not what you want either.

The only way around that is to work on the raw easyconfig.

I'm not considering that a blocker for this PR though, since that problem already exists with the --try-* options I think, but it's something to keep in mind when we start working on the eb --tweak-ec (or eb --change-ec idea I mentioned during the user meeting).

The main difference with --try-* would be that it only makes minimal changes on the raw easyconfig contents, and then dumps the tweaked (raw) easyconfig file (rather than starting to build it), perhaps with an additional option to opt-in to also building the resulting easyconfigs.

We can probably learn a great deal from the current support we have for --try-*, but the current implementation is complicated, and I think we can do a better job starting afresh.

With that being said, I'll take another detailed look at this, and go ahead on merge it unless I run into any blockers (hopefully not).

ocaisa · 2020-03-09T16:58:20Z

@boegel I should say that it doesn't actually seem to be the case that the templating is lost, there is a little bit of voodoo in here that I don't fully understand (see #2599 (comment)) and I think that might be why things are more robust than one might expect.

ocaisa · 2020-03-11T14:02:04Z

@boegel I tested this with

eb fastq-tools-0.8-foss-2018b.eb --experimental --try-software-version 6.30.2 --try-toolchain-version 2019b --try-update-deps --ignore-osdeps -D --disable-cleanup-tmpdir

and indeed the list comprehensions are being expanded, as well as the subtoolchains for dependencies being explicitly inserted. Functionally this is no different but of course we can perhaps do better in the dump() method in maintaining the original formatting.

ocaisa · 2020-03-31T11:43:09Z

#3261 solves the problem of the dumped easyconfig not being "toolchain hierarchy aware", for list comprehensions this is not really relevant to this PR, it is more an issue with the dump method in general.

…er_extra_paths option

…target_tc_hierarchy in docstring

…te docstring for get_matching_easyconfig_candidates

… to test

boegel · 2020-04-11T11:37:54Z

@ocaisa I've done a thorough last review of this PR this morning, and just pushed in a couple of minor changes, mostly:

enhanced test for search_easyconfigs + minor bug fix
simple end-to-end test for --try-upgrade-deps
fixed broken test_index_functions (due to 2 additional easyconfigs)
minor changes in comments & docstrings

With this, I consider this PR good to go, I'll merge it as soon as the tests go green...

Thanks a lot for all your efforts on this, this clearly took a lot of effort, and was dragged along for way too long, I'm happy we can finally get this merged...

Add ability to upgrade dependencies based on available easyconfigs

882b3ec

houndci-bot reviewed Sep 29, 2018

View reviewed changes

Comment thread easybuild/framework/easyconfig/tweak.py Outdated

Comment thread easybuild/framework/easyconfig/tweak.py Outdated

MAke hound happy

8509d77

ocaisa commented Sep 29, 2018

View reviewed changes

Comment thread easybuild/framework/easyconfig/tweak.py Outdated

Alan O'Cais added 7 commits September 29, 2018 17:08

Fix typo

e56671b

Change to use search_easyconfigs

ca8911c

Revert unneeded modifications

2fdbf51

make the option name more explicit

5f7c297

Should have refactored!

a913bd3

Start writing a test

cd4dd1b

Correct test

8281e58

houndci-bot reviewed Oct 1, 2018

View reviewed changes

Comment thread easybuild/framework/easyconfig/tweak.py Outdated

Alan O'Cais added 4 commits October 1, 2018 10:52

Appease hound

b51f87d

Add proper docstring for search_easyconfig

75fef2a

Add test for dependency upgrading

b902f83

Add build option for dependency upgrading

601010c

ocaisa changed the title ~~WIP: Add ability to upgrade dependencies based on available easyconfigs~~ Add ability to upgrade dependencies based on available easyconfigs Oct 1, 2018

Alan O'Cais added 2 commits October 1, 2018 12:40

Add new files to test dependency upgrades

ca190f2

Correctly add build option

dbd450d

Alan O'Cais added 2 commits October 1, 2018 13:10

Correct logic for build option

366989d

Shorten opt name

a457179

houndci-bot reviewed Oct 1, 2018

View reviewed changes

Comment thread easybuild/tools/options.py Outdated

Alan O'Cais added 3 commits October 1, 2018 13:16

Correct omissions

5ea9fa5

Appease the hound

2b3aa87

Delete gzip-1.4-iccifort-2016.1.150-GCC-4.9.3-2.25.eb

03dfbb5

Not needed since this will be created if required

ocaisa commented Oct 1, 2018

View reviewed changes

Comment thread easybuild/framework/easyconfig/tweak.py Outdated

Alan O'Cais added 2 commits October 1, 2018 17:10

Correct stuff related to versionprefix

8c87d39

Correct stuff related to versionprefix

8fbb927

houndci-bot reviewed Feb 27, 2020

View reviewed changes

Comment thread test/framework/tweak.py Outdated

Appease hound

d816bb8

Alan O'Cais added 2 commits February 27, 2020 13:53

Use proper looking fake checksum value

b3c536d

Use correct checksum value

3585c22

ocaisa mentioned this pull request Mar 17, 2020

Dump method aggressive in use of pyver #3250

Closed

ocaisa mentioned this pull request Mar 31, 2020

make EasyConfig.dump aware of toolchain hierarchy, to avoid hardcoded subtoolchains in dependencies easyconfig parameters #3261

Merged

boegel added 11 commits April 11, 2020 10:40

also honor filename_only if print_only is disabled in search_easyconfigs

b661ffd

enhance test_search_easyconfigs to also cover print_result option

8ee7738

enhance test_search_easyconfigs to also covered having enabled consid…

89a2b9e

…er_extra_paths option

minor code cleanup in test_recursive_try

fd953e6

also document update_build_specs named argument of map_easyconfig_to_…

d65d12d

…target_tc_hierarchy in docstring

change log.info to log.warning in map_common_versionsuffixes + comple…

10f834d

…te docstring for get_matching_easyconfig_candidates

tweak comments in find_potential_version_mappings + minor enhancement…

9f460a3

… to test

minor comments cleanup in map_easyconfig_to_target_tc_hierarchy

88be361

Merge branch 'develop' into allow_dep_upgrades

7fe9cd5

add (simple) end-to-end test for --try-update-deps

a05f962

take into account additional test easyconfigs in test_index_functions

ee63271

boegel approved these changes Apr 11, 2020

View reviewed changes

boegel merged commit faa3e2a into easybuilders:develop Apr 11, 2020

ocaisa deleted the allow_dep_upgrades branch April 11, 2020 12:20

boegel changed the title ~~Add ability to upgrade dependencies based on available easyconfigs~~ add support for --try-update-deps, to upgrade dependencies based on available easyconfigs Apr 11, 2020

Conversation

ocaisa commented Sep 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ocaisa commented Oct 1, 2018

Uh oh!

ocaisa commented Oct 1, 2018

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ocaisa commented Feb 27, 2020

Uh oh!

ocaisa commented Mar 4, 2020

Uh oh!

Micket commented Mar 4, 2020

Uh oh!

ocaisa commented Mar 5, 2020

Uh oh!

boegel commented Mar 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ocaisa commented Mar 9, 2020

Uh oh!

ocaisa commented Mar 11, 2020

Uh oh!

ocaisa commented Mar 31, 2020

Uh oh!

boegel commented Apr 11, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ocaisa commented Sep 29, 2018 •

edited

Loading

boegel commented Mar 8, 2020 •

edited

Loading