also install `easybuild` package for `EasyBuild >= 5` by branfosj · Pull Request #3700 · easybuilders/easybuild-easyblocks

branfosj · 2025-04-17T12:08:47Z

(created using eb --new-pr)

Micket

lgtm

boegel · 2025-04-23T07:42:54Z

@branfosj Can you share some more information on this, explain why this is required?

jfgrimm · 2025-05-21T12:40:38Z

@branfosj Can you share some more information on this, explain why this is required?

I think this was discussed in a conf call, but for clarity this is useful for e.g. easybuilders/easybuild-easyconfigs#22777

branfosj · 2025-05-21T13:01:01Z

@branfosj Can you share some more information on this, explain why this is required?

I think this was discussed in a conf call, but for clarity this is useful for e.g. easybuilders/easybuild-easyconfigs#22777

Some part of the tools developed by @Micket, and his group, require that the easybuild package is available (in the pip sense). Currently we only have easybuild-framework, easybuild-easyblocks, and easybuild-easyconfigs available.

I have a concern that we will hit an issue that easybuilders/easybuild-easyconfigs#22774 will break without the changes in this PR, meaning that the EB install latest command line option will not work. I am not sure how to check that though. This may mean that an alternative solution (so that easybuild appears in pip list) would be better than the changes here.

jfgrimm · 2025-05-21T13:27:27Z

I have a concern that we will hit an issue that easybuilders/easybuild-easyconfigs#22774 will break without the changes in this PR, meaning that the EB install latest command line option will not work. I am not sure how to check that though. This may mean that an alternative solution (so that easybuild appears in pip list) would be better than the changes here.

ah, I think I see what you mean. That's not really something we want to break :/
I think creating even an empty lib/python{ver}/easybuild-5.0.0-py{ver}.egg-info/PKG-INFO should be sufficient for it to show up in pip list?

branfosj · 2025-05-21T14:01:09Z

I have a concern that we will hit an issue that easybuilders/easybuild-easyconfigs#22774 will break without the changes in this PR, meaning that the EB install latest command line option will not work. I am not sure how to check that though. This may mean that an alternative solution (so that easybuild appears in pip list) would be better than the changes here.

ah, I think I see what you mean. That's not really something we want to break :/ I think creating even an empty lib/python{ver}/easybuild-5.0.0-py{ver}.egg-info/PKG-INFO should be sufficient for it to show up in pip list?

See #3725

concerns over impact on --install-latest-eb-release

Micket · 2025-05-23T08:26:48Z

Solved in #3725 instead

Co-authored-by: Jasper Grimm <[email protected]>

jfgrimm · 2025-05-23T10:47:56Z

LGTM

boegel · 2025-05-23T10:52:45Z

Don't merge this yet please, I'm concerned this may cause trouble, for example with exsiting EasyBuild-5.*eb easyconfigs

jfgrimm · 2025-05-23T10:57:43Z

@boegel was planning to wait until submitting a test report anyway

jfgrimm · 2025-05-23T11:00:02Z

Test report by @jfgrimm

Overview of tested easyconfigs (in order)

SUCCESS EasyBuild-5.0.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node034.viking2.yor.alces.network - Linux Rocky Linux 8.9, x86_64, AMD EPYC 7643 48-Core Processor, Python 3.6.8
See https://gist.github.com/jfgrimm/6b8f98811201017b551511f27aee74a1 for a full test report.

edit: correctly builds as normal, no easybuild package since it's not in sources

jfgrimm · 2025-05-23T11:08:53Z

successful report using the EasyBuild-5.0.0.eb easyconfig (with easybuild added to the sources) in: easybuilders/easybuild-easyconfigs#22774 (comment)

It correctly installed the easybuild package

boegel · 2025-05-23T11:12:04Z

successful report using the EasyBuild-5.0.0.eb easyconfig (with easybuild added to the sources) in: easybuilders/easybuild-easyconfigs#22774 (comment)

It correctly installed the easybuild package

OK, but what if the easybuild package is not included in the sources, then it'll go down hard?

branfosj · 2025-05-23T11:22:21Z

successful report using the EasyBuild-5.0.0.eb easyconfig (with easybuild added to the sources) in: easybuilders/easybuild-easyconfigs#22774 (comment)
It correctly installed the easybuild package

OK, but what if the easybuild package is not included in the sources, then it'll go down hard?

That works fine. This PR is required if the easybuild packages is in the sources. If it does not appear then there is no change in behaviour.

@jfgrimm test report #3700 (comment) is that case. And I'll have one appear here soon for that case as well.

jfgrimm · 2025-05-23T11:34:23Z

the only edge case I can think of is that with #3725 merged, we might be overwriting the egg-info file in the case where easbuild is in the sources

…ildmeta

branfosj · 2025-05-23T11:36:01Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS EasyBuild-5.0.0.eb
SUCCESS EasyBuild-4.9.4.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
bear-pg0208u17a - Linux RHEL 8.10, x86_64, AMD EPYC 9554 64-Core Processor (zen4), Python 3.6.8
See https://gist.github.com/branfosj/f0f3695b09c12b27a53f4da90a5804e3 for a full test report.

branfosj · 2025-05-23T11:38:29Z

the only edge case I can think of is that with #3725 merged, we might be overwriting the egg-info file in the case where easbuild is in the sources

6aa115e to protect against that.

I'm happy for this to wait to after the release of 5.1.0.

branfosj · 2025-05-23T11:48:11Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS EasyBuild-5.0.0.eb
SUCCESS EasyBuild-4.9.4.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
bear-pg0208u17a - Linux RHEL 8.10, x86_64, AMD EPYC 9554 64-Core Processor (zen4), Python 3.6.8
See https://gist.github.com/branfosj/d55d27afae7f862a9d11e1694626af09 for a full test report.

jfgrimm · 2025-05-23T11:50:11Z

@branfosj I believe this is sufficiently guarded, so imo we should be fine to merge
I'll wait to see if @boegel objects

branfosj · 2025-05-23T11:51:12Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS EasyBuild-5.0.0-eb_in_src.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
bear-pg0208u17a - Linux RHEL 8.10, x86_64, AMD EPYC 9554 64-Core Processor (zen4), Python 3.6.8
See https://gist.github.com/branfosj/7ee5c636f2de64db36a9c3f657307c2f for a full test report.

edit EasyBuild-5.0.0-eb_in_src.eb is the easyconfig from easybuilders/easybuild-easyconfigs#22774

…cts in constructor of EasyBuildMeta easyblock

boegel · 2025-05-23T13:03:08Z

            # consider setuptools first, in case it is listed as a sources
            self.easybuild_pkgs.insert(0, 'setuptools')
+        elif (LooseVersion(self.version) >= LooseVersion('5.0') and
+              any(f'easybuild-{self.version}' in source['filename'] for source in self.cfg['sources'])):


source may not be a dict, depends on the easyconfig file being used, see EasyBuild-4.9.1.eb for example, so we need be be more careful here: branfosj#2

boegel · 2025-05-23T13:05:30Z

Test report by @boegel

Overview of tested easyconfigs (in order)

**FAIL (unhandled exception: string indices must be integers)Traceback (most recent call last):
File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/main.py", line 160, in build_and_install_software
(ec_res['success'], app_log, err_msg, err_code) = build_and_install_one(ec, init_env)
File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 5015, in build_and_install_one
result = app.run_all_steps(run_test_cases=run_test_cases)
File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 4835, in run_all_steps
self.run_step(step_name, step_methods)
File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 4677, in run_step
step_method(self)()
File "/tmp/eb-8d442hzp/included-easyblocks-mp0_j1nj/easybuild/easyblocks/easybuildmeta.py", line 148, in install_step
if not any(f'easybuild-{self.version}' in source['filename'] for source in self.cfg['sources']):
File "/tmp/eb-8d442hzp/included-easyblocks-mp0_j1nj/easybuild/easyblocks/easybuildmeta.py", line 148, in
if not any(f'easybuild-{self.version}' in source['filename'] for source in self.cfg['sources']):
TypeError: string indices must be integers
** EasyBuild-4.9.1.eb

Build succeeded for 0 out of 1 (1 easyconfigs in total)
node3502.doduo.os - Linux RHEL 9.4, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.9.18
See https://gist.github.com/boegel/abd97e6f226207471a6dfeee279a43c9 for a full test report.

take into account that sources may be a list of strings instead of dicts in constructor of `EasyBuildMeta` easyblock

jfgrimm · 2025-05-23T13:20:35Z

Test report by @jfgrimm

Overview of tested easyconfigs (in order)

SUCCESS EasyBuild-5.0.0.eb
SUCCESS EasyBuild-4.9.1.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node034.viking2.yor.alces.network - Linux Rocky Linux 8.9, x86_64, AMD EPYC 7643 48-Core Processor, Python 3.6.8
See https://gist.github.com/jfgrimm/7f975722d3a5a7caa7103c4e48519b60 for a full test report.

jfgrimm · 2025-05-23T13:23:20Z

Test report by @jfgrimm

Overview of tested easyconfigs (in order)

SUCCESS EasyBuild-5.0.0_eb_in_src.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node034.viking2.yor.alces.network - Linux Rocky Linux 8.9, x86_64, AMD EPYC 7643 48-Core Processor, Python 3.6.8
See https://gist.github.com/jfgrimm/5b7b1c122e6e45ff57cc260ae9e63fe6 for a full test report.

boegel

lgtm

boegel · 2025-05-23T13:27:49Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS EasyBuild-4.4.0.eb
SUCCESS EasyBuild-4.7.0.eb
SUCCESS EasyBuild-4.9.4.eb
SUCCESS EasyBuild-5.0.0.eb
SUCCESS EasyBuild-0.5.0.0beta2.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
node4221.shinx.os - Linux RHEL 9.4, x86_64, AMD EPYC 9654 96-Core Processor (zen4), Python 3.9.18
See https://gist.github.com/boegel/781ac515eaba40b84cefdf8635a63cb6 for a full test report.

branfosj · 2025-05-23T13:33:31Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS EasyBuild-5.0.0-eb_in_src.eb
SUCCESS EasyBuild-5.0.0.eb
SUCCESS EasyBuild-4.9.1.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
bear-pg0210u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8480CL (sapphirerapids), Python 3.6.8
See https://gist.github.com/branfosj/d9d8fdd324944c04559200e568ad054c for a full test report.

jfgrimm · 2025-05-23T13:35:41Z

I've created #3733 as a reminder to remove the egg-info workaround at some future point
I'd tag it as 6.0, but we don't have a milestone for that yet :P

also install easybuild package for EasyBuild >= 5

78b674b

branfosj added the change label Apr 17, 2025

branfosj mentioned this pull request Apr 17, 2025

add easybuild package to EasyBuild/5.0.0 easybuilders/easybuild-easyconfigs#22774

Closed

1 task

Micket previously approved these changes Apr 17, 2025

View reviewed changes

Micket added this to the release after 5.0.0 milestone Apr 17, 2025

boegel modified the milestones: next release (5.1.0), release after 5.1.0 May 21, 2025

jfgrimm reviewed May 21, 2025

View reviewed changes

Comment thread easybuild/easyblocks/e/easybuildmeta.py Outdated

branfosj mentioned this pull request May 21, 2025

generate an egg-info file for the easybuild meta package #3725

Merged

Micket closed this May 23, 2025

jfgrimm reopened this May 23, 2025

branfosj and others added 3 commits May 23, 2025 10:52

protect the check for the package behind a source check

9dbc4db

Co-authored-by: Jasper Grimm <[email protected]>

break up the long line

1311e56

indentation

42b445a

jfgrimm added the enhancement label May 23, 2025

jfgrimm modified the milestones: release after 5.1.0, next release (5.1.0) May 23, 2025

Merge branch 'easybuilders:develop' into 20250417130842_new_pr_easybu…

b342978

…ildmeta

only install the egginfo if easybuild is not in the sources

6aa115e

take into account that sources may be a list of strings instead of di…

4d0b642

…cts in constructor of EasyBuildMeta easyblock

boegel mentioned this pull request May 23, 2025

take into account that sources may be a list of strings instead of dicts in constructor of EasyBuildMeta easyblock branfosj/easybuild-easyblocks#2

Merged

boegel requested changes May 23, 2025

View reviewed changes

Merge pull request #2 from boegel/20250417130842_new_pr_easybuildmeta

e8a209a

take into account that sources may be a list of strings instead of dicts in constructor of `EasyBuildMeta` easyblock

boegel approved these changes May 23, 2025

View reviewed changes

boegel merged commit 72967c5 into easybuilders:develop May 23, 2025
17 checks passed

jfgrimm mentioned this pull request May 23, 2025

(EB 6.0?) Revert easybuild meta package egg-info workaround #3733

Open

branfosj deleted the 20250417130842_new_pr_easybuildmeta branch May 23, 2025 13:35

Conversation

branfosj commented Apr 17, 2025

Uh oh!

Micket left a comment

Choose a reason for hiding this comment

Uh oh!

boegel commented Apr 23, 2025

Uh oh!

jfgrimm commented May 21, 2025

Uh oh!

branfosj commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfgrimm commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

branfosj commented May 21, 2025

Uh oh!

Micket commented May 23, 2025

Uh oh!

jfgrimm commented May 23, 2025

Uh oh!

boegel commented May 23, 2025

Uh oh!

jfgrimm commented May 23, 2025

Uh oh!

jfgrimm commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview of tested easyconfigs (in order)

Uh oh!

jfgrimm commented May 23, 2025

Uh oh!

boegel commented May 23, 2025

Uh oh!

branfosj commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfgrimm commented May 23, 2025

Uh oh!

branfosj commented May 23, 2025

Overview of tested easyconfigs (in order)

Uh oh!

branfosj commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

branfosj commented May 23, 2025

Overview of tested easyconfigs (in order)

Uh oh!

jfgrimm commented May 23, 2025

Uh oh!

branfosj commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview of tested easyconfigs (in order)

Uh oh!

boegel May 23, 2025

Choose a reason for hiding this comment

Uh oh!

boegel commented May 23, 2025

Overview of tested easyconfigs (in order)

Uh oh!

jfgrimm commented May 23, 2025

Overview of tested easyconfigs (in order)

Uh oh!

jfgrimm commented May 23, 2025

Overview of tested easyconfigs (in order)

Uh oh!

boegel left a comment

Choose a reason for hiding this comment

Uh oh!

boegel commented May 23, 2025

Overview of tested easyconfigs (in order)

Uh oh!

branfosj commented May 23, 2025

Overview of tested easyconfigs (in order)

Uh oh!

Uh oh!

jfgrimm commented May 23, 2025

Uh oh!

Reviewers

branfosj commented May 21, 2025 •

edited

Loading

jfgrimm commented May 21, 2025 •

edited

Loading

jfgrimm commented May 23, 2025 •

edited

Loading

branfosj commented May 23, 2025 •

edited

Loading

branfosj commented May 23, 2025 •

edited

Loading

branfosj commented May 23, 2025 •

edited

Loading