allow Python bundles and packages to specify a maximum Python version for the system toolchain by casparvl · Pull Request #3431 · easybuilders/easybuild-easyblocks

casparvl · 2024-09-04T11:22:55Z

We implement support for specifying a maximum python version for EasyConfigs that use system python (through the max_py_majver, max_py_minver keywords). This is is required for easybuilders/easybuild-easyconfigs#21307 which has one EasyConfig that will only work on the python 3.6 - 3.11 range (3.12 will fail because the setuptools is too old, but newer setuptools do not support python3.6).

This PR is built on top of #3430 . That one should probably be reviewed & merged first, after which this PR should be synced with develop.

… based easyconfigs

…e and also implement in pythonbundle

… a maximum version

…les#r1745641431

…k, there the req_py_majver and friends are guaranteed to be defined

…olchain

bedroge · 2024-09-06T08:05:01Z

Test report by @bedroge

Overview of tested easyconfigs (in order)

SUCCESS pyBigWig-0.3.22-gfbf-2023a.eb
SUCCESS PyYAML-6.0-GCCcore-12.2.0.eb
SUCCESS sktime-0.25.0-gfbf-2023a.eb
SUCCESS NiBabel-4.0.2-foss-2022a.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total)
bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3
See https://gist.github.com/bedroge/3235d0ff53051a7f7472f287c85e5194 for a full test report.

bedroge · 2024-09-06T08:23:40Z

Test report by @bedroge

Overview of tested easyconfigs (in order)

FAIL (build issue) ReFrame-4.6.2-Python-3.6.eb (partial log available at https://gist.github.com/bedroge/bdacc658fc5a6a96e373ad8f5505b7c2)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3
See https://gist.github.com/bedroge/c5bc91e9bf2dab40ee1d2eec64f6eba2 for a full test report.

Expected failure that confirms that the new functionality works. I used Python 3.12 for a ReFrame version (from easybuilders/easybuild-easyconfigs#21307) that only supports Python <= 3.11.

== FAILED: Installation ended unsuccessfully (build directory: /data/eb/build/ReFrame/4.6.2/system-system-Python-3.6): build failed (first 300 chars): Failed to pick Python command that satisfies requirements in the easyconfig (req_py_majver = 3, req_py_minver = 6, max_py_majver = 3, max_py_minver = 11) (took 0 secs)

bedroge · 2024-09-06T08:25:58Z

Test report by @bedroge

Overview of tested easyconfigs (in order)

SUCCESS ReFrame-4.6.2.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3
See https://gist.github.com/bedroge/0a9eb4ce1c0a45c11a91de2ef7e08122 for a full test report.

bedroge · 2024-09-06T08:37:30Z

Test report by @bedroge

Overview of tested easyconfigs (in order)

FAIL (build issue) ReFrame-4.6.2-Python-3.6.eb (partial log available at https://gist.github.com/bedroge/78dc999c06ebd0f46f3bfe56f69a5a70)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3
See https://gist.github.com/bedroge/5459719d39a8ec3d367525b0a0eb14cc for a full test report.

bedroge · 2024-09-06T08:37:38Z

Redoing all test builds to make sure things still work.

bedroge · 2024-09-06T08:39:48Z

Test report by @bedroge

Overview of tested easyconfigs (in order)

SUCCESS pyBigWig-0.3.22-gfbf-2023a.eb
SUCCESS PyYAML-6.0-GCCcore-12.2.0.eb
SUCCESS sktime-0.25.0-gfbf-2023a.eb
SUCCESS NiBabel-4.0.2-foss-2022a.eb
SUCCESS ReFrame-4.6.2.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3
See https://gist.github.com/bedroge/9e6d8b842a542e42333cb212c2c2b18b for a full test report.

bedroge · 2024-09-06T08:47:52Z

Test report by @bedroge

Overview of tested easyconfigs (in order)

FAIL (build issue) ReFrame-4.6.2-Python-3.6.eb (partial log available at https://gist.github.com/bedroge/79e8d70e09d85997faee4b54764edf73)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3
See https://gist.github.com/bedroge/62e41ef10b452bb177eeb4c9d67484c7 for a full test report.

Reworded the debug message a bit:

Python version (3.12.3) on the system is newer than the maximum supported Python version specified in the easyconfig (3.11)

bedroge · 2024-09-06T08:48:47Z

Test report by @bedroge

Overview of tested easyconfigs (in order)

SUCCESS ReFrame-4.6.2.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3
See https://gist.github.com/bedroge/1a18889f623416471276207030f5d75f for a full test report.

bedroge · 2024-09-06T10:02:40Z

@casparvl I did a search for pick_python_cmd, and found that there are unit tests for this function. So it would be good to add some additional ones here https://github.com/easybuilders/easybuild-easyblocks/blob/develop/test/easyblocks/module.py#L227.

edit: added two tests in casparvl#4.

…inver

Add two tests for the max_py_* parameters

casparvl · 2024-09-16T12:53:14Z

@bedroge I merged your unit test. Is there anything else that needs to be done for this PR to be merged? :)

bedroge

Lgtm

bedroge · 2024-09-17T11:23:12Z

Test report by @bedroge

Overview of tested easyconfigs (in order)

SUCCESS pyBigWig-0.3.22-gfbf-2023a.eb
SUCCESS PyYAML-6.0-GCCcore-12.2.0.eb
SUCCESS sktime-0.25.0-gfbf-2023a.eb
SUCCESS NiBabel-4.0.2-foss-2022a.eb
SUCCESS ReFrame-4.6.2.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3
See https://gist.github.com/bedroge/e084b7ecccf94c2730b4362c9e8887d3 for a full test report.

Flamefire · 2024-10-09T12:45:49Z

@casparvl What is the intended semantic for specifying only a major version? I would expect that max_py_majver = 3 would accept any Python 3 version. Or maybe more sensible: max_py_majver = 2 would only exclude Python 3. Currently the implementation seems to check for < max_py_majver, which means Python 3.1 and Python 2.1 respectively would be excluded.

Caspar van Leeuwen and others added 10 commits September 4, 2024 11:48

allow specifying a max version for the system python in pythonpackage…

7fd6e9d

… based easyconfigs

Fix implementation of max_py_majver and max_py_minver in pythonpackag…

fdf535b

…e and also implement in pythonbundle

Fix hound issues

944872c

Formatting

1d24517

Fix more formatting issues

2194e64

Fix more formatting issues

7256a1c

Fix indent

8f2244f

Fix indent

beb99f5

Add missing colon

40020d2

Renamed parameters to match argument names

debc72e

casparvl pushed a commit to casparvl/easybuild-easyconfigs that referenced this pull request Sep 4, 2024

Use implementation from easybuilders/easybuild-easyblocks#3431 to set…

94d34a5

… a maximum version

casparvl mentioned this pull request Sep 4, 2024

{devel}[SYSTEM] Reframe v4.6.2 easybuilders/easybuild-easyconfigs#21307

Merged

Caspar van Leeuwen and others added 5 commits September 5, 2024 16:31

Fix https://github.com/easybuilders/easybuild-easyblocks/pull/3430/fi…

1a6802b

…les#r1745641431

Initialize variables, so that they are present when we check for them

191a691

Move the complicated error reporting within the 'python is None' bloc…

4525a76

…k, there the req_py_majver and friends are guaranteed to be defined

Pull error inside block that is executed when using system python

9999bf1

Merge branch 'develop' into allow_specifying_max_python_ver_system_to…

1c2947c

…olchain

bedroge changed the title ~~Allow specifying max python ver system toolchain~~ allow Python bundles and packages to specify a maximum Python version for the system toolchain Sep 6, 2024

bedroge reviewed Sep 6, 2024

View reviewed changes

Comment thread easybuild/easyblocks/generic/pythonbundle.py Outdated

No need to initialize the req/max python version variables

d552094

Rephrase debug message

1a23bf3

bedroge added 3 commits September 6, 2024 13:23

add two tests for pick_python_cmd that use max_py_majver and max_py_m…

1e6f773

…inver

deliberately try to break a test

0c5cc9d

this indeed broke the test, so revert it

53778ed

boegel added this to the 4.x milestone Sep 7, 2024

boegel added the enhancement label Sep 11, 2024

Merge pull request #4 from bedroge/tests_max_python_ver

efddeb0

Add two tests for the max_py_* parameters

bedroge approved these changes Sep 17, 2024

View reviewed changes

bedroge modified the milestones: 4.x, release after 4.9.3 Sep 17, 2024

bedroge merged commit 1ca6dc3 into easybuilders:develop Sep 17, 2024

boegel modified the milestones: release after 4.9.3, 4.9.4 Sep 19, 2024

Conversation

casparvl commented Sep 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedroge commented Sep 6, 2024

Overview of tested easyconfigs (in order)

Uh oh!

bedroge commented Sep 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview of tested easyconfigs (in order)

Uh oh!

bedroge commented Sep 6, 2024

Overview of tested easyconfigs (in order)

Uh oh!

Uh oh!

bedroge commented Sep 6, 2024

Overview of tested easyconfigs (in order)

Uh oh!

bedroge commented Sep 6, 2024

Uh oh!

bedroge commented Sep 6, 2024

Overview of tested easyconfigs (in order)

Uh oh!

bedroge commented Sep 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview of tested easyconfigs (in order)

Uh oh!

bedroge commented Sep 6, 2024

Overview of tested easyconfigs (in order)

Uh oh!

bedroge commented Sep 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

casparvl commented Sep 16, 2024

Uh oh!

bedroge left a comment

Choose a reason for hiding this comment

Uh oh!

bedroge commented Sep 17, 2024

Overview of tested easyconfigs (in order)

Uh oh!

Flamefire commented Oct 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

casparvl commented Sep 4, 2024 •

edited

Loading

bedroge commented Sep 6, 2024 •

edited

Loading

bedroge commented Sep 6, 2024 •

edited

Loading

bedroge commented Sep 6, 2024 •

edited

Loading