{tools}[GCCcore/12.3.0] Python-bundle-PyPI v2023.06, hatchling v1.18.0, poetry v1.5.1, setuptools-rust v1.6.0, virtualenv v20.23.1, ... w/ Python 3.11.3 by SebastianAchilles · Pull Request #18208 · easybuilders/easybuild-easyconfigs

SebastianAchilles · 2023-06-27T15:05:15Z

(created using eb --new-pr)

Alternative for #18193

This PR splits of poetry, hatchling, and virtualenv into their own easyconfigs, since they are mostly used for building Python packages. Poetry is using hatchling and virtualenv as dependencies.

Python-bundle-PyPI uses poetry, hatchling, and virtualenv as builddependencies. It contains the same packages (but newer versions) as https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/p/Python/Python-3.10.8-GCCcore-12.2.0.eb except the following packages which I think are mostly needed for building python packages:

tomli
setuptools_scm
setuptools-rust
distlib
pathspec
editables
trove_classifiers
hatchling
hatch_vcs
filelock
virtualenv
hatch_fancy_pypi_readme
flit_scm
tomli_w
flit
clikit
tomlkit
shellingham
requests-toolbelt
pkginfo
ptyprocess
pexpect
distro
scikit_build
lockfile
poetry_core
dulwich
poetry_plugin_export
build
installer
pyproject_hooks
poetry

For now I am keeping the removed packages as comments in case we want to add any of them back. However, before merging this MR, I want to clean up the comments.

…chling-1.18.0-GCCcore-12.3.0.eb, virtualenv-20.23.1-GCCcore-12.3.0.eb, poetry-1.5.1-GCCcore-12.3.0.eb and patches: Python-3_9-blist-1.3.6-fix-undefined_symbol_PyObject_GC_IS_TRACKED.patch, Python-3.10-bist-1.3.6-compatibility.patch, Python-3.11-bist-1.3.6-compatibility.patch

…and hatchling-1.18.0-GCCcore-12.3.0.eb

SebastianAchilles · 2023-06-27T16:34:59Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
bwd-rockylinux-92 - Linux Rocky Linux 9.2 (Blue Onyx), x86_64, Intel(R) Core(TM) i7-6900K CPU @ 3.20GHz (broadwell), 2 x NVIDIA NVIDIA GeForce GTX 1060 6GB, 525.105.17, Python 3.9.16
See https://gist.github.com/SebastianAchilles/e2c82712d2d85a0c9c89078afca15bbc for a full test report.

SebastianAchilles · 2023-06-27T16:35:18Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
zen2-rockylinux-88 - Linux Rocky Linux 8.8, x86_64, AMD EPYC 7452 32-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/SebastianAchilles/61dd0363884ec7582af938ebffc7362d for a full test report.

SebastianAchilles · 2023-06-27T16:37:05Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
skl-centos-79 - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (skylake), Python 3.6.8
See https://gist.github.com/SebastianAchilles/8e27debcdbecd8d4da6b4848d62c044a for a full test report.

SebastianAchilles · 2023-06-27T16:50:55Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
zen2-ubuntu-2304 - Linux Ubuntu 23.04 (Lunar Lobster), x86_64, AMD EPYC 7452 32-Core Processor (zen2), Python 3.11.2
See https://gist.github.com/SebastianAchilles/4d66cd927a136614c1b221d7c6a70805 for a full test report.

SebastianAchilles · 2023-06-27T16:53:24Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
skl-opensuse-155 - Linux openSUSE Leap 15.5, x86_64, Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (skylake), Python 3.6.15
See https://gist.github.com/SebastianAchilles/cb3159c93f8ea3dd228d9959353e4f36 for a full test report.

boegel · 2023-06-28T08:23:03Z

@SebastianAchilles Except for hatchling (and hatch*), poetry, and virtualenv, I would include all other extensions we had in Python into the bundle.

It's very difficult to asses whether they're only used as build dependencies or not, I think.

There may be some clear exceptions, like scikit-build and setuptools-rust, which should maybe be moved to their own easyconfig as well, but I would keep the rest in the bundle (for now), and re-evaluate that approach later.

…or building python packages

SebastianAchilles · 2023-06-28T09:26:12Z

@boegel Sounds reasonable. I have added most extensions back, except:

setuptools_scm
setuptools-rust
trove_classifiers a new dependency for hatchling which was not part of Python-3.10.8-GCCcore-12.2.0.eb
hatch*
virtualenv
flit_scm and flit a simple packaging tool for simple packages, both are part of poetry
clikit because it would require using an old version of crashtest
scikit_build because it is only a build dependency needed in the poetry easyconfig and was not part of Python-3.10.8-GCCcore-12.2.0.eb
build, installer and pyproject_hooks which are new build dependencies for poetry and were not part of Python-3.10.8-GCCcore-12.2.0.eb
poetry*

SebastianAchilles · 2023-06-28T09:54:55Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
zen2-ubuntu-2304 - Linux Ubuntu 23.04 (Lunar Lobster), x86_64, AMD EPYC 7452 32-Core Processor (zen2), Python 3.11.2
See https://gist.github.com/SebastianAchilles/2255c819e8ab842531c62a8e1ea911a9 for a full test report.

SebastianAchilles · 2023-06-28T09:55:56Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
zen2-rockylinux-88 - Linux Rocky Linux 8.8, x86_64, AMD EPYC 7452 32-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/SebastianAchilles/a20486465444fd22853ead9d08f54e1d for a full test report.

SebastianAchilles · 2023-06-28T09:56:42Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
skl-opensuse-155 - Linux openSUSE Leap 15.5, x86_64, Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (skylake), Python 3.6.15
See https://gist.github.com/SebastianAchilles/90583ca76d96fcb8116b300c12d14181 for a full test report.

SebastianAchilles · 2023-06-28T09:56:53Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
skl-centos-79 - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (skylake), Python 3.6.8
See https://gist.github.com/SebastianAchilles/4c21c8f83099780aa7e5ca8233f1a50e for a full test report.

SebastianAchilles · 2023-06-28T10:44:01Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
bwd-rockylinux-92 - Linux Rocky Linux 9.2 (Blue Onyx), x86_64, Intel(R) Core(TM) i7-6900K CPU @ 3.20GHz (broadwell), 2 x NVIDIA NVIDIA GeForce GTX 1060 6GB, 525.105.17, Python 3.9.16
See https://gist.github.com/SebastianAchilles/7fe1c4a14844c45a2568cb1fc0324345 for a full test report.

boegel

changes that should be made here:

separate easyconfig for cryptography using CargoPythonPackage (mainly to avoid download of Rust crates during installation)
- maybe using the wheel, if needed?
- use python3 -m easybuild.easyblocks.cargo . in src/rust directory of cryptography to generate crates list
separate easyconfig for setuptools-rust (build dep for cryptography)
no need for virtualenv as build dep in Python-bundle-PyPI + virtualenv should depend on Python-bundle-PyPI

…te easyconfig for setuptools-rust, virtualenv, pycparser, cffi and scikit-build

SebastianAchilles · 2023-07-13T13:09:45Z

changes that should be made here:

* separate easyconfig for `cryptography` using `CargoPythonPackage` (mainly to avoid download of Rust crates during installation)
  
  * maybe using the wheel, if needed?
  * use `python3 -m easybuild.easyblocks.cargo .` in [`src/rust`](https://github.com/pyca/cryptography/blob/main/src/rust/Cargo.lock) directory of `cryptography` to generate `crates` list

* separate easyconfig for `setuptools-rust` (build dep for `cryptography`)

* no need for `virtualenv` as build dep in `Python-bundle-PyPI` + `virtualenv` should depend on `Python-bundle-PyPI`

Good ideas 👍 The requested changes are now implemented.

SebastianAchilles · 2023-07-13T13:21:17Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 9 out of 9 (9 easyconfigs in total)
skl-rockylinux-88 - Linux Rocky Linux 8.8, x86_64, Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz (skylake), 1 x NVIDIA NVIDIA RTX A4000, 530.30.02, Python 3.6.8
See https://gist.github.com/SebastianAchilles/e9da6fb192e9e969e1ec1028b0bac007 for a full test report.

SebastianAchilles · 2023-07-13T13:26:13Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 9 out of 9 (9 easyconfigs in total)
zen2-ubuntu-2304 - Linux Ubuntu 23.04 (Lunar Lobster), x86_64, AMD EPYC 7452 32-Core Processor (zen2), Python 3.11.2
See https://gist.github.com/SebastianAchilles/2a875ea72fb5e2ab6db4750fee911513 for a full test report.

lexming

This looks very good, I have a few more changes and this is ready to go on my side 🙂

Micket · 2023-07-18T16:43:08Z

Was there any reason not to bundle in pycparser with cffi (in an effort to minimize number of PYTHONPATHs)?

lexming

Two more changes and this is totally ready on my side.

Co-authored-by: Alex Domingo <[email protected]>

SebastianAchilles · 2023-07-19T20:07:36Z

Was there any reason not to bundle in pycparser with cffi (in an effort to minimize number of PYTHONPATHs)?

There was no reason. When I worked on that, I still though both cffi and pycparser would only be build dependencies, so the would not end up in the PYTHONPATH. Let me change this.

…PyPI-2023.06-GCCcore-12.3.0.eb

SebastianAchilles · 2023-07-19T20:40:22Z

Here's the updated dependency graph:

SebastianAchilles · 2023-07-19T20:59:29Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 11 out of 11 (11 easyconfigs in total)
bwd-rockylinux-92 - Linux Rocky Linux 9.2 (Blue Onyx), x86_64, Intel(R) Core(TM) i7-6900K CPU @ 3.20GHz (broadwell), 2 x NVIDIA NVIDIA GeForce GTX 1060 6GB, 535.54.03, Python 3.9.16
See https://gist.github.com/SebastianAchilles/8f97d8efd536891f96d0383bc83051dd for a full test report.

lexming

@SebastianAchilles thanks for all the updates, I just made a last minor change removing 2 extensions from poetry already provided by the dep in virtualenv

LGTM

I've been also keeping track of the dependency graph with a diagram 🙂

lexming · 2023-07-20T13:19:30Z

@boegelbot: please test @ generoso

boegelbot · 2023-07-20T13:20:10Z

@lexming: Request for testing this PR well received on login1

PR test command 'EB_PR=18208 EB_ARGS= EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_18208 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 11274

Test results coming soon (I hope)...

Details

- notification for comment with ID 1643914410 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2023-07-20T13:53:39Z

Test report by @boegelbot
SUCCESS
Build succeeded for 11 out of 11 (11 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/7f33d2221db9110da2a559768b265329 for a full test report.

lexming · 2023-07-21T09:46:05Z

@boegelbot please test @ jsc-zen2

boegelbot · 2023-07-21T09:47:08Z

@lexming: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=18208 EB_ARGS= /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_18208 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

exit code: 0
output:

Submitted batch job 3005

Test results coming soon (I hope)...

Details

- notification for comment with ID 1645311125 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2023-07-21T10:12:27Z

Test report by @boegelbot
SUCCESS
Build succeeded for 11 out of 11 (11 easyconfigs in total)
jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/boegelbot/483767dd4c5ed4ef5e33268b3af5410d for a full test report.

lexming · 2023-07-21T10:23:11Z

This is ready to be merged. I think the best test now is to start building on top of these easyconfigs in case potential issues would show up down the line.

So going in, thanks @SebastianAchilles !

Review addressed by author

SebastianAchilles added new update 2023a labels Jun 27, 2023

SebastianAchilles added this to the 4.x milestone Jun 27, 2023

SebastianAchilles added 2 commits June 27, 2023 17:19

add pkgconf builddependency to poetry-1.5.1-GCCcore-12.3.0.eb

6d49b8d

add binutils builddependency to virtualenv-20.23.1-GCCcore-12.3.0.eb …

e54f709

…and hatchling-1.18.0-GCCcore-12.3.0.eb

This comment was marked as outdated.

Sign in to view

boegel mentioned this pull request Jun 28, 2023

{lang}[GCCcore/12.3.0] Python-bundle-PyPI v2023.06 w/ Python 3.11.3 #18193

Closed

1 task

include all extensions except the ones that are clearly only needed f…

86d1f16

…or building python packages

boegel previously requested changes Jul 5, 2023

View reviewed changes

SebastianAchilles added 4 commits July 13, 2023 14:10

separate easyconfig for cryptography using CargoPythonPackage, separa…

f164003

…te easyconfig for setuptools-rust, virtualenv, pycparser, cffi and scikit-build

add pkgconf builddependency to cryptography easyconfig

82069cc

add binutils builddependency to pycparser and cffi easyconfig

34e5812

update virtualenv to depend on Python-bundle-PyPI

02e8f5c

lexming requested changes Jul 18, 2023

View reviewed changes

Comment thread easybuild/easyconfigs/p/poetry/poetry-1.5.1-GCCcore-12.3.0.eb Outdated

Comment thread easybuild/easyconfigs/p/poetry/poetry-1.5.1-GCCcore-12.3.0.eb Outdated

SebastianAchilles and others added 6 commits July 18, 2023 20:04

update description in setuptools-rust

51fbca8

Co-authored-by: Alex Domingo <[email protected]>

update homepage in virtualenv

233bd75

Co-authored-by: Alex Domingo <[email protected]>

update description in hatchling

7afcc7e

Co-authored-by: Alex Domingo <[email protected]>

update dependencies in poetry

56593ee

Co-authored-by: Alex Domingo <[email protected]>

update extensions in poetry-1.5.1-GCCcore-12.3.0.eb

542beb8

Co-authored-by: Alex Domingo <[email protected]>

add virtualenv dependency in poetry-1.5.1-GCCcore-12.3.0.eb

ae8a78f

SebastianAchilles added 3 commits July 19, 2023 22:07

bundle pycparser with cffi in cffi-1.15.1-GCCcore-12.3.0.eb

0c6111c

adding easyconfigs: flit-3.9.0-GCCcore-12.3.0.eb

30cf48a

add flit build dependency and virtualenv dependency to Python-bundle-…

03a3b99

…PyPI-2023.06-GCCcore-12.3.0.eb

remove extensions provided by virtualenv from poetry v1.5.1

cd8848d

lexming approved these changes Jul 20, 2023

View reviewed changes

lexming merged commit 587e7c8 into easybuilders:develop Jul 21, 2023

casparvl mentioned this pull request Aug 22, 2023

{tools}[GCCcore/12.3.0] PycURL v7.45.2, PyZMQ v25.1.1, jupyter-server v2.7.2, ... #18617

Merged

4 tasks

boegel mentioned this pull request Sep 1, 2023

{lib}[GCCcore/12.3.0] html5lib v1.1 #18597

Merged

Conversation

SebastianAchilles commented Jun 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

SebastianAchilles commented Jun 27, 2023

Uh oh!

SebastianAchilles commented Jun 27, 2023

Uh oh!

SebastianAchilles commented Jun 27, 2023

Uh oh!

SebastianAchilles commented Jun 27, 2023

Uh oh!

SebastianAchilles commented Jun 27, 2023

Uh oh!

boegel commented Jun 28, 2023

Uh oh!

SebastianAchilles commented Jun 28, 2023

Uh oh!

SebastianAchilles commented Jun 28, 2023

Uh oh!

SebastianAchilles commented Jun 28, 2023

Uh oh!

SebastianAchilles commented Jun 28, 2023

Uh oh!

SebastianAchilles commented Jun 28, 2023

Uh oh!

SebastianAchilles commented Jun 28, 2023

Uh oh!

boegel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SebastianAchilles commented Jul 13, 2023

Uh oh!

SebastianAchilles commented Jul 13, 2023

Uh oh!

SebastianAchilles commented Jul 13, 2023

Uh oh!

lexming left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Micket commented Jul 18, 2023

Uh oh!

lexming left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

SebastianAchilles commented Jul 19, 2023

Uh oh!

SebastianAchilles commented Jul 19, 2023

Uh oh!

SebastianAchilles commented Jul 19, 2023

Uh oh!

lexming left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lexming commented Jul 20, 2023

Uh oh!

boegelbot commented Jul 20, 2023

Uh oh!

boegelbot commented Jul 20, 2023

Uh oh!

lexming commented Jul 21, 2023

Uh oh!

boegelbot commented Jul 21, 2023

Uh oh!

boegelbot commented Jul 21, 2023

Uh oh!

lexming commented Jul 21, 2023

Uh oh!

Reviewers

SebastianAchilles commented Jun 27, 2023 •

edited

Loading

boegel left a comment •

edited

Loading

lexming left a comment •

edited

Loading

lexming left a comment •

edited

Loading