Skip to content

enhance PythonPackage & PythonBundle easyblock to consider $EBPYTHONPREFIXES for test installs and path configuration files#3953

Merged
Micket merged 6 commits intoeasybuilders:developfrom
Thyre:20251009223823_new_pr_pythonpackage
Oct 15, 2025
Merged

enhance PythonPackage & PythonBundle easyblock to consider $EBPYTHONPREFIXES for test installs and path configuration files#3953
Micket merged 6 commits intoeasybuilders:developfrom
Thyre:20251009223823_new_pr_pythonpackage

Conversation

@Thyre
Copy link
Copy Markdown
Collaborator

@Thyre Thyre commented Oct 9, 2025

(created using eb --new-pr)


There are rare cases where a Python package provides path configuration files to set-up some state for the package to work correctly. One such example was recently encountered in easybuilders/easybuild-easyconfigs#24081: numba-cuda.

Numba has deprecated their CUDA bindings. Instead, NVIDIA provides bindings which hook into numba. The installation is entirely separate. However, they provide a path configuration file (.pth) which basically overwrites the include for numba.cuda.
Typically, these files only work when they are in a system directory checked by Python. This is the case in a venv or in a user/system wide installation. In EasyBuild however, this fails. Writing our own sitecustomize.py for numba-cuda would shadow the one provided in EasyBuild. For this to work, we require EBPYTHONPREFIXES, as our sitecustomize.py adds the directories to the site dir, therefore being considered by Python

if os.path.isdir(sitedir):
if debug:
print("[%(EBPYTHONPREFIXES)s] adding site dir: %%s" %% sitedir)
site.addsitedir(sitedir)


This PR implements these changes:

  1. Refactor the logic to determine if we need EBPYTHONPREFIXES into its own function using_ebpythonprefixes
  2. Add a check if this is required, by checking the install directory and, if needed, the test install directory.
  3. Use this refactored method in make_module_extra and test_step to check if we need EBPYTHONPREFIXES.

Marked as draft until I was able to verify that these changes work. Unfortunately, none of my systems are using EBPYTHONPREFIXES, making test reports like easybuilders/easybuild-easyconfigs#24081 (comment) harder to reproduce.

@Thyre Thyre marked this pull request as draft October 9, 2025 20:38
Comment thread easybuild/easyblocks/generic/pythonpackage.py Outdated
Comment thread easybuild/easyblocks/generic/pythonpackage.py Outdated
Thyre added a commit to Thyre/easybuild-easyconfigs that referenced this pull request Oct 10, 2025
@Thyre Thyre changed the title enhance PythonPackage easyblock to consider EBPYTHONPREFIXES for test installs and path configuration files enhance PythonPackage & PythonBundle easyblock to consider EBPYTHONPREFIXES for test installs and path configuration files Oct 10, 2025
@Thyre Thyre removed the help wanted label Oct 10, 2025
@Thyre Thyre requested a review from Micket October 10, 2025 18:13
@Thyre Thyre added this to the next release (5.2.0?) milestone Oct 10, 2025
@Thyre Thyre marked this pull request as ready for review October 10, 2025 18:14
@Thyre
Copy link
Copy Markdown
Collaborator Author

Thyre commented Oct 10, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="Python-bundle-PyPI-2025.07-GCCcore-14.3.0.eb SciPy-bundle-2025.07-gfbf-2025b.eb --installpath=/tmp/$USER/ebpr-3953"

@boegelbot
Copy link
Copy Markdown

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3953 EB_ARGS="Python-bundle-PyPI-2025.07-GCCcore-14.3.0.eb SciPy-bundle-2025.07-gfbf-2025b.eb --installpath=/tmp/$USER/ebpr-3953" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3953 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 8291

Test results coming soon (I hope)...

Details

- notification for comment with ID 3391772202 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS Python-bundle-PyPI-2025.07-GCCcore-14.3.0.eb
  • SUCCESS SciPy-bundle-2025.07-gfbf-2025b.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/25ea7470369c3209f332905d8284432d for a full test report.

boegel
boegel previously requested changes Oct 13, 2025
Comment thread easybuild/easyblocks/generic/pythonpackage.py Outdated
Comment thread easybuild/easyblocks/generic/pythonpackage.py Outdated
Comment thread easybuild/easyblocks/generic/pythonpackage.py Outdated
Comment thread easybuild/easyblocks/generic/pythonpackage.py Outdated
Comment thread easybuild/easyblocks/generic/pythonpackage.py Outdated
@Thyre
Copy link
Copy Markdown
Collaborator Author

Thyre commented Oct 13, 2025

@boegel This is the output now for --rebuild (from test report easybuilders/easybuild-easyconfigs#24081 (comment)):

[reuter1@jrc0900 ~]$ grep -rni "path configuration" /p/project1/cswmanage/reuter1/EasyBuild/jedi/apps/software/numba-cuda/0.20.0-foss-2025b-CUDA-12.9.1/easybuild/easybuild-numba-cuda-0.20.0-20251013.111451.log
6498:== 2025-10-13 11:10:22,424 pythonbundle.py:136 INFO Found path configuration file in installation directory '/p/project1/cswmanage/reuter1/EasyBuild/jedi/apps/software/numba-cuda/0.20.0-foss-2025b-CUDA-12.9.1'. Enabling $EBPYTHONPREFIXES...
8544:== 2025-10-13 11:10:24,942 pythonpackage.py:648 INFO Found path configuration file in installation directory '/p/project1/cswmanage/reuter1/EasyBuild/jedi/apps/software/numba-cuda/0.20.0-foss-2025b-CUDA-12.9.1'. Enabling $EBPYTHONPREFIXES...
8547:== 2025-10-13 11:10:24,943 pythonpackage.py:655 INFO Found path configuration file in test installation directory '/tmp/eb-s0j8ee0c/tmpsp171pw4'. Enabling $EBPYTHONPREFIXES...

Agreed that this helps. I also applied your other suggestions 👍

Copy link
Copy Markdown
Contributor

@Micket Micket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Micket Micket merged commit 41ea350 into easybuilders:develop Oct 15, 2025
22 checks passed
@boegel boegel changed the title enhance PythonPackage & PythonBundle easyblock to consider EBPYTHONPREFIXES for test installs and path configuration files enhance PythonPackage & PythonBundle easyblock to consider $EBPYTHONPREFIXES for test installs and path configuration files Oct 22, 2025
@boegel boegel removed the change label Oct 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants