Skip to content

fix search_file for path configuration files in PythonBundle and PythonPackage#3975

Merged
Micket merged 2 commits intoeasybuilders:developfrom
Thyre:20251024233543_new_pr_pythonbundle
Nov 25, 2025
Merged

fix search_file for path configuration files in PythonBundle and PythonPackage#3975
Micket merged 2 commits intoeasybuilders:developfrom
Thyre:20251024233543_new_pr_pythonbundle

Conversation

@Thyre
Copy link
Copy Markdown
Collaborator

@Thyre Thyre commented Oct 24, 2025 •
edited
Loading

(created using eb --new-pr)

search_file always returns at least a tuple of empty arrays, which always passes the if condition used in #3953. This enabled EBPYTHONPREFIXES for basically all Python packages, which is not intended.

To fix this, use the returned hits array to check if we found a file. This should result in the intended behavior.
We just need to check if this breaks numba-cuda again.

@Thyre Thyre added this to the next release (5.2.0?) milestone Oct 24, 2025
@Thyre Thyre added the bug fix label Oct 24, 2025
@Thyre Thyre marked this pull request as draft October 24, 2025 21:47
@Thyre
Copy link
Copy Markdown
Collaborator Author

Thyre commented Oct 24, 2025

@boegelbot please test @ jsc-zen3-a100
EB_ARGS="numba-cuda-0.20.0-foss-2025b-CUDA-12.9.1.eb --installpath=/tmp/$USER/ebpr-3975"

@boegelbot
Copy link
Copy Markdown

boegelbot commented Oct 24, 2025

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3975 EB_ARGS="numba-cuda-0.20.0-foss-2025b-CUDA-12.9.1.eb --installpath=/tmp/$USER/ebpr-3975" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3975 --ntasks=8 --partition=jsczen3g --gres=gpu:1 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 8542

Test results coming soon (I hope)...

Details

- notification for comment with ID 3445072321 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@Thyre Thyre marked this pull request as ready for review October 24, 2025 22:13
@Thyre
Copy link
Copy Markdown
Collaborator Author

Thyre commented Oct 24, 2025 •
edited
Loading

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS numba-cuda-0.20.0-foss-2025b-CUDA-12.9.1.eb

Build succeeded (with --ignore-test-failure) for 1 out of 1 (1 easyconfigs in total)
ZAM054 - Linux Zorin OS 18, x86_64, 12th Gen Intel(R) Core(TM) i7-1260P (skylake), 1 x NVIDIA NVIDIA GeForce MX550, 580.95.05, Python 3.12.3
See https://gist.github.com/Thyre/df154d3ff3e95a7176ff529306aa1565 for a full test report.

--ignore-test-failure to ignore the single test that fails due to insufficient VRAM on my system:

=================================================================================================================== short test summary info ===================================================================================================================
FAILED doc_examples/test_random.py::TestRandom::test_ex_3d_grid - numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY
======================================================================================= 1 failed, 1650 passed, 69 skipped, 11 xfailed, 18 warnings in 153.47s (0:02:33) =======================================================================================

@boegelbot
Copy link
Copy Markdown

boegelbot commented Oct 24, 2025

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS numba-cuda-0.20.0-foss-2025b-CUDA-12.9.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen3g1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), 1 x NVIDIA NVIDIA A100 80GB PCIe, 580.95.05, Python 3.9.21
See https://gist.github.com/boegelbot/147c1aa602843e9823c960def192f100 for a full test report.

@Thyre
Copy link
Copy Markdown
Collaborator Author

Thyre commented Oct 24, 2025

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS astropy-7.1.0-gfbf-2025b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
Linux - Linux Arch Linux UNKNOWN, x86_64, AMD Ryzen 7 7800X3D 8-Core Processor (zen4), 1 x AMD Navi 48 [Radeon RX 9070/9070 XT/9070 GRE] (device id: 0x7550, gfx: gfx1201, driver: Linuxversion6.17.4-arch2-1(linux@archlinux)(gcc(GCC)15.2.120250813,GNUld(GNUBinutils)2.45.0)#1SMPPREEMPT_DYNAMICSun,19Oct202519:21:18+0000), 1 x AMD Raphael (device id: 0x164e, gfx: gfx1036, driver: Linuxversion6.17.4-arch2-1(linux@archlinux)(gcc(GCC)15.2.120250813,GNUld(GNUBinutils)2.45.0)#1SMPPREEMPT_DYNAMICSun,19Oct202519:21:18+0000), Python 3.13.7
See https://gist.github.com/Thyre/c756ef808dedf3e54cb8ffd925ae6fce for a full test report.

@Thyre
Copy link
Copy Markdown
Collaborator Author

Thyre commented Oct 24, 2025 •
edited
Loading

Checking e.g. astropy, EBPYTHONPREFIXES now doesn't show up anymore, as it should be:

Before:

>> running shell command:
        export PYTHONPATH=/tmp/eb-tve7dwhm/tmpeb0fct0h/lib/python3.13/site-packages:$PYTHONPATH && export EBPYTHONPREFIXES=/tmp/eb-tve7dwhm/tmpeb0fct0h:$EBPYTHONPREFIXES && export PATH=/tmp/eb-tve7dwhm/tmpeb0fct0h/bin:$PATH &&
/data/EasyBuild-develop/software/Python/3.13.5-GCCcore-14.3.0/bin/python -m pip install --prefix=/tmp/eb-tve7dwhm/tmpeb0fct0h  --no-deps --ignore-installed --no-build-isolation .
        [started at: 2025-10-25 00:25:11]
        [working dir: /data/EasyBuild-develop/build/astropy/7.1.0/gfbf-2025b/astropy/astropy-7.1.0]
        [output and state saved to /tmp/eb-tve7dwhm/run-shell-cmd-output/export-7la1qk7n]
  >> command completed: exit 0, ran in 00h00m43s
  >> running shell command:
        export PYTHONPATH=/tmp/eb-tve7dwhm/tmpeb0fct0h/lib/python3.13/site-packages:$PYTHONPATH && export EBPYTHONPREFIXES=/tmp/eb-tve7dwhm/tmpeb0fct0h:$EBPYTHONPREFIXES && export PATH=/tmp/eb-tve7dwhm/tmpeb0fct0h/bin:$PATH &&   cd
$EB_PYTHONPACKAGE_TEST_INSTALLDIR && pytest -n 8 -k "not test_delay_doc_updates and not test_datetime_timedelta_roundtrip and not remote_data and not test_write_jsviewer_local and not test_datetime_timedelta_sum and not test_imports" --pyargs astropy
        [started at: 2025-10-25 00:25:54]
        [working dir: /data/EasyBuild-develop/build/astropy/7.1.0/gfbf-2025b/astropy/astropy-7.1.0]
        [output and state saved to /tmp/eb-tve7dwhm/run-shell-cmd-output/export-tii5rweu]

Now:

>> running shell command:
        export PYTHONPATH=/tmp/eb-rnuo79c7/tmp8zzuh0ff/lib/python3.13/site-packages:$PYTHONPATH && export PATH=/tmp/eb-rnuo79c7/tmp8zzuh0ff/bin:$PATH &&   cd $EB_PYTHONPACKAGE_TEST_INSTALLDIR && pytest -n 8 -k "not test_delay_doc_updates and not
test_datetime_timedelta_roundtrip and not remote_data and not test_write_jsviewer_local and not test_datetime_timedelta_sum and not test_imports" --pyargs astropy
        [started at: 2025-10-25 00:22:33]
        [working dir: /data/EasyBuild-develop/build/astropy/7.1.0/gfbf-2025b/astropy/astropy-7.1.0]
        [output and state saved to /tmp/eb-rnuo79c7/run-shell-cmd-output/export-yabnsvq7]
  >> command completed: exit 0, ran in 00h00m44s

@Thyre Thyre changed the title fix search_file for path configuration files in PythonBundle and PythonPackage fix search_file for path configuration files in PythonBundle and PythonPackage Oct 24, 2025
@Thyre
Copy link
Copy Markdown
Collaborator Author

Thyre commented Nov 17, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="--installpath=/tmp/$USER/ebpr-3975 SciPy-bundle-2025.07-gfbf-2025b.eb Python-bundle-PyPI-2025.07-GCCcore-14.3.0.eb"

@boegelbot
Copy link
Copy Markdown

boegelbot commented Nov 17, 2025

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3975 EB_ARGS="--installpath=/tmp/$USER/ebpr-3975 SciPy-bundle-2025.07-gfbf-2025b.eb Python-bundle-PyPI-2025.07-GCCcore-14.3.0.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3975 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 8796

Test results coming soon (I hope)...

Details

- notification for comment with ID 3541069450 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown

boegelbot commented Nov 17, 2025

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS Python-bundle-PyPI-2025.07-GCCcore-14.3.0.eb
  • SUCCESS SciPy-bundle-2025.07-gfbf-2025b.eb

Build succeeded for 2 out of 2 (total: 3 hours 13 mins 41 secs) (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/408f94ef98929e543feeed518f456eb5 for a full test report.

@boegel boegel changed the title fix search_file for path configuration files in PythonBundle and PythonPackage fix search_file for path configuration files in PythonBundle and PythonPackage Nov 19, 2025
@Thyre Thyre mentioned this pull request Nov 24, 2025
Merged
Micket
Micket approved these changes Nov 25, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@Micket Micket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Micket
Copy link
Copy Markdown
Contributor

Micket commented Nov 25, 2025

I've also used this a bunch now testing out numba-cuda (and others) with no issues (caused by this, numba-cuda fails due to some unrelated cuda issue, but the rest of the pythonpackages built just fine).

@Micket Micket merged commit 0f5097a into easybuilders:develop Nov 25, 2025
22 checks passed
@Micket Micket mentioned this pull request Nov 29, 2025
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Micket Micket Micket approved these changes

Assignees

No one assigned

Labels

bug fix priority:high

Projects

None yet

Milestone

5.2.0

Development

Successfully merging this pull request may close these issues.

3 participants

@Thyre @boegelbot @Micket