Skip to content

fix recent Arrow easyconfigs to not use system libraries + use PythonPackage easyblock for pyarrow#24283

Merged
Micket merged 3 commits intoeasybuilders:developfrom
Flamefire:20251016162116_new_pr_Arrow1100
Oct 17, 2025
Merged

fix recent Arrow easyconfigs to not use system libraries + use PythonPackage easyblock for pyarrow#24283
Micket merged 3 commits intoeasybuilders:developfrom
Flamefire:20251016162116_new_pr_Arrow1100

Conversation

@Flamefire
Copy link
Copy Markdown
Contributor

@Flamefire Flamefire commented Oct 16, 2025 •
edited
Loading

(created using eb --new-pr)

The raw pip install misses important env variables causing failures in some environments.

"PYARROW_CMAKE_OPTIONS='-DZSTD_LIB=$EBROOTZSTD/lib/libzstd.%s ' " % SHLIB_EXT

uses single quotes on the commandline so $EBROOTZSTD is not resolved causing it to potentially use a system zstd

@Thyre Thyre added 2023a 2023b 2024a issues & PRs related to 2024a common toolchains 2022b bug fix and removed change labels Oct 16, 2025
@Flamefire
Copy link
Copy Markdown
Contributor Author

Flamefire commented Oct 16, 2025

Test report by @Flamefire
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
c144 - Linux AlmaLinux 9.4, x86_64, AMD EPYC 9334 32-Core Processor (zen4), 4 x NVIDIA NVIDIA H100, 560.35.03, Python 3.9.18
See https://gist.github.com/Flamefire/df944aca32d8a9c2de08c7c6f2affcea for a full test report.

Micket
Micket previously approved these changes Oct 17, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@Micket Micket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Micket
Micket approved these changes Oct 17, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@Micket Micket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm again

@Micket
Copy link
Copy Markdown
Contributor

Micket commented Oct 17, 2025

@boegelbot please test @ jsc-zen3

@boegelbot
Copy link
Copy Markdown
Collaborator

boegelbot commented Oct 17, 2025

@Micket: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=24283 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_24283 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 8447

Test results coming soon (I hope)...

Details

- notification for comment with ID 3414449165 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@Micket Micket added this to the next release (5.2.0?) milestone Oct 17, 2025
@Flamefire
Copy link
Copy Markdown
Contributor Author

Flamefire commented Oct 17, 2025

Test report by @Flamefire
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
c144 - Linux AlmaLinux 9.4, x86_64, AMD EPYC 9334 32-Core Processor (zen4), 4 x NVIDIA NVIDIA H100, 560.35.03, Python 3.9.18
See https://gist.github.com/Flamefire/4b49794f69ad16564d09a7661b2a2ef9 for a full test report.

@boegelbot
Copy link
Copy Markdown
Collaborator

boegelbot commented Oct 17, 2025

Test report by @boegelbot
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/07238a62ae910c5976c341242da0b1da for a full test report.

@Flamefire
Copy link
Copy Markdown
Contributor Author

Flamefire commented Oct 17, 2025

Test report by @Flamefire
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
i8019 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 7352 24-Core Processor (zen2), 8 x NVIDIA NVIDIA A100-SXM4-40GB, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/aa6f850747af4835bc8930259bc76a72 for a full test report.

@Micket Micket merged commit 107d1c7 into easybuilders:develop Oct 17, 2025
8 checks passed
@Flamefire Flamefire deleted the 20251016162116_new_pr_Arrow1100 branch October 17, 2025 11:48
@boegel boegel changed the title Fix Arrow 1x ECs to not use system libraries and use PythonPackage for pyarrow fix recent Arrow easyconfigs to not use system libraries + use PythonPackage easyblock for pyarrow Dec 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Micket Micket Micket approved these changes

Assignees

No one assigned

Labels

2022b 2023a 2023b 2024a issues & PRs related to 2024a common toolchains bug fix change

Projects

None yet

Milestone

5.2.0

Development

Successfully merging this pull request may close these issues.

4 participants

@Flamefire @Micket @boegelbot @Thyre