Skip to content

use mimalloc instead of jemalloc for Arrow 17.0.0#22858

Merged
Micket merged 1 commit intoeasybuilders:developfrom
smoors:20250508152842_new_pr_Arrow1700
May 8, 2025
Merged

use mimalloc instead of jemalloc for Arrow 17.0.0#22858
Micket merged 1 commit intoeasybuilders:developfrom
smoors:20250508152842_new_pr_Arrow1700

Conversation

@smoors
Copy link
Copy Markdown
Contributor

@smoors smoors commented May 8, 2025 •
edited
Loading

(created using eb --new-pr)

motivation:

  • without mimalloc the build fails for Arrow versions < 19 on zen5 (see error below)
  • mimalloc is now the recommended allocator for Arrow, see https://lists.apache.org/thread/dts9ggvkthczfpmd25wrz449mxod76o2
    • less variability between platforms
    • mimalloc generally has a nicer, more consistent API and is easier to
      work with (in particular, jemalloc's configuration scheme is slightly
      abtruse)
    • potentially better performance, or at least not significantly worse,
      than the statu quo
  • mimalloc is now set as the default allocator, see https://github.com/apache/arrow/pull/40875/files

error on zen5:

realloc failed ! 
make[3]: *** [Makefile:383: doc/jemalloc.html] Segmentation fault (core dumped)

CMake Error at /tmp/Arrow/17.0.0/gfbf-2024a/easybuild_obj/jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-install-RELEASE.cmake:47 (message):
  Stopping after outputting logs.


make[2]: *** [CMakeFiles/jemalloc_ep.dir/build.make:107: jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-install] Error 1 
make[2]: Leaving directory '/tmp/Arrow/17.0.0/gfbf-2024a/easybuild_obj'
make[1]: *** [CMakeFiles/Makefile2:887: CMakeFiles/jemalloc_ep.dir/all] Error 2 
make[1]: *** Waiting for unfinished jobs....

@github-actions github-actions Bot added the change label May 8, 2025
@smoors
Copy link
Copy Markdown
Contributor Author

smoors commented May 8, 2025

Test report by @smoors
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
node703.hydra.os - Linux Rocky Linux 9.5, x86_64, AMD EPYC 9535 64-Core Processor, Python 3.9.21
See https://gist.github.com/smoors/059c48b37ccc6e6b22f9a296fdd57942 for a full test report.

@smoors
Copy link
Copy Markdown
Contributor Author

smoors commented May 8, 2025

@boegelbot please test @ jsc-zen3

@boegelbot
Copy link
Copy Markdown
Collaborator

boegelbot commented May 8, 2025

@smoors: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=22858 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_22858 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 6314

Test results coming soon (I hope)...

Details

- notification for comment with ID 2863192652 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown
Collaborator

boegelbot commented May 8, 2025

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/ad91f139a405421123bcbf4a04a17cd9 for a full test report.

Micket
Micket approved these changes May 8, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@Micket Micket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Micket Micket added this to the release after 5.0.1 milestone May 8, 2025
@Micket Micket merged commit e932962 into easybuilders:develop May 8, 2025
8 checks passed
@smoors smoors deleted the 20250508152842_new_pr_Arrow1700 branch May 8, 2025 19:44
@boegel boegel changed the title use mimalloc instead of jemalloc for Arrow-17.0.0 use mimalloc instead of jemalloc for Arrow 17.0.0 May 26, 2025
@boegel boegel added the bug fix label May 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Micket Micket Micket approved these changes

Assignees

No one assigned

Labels

bug fix change

Projects

None yet

Milestone

5.1.0

Development

Successfully merging this pull request may close these issues.

4 participants

@smoors @boegelbot @Micket @boegel