Skip to content

fix detection of math library in numpy build#3520

Merged
boegel merged 2 commits intoeasybuilders:developfrom
Flamefire:20241128094035_new_pr_numpy
Dec 4, 2024
Merged

fix detection of math library in numpy build#3520
boegel merged 2 commits intoeasybuilders:developfrom
Flamefire:20241128094035_new_pr_numpy

Conversation

@Flamefire
Copy link
Copy Markdown
Contributor

@Flamefire Flamefire commented Nov 28, 2024 •
edited
Loading

(created using eb --new-pr)

NumPy detects the required math by trying to link a minimal code containing a call to log(0.).
The first try is without any libraries, which works with gcc -fno-math-errno (our optimization default)
because the call gets removed due to not having any effect. So it concludes that -lm is not required.
This then fails to detect availability of functions such as acosh which do not get removed in the same
way and so less exact replacements are used instead which e.g. fail the tests on PPC.

$MATHLIB=m makes it try -lm first and is supported until the Meson backend is used in 1.26+.

The test code used is:

double log(double);

int main(){
  log(0.);
}

The issue can be reproduced by putting it in a file test.c and running:

$ gcc test.c
# OK
$ gcc -fno-math-errno test.c
/software/.../bin/ld: /tmp/ccGF6GFx.o: in function `main':
test.c:(.text+0x20): undefined reference to `log'
collect2: error: ld returned 1 exit status

On PPC I see test failures in e.g. SciPy-bundle-2023.02-gfbf-2022b.eb & SciPy-bundle-2023.07-gfbf-2023a.eb
FAILED core/tests/test_umath.py::TestComplexFunctions::test_loss_of_precision[complex256] - AssertionError: (120, 1.78095631306534887655740488957533e-10, 5.2863423150802063925011506713427e-21, 'arcsinh')

However the issue is not limited to PPC although the fallback functions seem to be "good enough" on x86.
A minimal example I made: https://godbolt.org/z/zGYcW45qE and one that it fails to detect casinh without -lm

Thyre
Thyre reviewed Nov 28, 2024
View reviewed changes
Comment thread easybuild/easyblocks/n/numpy.py Outdated
@Flamefire @Thyre
Remove numpy_version variable again
259051b 
Co-authored-by: Jan André Reuter <[email protected]>
@Flamefire
Copy link
Copy Markdown
Contributor Author

Flamefire commented Nov 29, 2024 •
edited
Loading

Test report by @Flamefire

Overview of tested easyconfigs (in order)

  • SUCCESS SciPy-bundle-2022.05-foss-2022a.eb
  • SUCCESS SciPy-bundle-2023.02-gfbf-2022b.eb
  • SUCCESS SciPy-bundle-2023.07-gfbf-2023a.eb
  • SUCCESS SciPy-bundle-2023.11-gfbf-2023.09.eb
  • SUCCESS SciPy-bundle-2023.11-gfbf-2023b.eb
  • SUCCESS Ninja-1.12.1-GCCcore-13.3.0.eb
  • SUCCESS Meson-1.4.0-GCCcore-13.3.0.eb
  • SUCCESS Cython-3.0.10-GCCcore-13.3.0.eb
  • SUCCESS hatchling-1.24.2-GCCcore-13.3.0.eb
  • SUCCESS expat-2.6.2-GCCcore-13.3.0.eb
  • SUCCESS Autoconf-2.72-GCCcore-13.3.0.eb
  • SUCCESS scikit-build-0.17.6-GCCcore-13.3.0.eb
  • SUCCESS Automake-1.16.5-GCCcore-13.3.0.eb
  • SUCCESS Eigen-3.4.0-GCCcore-13.3.0.eb
  • SUCCESS flit-3.9.0-GCCcore-13.3.0.eb
  • SUCCESS libtool-2.4.7-GCCcore-13.3.0.eb
  • SUCCESS Catch2-2.13.10-GCCcore-13.3.0.eb
  • SUCCESS libxml2-2.12.7-GCCcore-13.3.0.eb
  • SUCCESS Autotools-20231222-GCCcore-13.3.0.eb
  • SUCCESS setuptools-rust-1.9.0-GCCcore-13.3.0.eb
  • SUCCESS libiconv-1.17-GCCcore-13.3.0.eb
  • SUCCESS patchelf-0.18.0-GCCcore-13.3.0.eb
  • SUCCESS gettext-0.22.5-GCCcore-13.3.0.eb
  • SUCCESS Rust-1.78.0-GCCcore-13.3.0.eb
  • SUCCESS git-2.45.1-GCCcore-13.3.0.eb
  • SUCCESS maturin-1.6.0-GCCcore-13.3.0.eb
  • SUCCESS ICU-75.1-GCCcore-13.3.0.eb
  • SUCCESS gzip-1.13-GCCcore-13.3.0.eb
  • SUCCESS virtualenv-20.26.2-GCCcore-13.3.0.eb
  • SUCCESS lz4-1.9.4-GCCcore-13.3.0.eb
  • SUCCESS cffi-1.16.0-GCCcore-13.3.0.eb
  • SUCCESS zstd-1.5.6-GCCcore-13.3.0.eb
  • SUCCESS cryptography-42.0.8-GCCcore-13.3.0.eb
  • SUCCESS poetry-1.8.3-GCCcore-13.3.0.eb
  • SUCCESS Python-bundle-PyPI-2024.06-GCCcore-13.3.0.eb
  • SUCCESS meson-python-0.16.0-GCCcore-13.3.0.eb
  • SUCCESS SciPy-bundle-2024.05-gfbf-2024a.eb

Build succeeded
ml24 - Linux AlmaLinux 8.7 (Stone Smilodon), POWER, 8335-GTX (power9le), 6 x NVIDIA Tesla V100-SXM2-32GB, 530.30.02, Python 3.8.13
See https://gist.github.com/Flamefire/d8a5f3d67e75cf17c7741a9df366cf97 for a full test report.

@Flamefire
Copy link
Copy Markdown
Contributor Author

Flamefire commented Nov 29, 2024 •
edited
Loading

Test report by @Flamefire

Overview of tested easyconfigs (in order)

  • SUCCESS SciPy-bundle-2022.05-foss-2022a.eb
  • SUCCESS SciPy-bundle-2023.02-gfbf-2022b.eb
  • SUCCESS SciPy-bundle-2023.07-gfbf-2023a.eb
  • SUCCESS SciPy-bundle-2023.11-gfbf-2023.09.eb
  • SUCCESS SciPy-bundle-2023.11-gfbf-2023b.eb
  • SUCCESS make-4.4.1-GCCcore-13.3.0.eb
  • SUCCESS OpenBLAS-0.3.27-GCC-13.3.0.eb
  • SUCCESS FlexiBLAS-3.4.4-GCC-13.3.0.eb
  • SUCCESS gfbf-2024a.eb
  • SUCCESS SciPy-bundle-2024.05-gfbf-2024a.eb

Build succeeded for 10 out of 10 (6 easyconfigs in total)
n1166 - Linux RHEL 8.9 (Ootpa), x86_64, Intel(R) Xeon(R) Platinum 8470 (icelake), Python 3.8.17
See https://gist.github.com/Flamefire/5c77df272384501a0632a9d7d846fa7d for a full test report.

@boegel boegel added the bug fix label Dec 4, 2024
@boegel boegel added this to the release after 4.9.4 milestone Dec 4, 2024
boegel
boegel approved these changes Dec 4, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel
Copy link
Copy Markdown
Member

boegel commented Dec 4, 2024

@boegelbot please test @ jsc-zen3
EB_ARGS="--installpath /tmp/$USER/pr3520 SciPy-bundle-2023.07-iimkl-2023a.eb SciPy-bundle-2024.05-gfbf-2024a.eb SciPy-bundle-2022.05-foss-2022a.eb"

@boegelbot
Copy link
Copy Markdown

boegelbot commented Dec 4, 2024

@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3520 EB_ARGS="--installpath /tmp/$USER/pr3520 SciPy-bundle-2023.07-iimkl-2023a.eb SciPy-bundle-2024.05-gfbf-2024a.eb SciPy-bundle-2022.05-foss-2022a.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3520 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 5379

Test results coming soon (I hope)...

Details

- notification for comment with ID 2516576397 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown

boegelbot commented Dec 4, 2024

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS SciPy-bundle-2023.07-iimkl-2023a.eb
  • SUCCESS SciPy-bundle-2024.05-gfbf-2024a.eb
  • SUCCESS SciPy-bundle-2022.05-foss-2022a.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.19
See https://gist.github.com/boegelbot/acda5aa3c76df77df10976aa8783c5df for a full test report.

boegel
boegel approved these changes Dec 4, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel boegel merged commit 559e7b9 into easybuilders:develop Dec 4, 2024
@Flamefire Flamefire deleted the 20241128094035_new_pr_numpy branch December 5, 2024 08:19
@boegel boegel modified the milestones: release after 4.9.4, 5.0.0 Mar 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@boegel boegel boegel approved these changes

+1 more reviewer

@Thyre Thyre Thyre left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

bug fix

Projects

None yet

Milestone

5.0.0

Development

Successfully merging this pull request may close these issues.

4 participants

@Flamefire @boegel @boegelbot @Thyre