Skip to content

Explicitly disable libxml2 support in LLVM easyblock if libxml2 is not available as dependency#3657

Merged
boegel merged 1 commit intoeasybuilders:5.0.xfrom
Crivella:fix-libxml2
Mar 16, 2025
Merged

Explicitly disable libxml2 support in LLVM easyblock if libxml2 is not available as dependency#3657
boegel merged 1 commit intoeasybuilders:5.0.xfrom
Crivella:fix-libxml2

Conversation

@Crivella
Copy link
Copy Markdown
Contributor

@Crivella Crivella commented Mar 7, 2025 •
edited
Loading

Attempt to fix easybuilders/easybuild-easyconfigs#22491

Avoid using system libxml2 which could cause inconsistent test result failures (or possibly install failures?)

@Crivella Crivella changed the title Disable LIBXML2 if not explicitly passed as a dependency LLVM: Explicitly disable usage of LIBXML2 if not passed as a dependency Mar 7, 2025
@boegel
Copy link
Copy Markdown
Member

boegel commented Mar 8, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="LLVM-14.0.3-GCCcore-11.3.0.eb LLVM-14.0.6-GCCcore-12.3.0-llvmlite.eb LLVM-14.0.6-GCCcore-13.2.0-llvmlite.eb LLVM-15.0.5-GCCcore-12.2.0.eb LLVM-16.0.6-GCCcore-12.3.0.eb LLVM-16.0.6-GCCcore-13.2.0.eb LLVM-18.1.8-GCCcore-13.3.0-minimal.eb --installpath /tmp/$USER/pr3657"
CORE_CNT=16
EB_BRANCH=5.0.x

@boegel boegel added bug fix EasyBuild-5.0 EasyBuild 5.0 labels Mar 8, 2025
@boegel boegel added this to the 5.0 milestone Mar 8, 2025
@boegel boegel moved this to Blockers in EasyBuild v5.0 Mar 8, 2025
@boegelbot
Copy link
Copy Markdown

boegelbot commented Mar 8, 2025

@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ "5.0.x" != 'develop' ]]; then EB_BRANCH="5.0.x" ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/"5.0.x" source init_env_easybuild_develop.sh; fi; EB_PR=3657 EB_ARGS="LLVM-14.0.3-GCCcore-11.3.0.eb LLVM-14.0.6-GCCcore-12.3.0-llvmlite.eb LLVM-14.0.6-GCCcore-13.2.0-llvmlite.eb LLVM-15.0.5-GCCcore-12.2.0.eb LLVM-16.0.6-GCCcore-12.3.0.eb LLVM-16.0.6-GCCcore-13.2.0.eb LLVM-18.1.8-GCCcore-13.3.0-minimal.eb --installpath /tmp/$USER/pr3657" EB_REPO=easybuild-easyblocks EB_BRANCH="5.0.x" /opt/software/slurm/bin/sbatch --job-name test_PR_3657 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 5903

Test results coming soon (I hope)...

Details

- notification for comment with ID 2708184298 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown

boegelbot commented Mar 8, 2025

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS LLVM-14.0.3-GCCcore-11.3.0.eb
  • SUCCESS LLVM-14.0.6-GCCcore-12.3.0-llvmlite.eb
  • SUCCESS LLVM-14.0.6-GCCcore-13.2.0-llvmlite.eb
  • SUCCESS LLVM-15.0.5-GCCcore-12.2.0.eb
  • SUCCESS LLVM-16.0.6-GCCcore-12.3.0.eb
  • SUCCESS LLVM-16.0.6-GCCcore-13.2.0.eb
  • SUCCESS LLVM-18.1.8-GCCcore-13.3.0-minimal.eb

Build succeeded for 7 out of 7 (7 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/68f5a14991b36f638fa4411fade1dd4f for a full test report.

@Thyre Thyre mentioned this pull request Mar 8, 2025
Merged
@Thyre
Copy link
Copy Markdown
Collaborator

Thyre commented Mar 8, 2025 •
edited
Loading

Hm, looks like this isn't enough to fix the tests:
See easybuilders/easybuild-easyconfigs#22489 (comment)

In the log file, I see:


-- Found LibXml2: /opt/EasyBuild/apps/software/libxml2/2.12.7-GCCcore-13.3.0/lib/libxml2.so (found version "2.12.7")

[...]

[  9%] Building CXX object lib/WindowsManifest/CMakeFiles/LLVMWindowsManifest.dir/WindowsManifestMerger.cpp.o
cd /opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm.obj.1/lib/WindowsManifest && /tmp/eb-37pzfw3f/tmpv3trla15/rpath_wrappers/gxx_wrapper/g++ -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm.obj.1/lib/WindowsManifest -I/opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm-project-14.0.6.src/llvm/lib/WindowsManifest -I/opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm.obj.1/include -I/opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm-project-14.0.6.src/llvm/include -isystem /opt/EasyBuild/apps/software/libxml2/2.12.7-GCCcore-13.3.0/include/libxml2 -isystem /opt/EasyBuild/apps/software/zlib/1.3.1-GCCcore-13.3.0/include -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -std=gnu++11 -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-pessimizing-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wmisleading-indentation -fdata-sections -O3 -DNDEBUG -std=c++14 -fPIC  -fno-exceptions -MD -MT lib/WindowsManifest/CMakeFiles/LLVMWindowsManifest.dir/WindowsManifestMerger.cpp.o -MF CMakeFiles/LLVMWindowsManifest.dir/WindowsManifestMerger.cpp.o.d -o CMakeFiles/LLVMWindowsManifest.dir/WindowsManifestMerger.cpp.o -c /opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm-project-14.0.6.src/llvm/lib/WindowsManifest/WindowsManifestMerger.cpp

[...]


******************** TEST 'LLVM :: tools/llvm-mt/big_merge.test' FAILED ********************
Script:
--
: 'RUN: at line 4';   /opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm.obj.1/bin/llvm-mt /manifest /opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm-project-14.0.6.src/llvm/test/tools/llvm-mt/Inputs/trust_info.manifest    /manifest /opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm-project-14.0.6.src/llvm/test/tools/llvm-mt/Inputs/assembly_identity.manifest    /manifest /opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm-project-14.0.6.src/llvm/test/tools/llvm-mt/Inputs/trust_and_identity.manifest    /manifest /opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm-project-14.0.6.src/llvm/test/tools/llvm-mt/Inputs/compatibility.manifest    /manifest  /opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm-project-14.0.6.src/llvm/test/tools/llvm-mt/Inputs/windows_settings.manifest /out:/opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm.obj.1/test/tools/llvm-mt/Output/big_merge.test.tmp
: 'RUN: at line 9';   /opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm.obj.1/bin/FileCheck /opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm-project-14.0.6.src/llvm/test/tools/llvm-mt/big_merge.test -input-file=/opt/EasyBuild/apps/build/LLVM/14.0.6/GCCcore-13.3.0-llvmlite/llvm.obj.1/test/tools/llvm-mt/Output/big_merge.test.tmp
--
Exit Code: 1

Command Output (stderr):
--
llvm-mt: error: no libxml2

--

So it seems to build with libxml2 from EasyBuild, but it is not found later on.

libxml2 is loaded because we have git as a build dependency: git -> gettext -> libxml2

@boegel
Copy link
Copy Markdown
Member

boegel commented Mar 8, 2025

@Thyre What if you add libxml2 as a direct dependency to LLVM, does that change anything?

@Thyre
Copy link
Copy Markdown
Collaborator

Thyre commented Mar 8, 2025 •
edited
Loading

@Thyre What if you add libxml2 as a direct dependency to LLVM, does that change anything?

I added a direct dependency, see easybuilders/easybuild-easyconfigs@840f2b0. My notebook, running Zorin OS 17, still failed to pass the tests. Another system, with Arch Linux, was able to pass the tests successfully.

I'll try to debug this further...

@Thyre
Copy link
Copy Markdown
Collaborator

Thyre commented Mar 9, 2025 •
edited
Loading

llvm-mt was using the libLLVM-14.so from an incomplete installation in the software folder which failed before, not the build folder. This installation was missing enabling libxml2 for the build.
I restarted the build process with this installation removed.

Yeah, that was the issue, see: easybuilders/easybuild-easyconfigs#22489 (comment)
I'll remove the explicit dependency and try again.

Edit: Yeah, the incomplete installation was causing these issues.

@boegel
Copy link
Copy Markdown
Member

boegel commented Mar 13, 2025

@Thyre So, what's the conclusion here?

We need both this + add libxml2 as direct dependency?

@Thyre
Copy link
Copy Markdown
Collaborator

Thyre commented Mar 13, 2025

Oh sorry, forgot to add a comment with the results. We do not need to add libxml2 from what I've gathered (see easybuilders/easybuild-easyconfigs#22489 (comment)), though I haven't tested it with rpathing disabled. I'll start another test build with this real quick.

@Thyre
Copy link
Copy Markdown
Collaborator

Thyre commented Mar 13, 2025

Same result with rpathing disabled, we do not need libxml2 as dependency 😄

@Crivella Crivella marked this pull request as ready for review March 14, 2025 09:06
@Crivella
Copy link
Copy Markdown
Contributor Author

Crivella commented Mar 14, 2025

Edit: Yeah, the incomplete installation was causing these issues.

@Thyre Yeah also had similar problem when running with --rebuild as the RPATH to the install dir is being hardcoded during the build if the install dir is not deleted before it it will pick up symbols from there instead of the build directory

Not sure if this has been a problem also for other software packages build with rpath @boegel .
One thing we might try is to append instead of prepending the install dir rpath, but that would need some extensive testing to make sure we don't end up picking up wrong libraries in some dependecies combination?

@boegel boegel changed the title LLVM: Explicitly disable usage of LIBXML2 if not passed as a dependency Explicitly disable libxml2 support in LLVM easyblock if libxml2 is not available as dependency Mar 16, 2025
@boegel
Copy link
Copy Markdown
Member

boegel commented Mar 16, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="LLVM-14.0.6-GCCcore-12.3.0-llvmlite.eb LLVM-16.0.6-GCCcore-13.2.0.eb --installpath /tmp/$USER/pr3657"
CORE_CNT=16
EB_BRANCH=5.0.x

@boegelbot
Copy link
Copy Markdown

boegelbot commented Mar 16, 2025

@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ "5.0.x" != 'develop' ]]; then EB_BRANCH="5.0.x" ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/"5.0.x" source init_env_easybuild_develop.sh; fi; EB_PR=3657 EB_ARGS="LLVM-14.0.6-GCCcore-12.3.0-llvmlite.eb LLVM-16.0.6-GCCcore-13.2.0.eb --installpath /tmp/$USER/pr3657" EB_REPO=easybuild-easyblocks EB_BRANCH="5.0.x" /opt/software/slurm/bin/sbatch --job-name test_PR_3657 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 5967

Test results coming soon (I hope)...

Details

- notification for comment with ID 2727529627 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown

boegelbot commented Mar 16, 2025

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS LLVM-14.0.6-GCCcore-12.3.0-llvmlite.eb
  • SUCCESS LLVM-16.0.6-GCCcore-13.2.0.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen3c2.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/6ef9c6c0d328f530f0920d8f02a6577c for a full test report.

@boegel boegel merged commit 2451eeb into easybuilders:5.0.x Mar 16, 2025
13 checks passed
@boegel boegel mentioned this pull request Mar 16, 2025
Closed
@boegel boegel removed this from EasyBuild v5.0 Mar 16, 2025
@Crivella Crivella deleted the fix-libxml2 branch March 17, 2025 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@boegel boegel boegel approved these changes

Assignees

No one assigned

Labels

bug fix EasyBuild-5.0 EasyBuild 5.0

Projects

None yet

Milestone

5.0.0

Development

Successfully merging this pull request may close these issues.

4 participants

@Crivella @boegel @boegelbot @Thyre