Skip to content

fix creation of symlink for libraries in tbb easyblock#2927

Merged
akesandgren merged 4 commits intoeasybuilders:developfrom
Flamefire:tbb-fix
Sep 14, 2023
Merged

fix creation of symlink for libraries in tbb easyblock#2927
akesandgren merged 4 commits intoeasybuilders:developfrom
Flamefire:tbb-fix

Conversation

@Flamefire
Copy link
Copy Markdown
Contributor

@Flamefire Flamefire commented May 10, 2023 •
edited
Loading

PR #2539 introduced a suspicious os.path.join(libpath) which was likely intended to be os.path.join(self.installdir, libpath) but works due to a possible(!) chdir when TBB is installed via the system toolchain.
Fix this by making libpath always absolute and simplify the code a bit.

@mboisson Please take a look at this instead of ComputeCanada@f34b32b

@Flamefire
Fix symlink in TBB easyblock
5ed9f93 
PR easybuilders#2539 introduced a suspicious `os.path.join(libpath)` which was
likely intended to be `os.path.join(self.installdir, libpath)` but works
due to a possible(!) `chdir` when TBB is installed via the system
toolchain.
Fix this by making `libpath` always absolute and simplify the code a bit.
@boegel boegel added the bug fix label May 10, 2023
@boegel boegel added this to the next release (4.7.2) milestone May 10, 2023
@boegel boegel changed the title Fix symlink in TBB easyblock fix creation of symlink for libraries in TBB easyblock May 10, 2023
@Flamefire
Copy link
Copy Markdown
Contributor Author

Flamefire commented May 11, 2023 •
edited
Loading

Test report by @Flamefire

Overview of tested easyconfigs (in order)

  • SUCCESS tbb-2018_U5-GCCcore-7.3.0.eb
  • SUCCESS tbb-2019_U4-GCCcore-8.2.0.eb
  • SUCCESS tbb-2020.3-GCCcore-10.2.0.eb
  • SUCCESS tbb-2021.5.0-GCCcore-11.3.0.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total)
taurusi6605.taurus.hrsk.tu-dresden.de - Linux RHEL 7.9, x86_64, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (haswell), Python 2.7.5
See https://gist.github.com/Flamefire/eb53b8daf570cb59930233cf2f0bcd18 for a full test report.

From the logs:

== 2023-05-11 12:42:34,576 tbb.py:197 DEBUG libpath: /beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/tbb/2018_U5-GCCcore-7.3.0/build/linux_intel64_gcc_cc7.3.0_libc2.17_kernel3.10.0_release
== 2023-05-11 12:43:57,264 tbb.py:197 DEBUG libpath: /beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/tbb/2019_U4-GCCcore-8.2.0/build/linux_intel64_gcc_cc8.2.0_libc2.17_kernel3.10.0_release
== 2023-05-11 12:45:18,725 tbb.py:197 DEBUG libpath: /beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/tbb/2020.3-GCCcore-10.2.0/build/linux_intel64_gcc_cc10.2.0_libc2.17_kernel3.10.0_release

$ ls -l /beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/tbb/2018_U5-GCCcore-7.3.0/build/linux_intel64_gcc_cc7.3.0_libc2.17_kernel3.10.0_release
lrwxr-xr-x 1 s3248973 zihforschung 6 11. Mai 12:42 /beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/tbb/2018_U5-GCCcore-7.3.0/build/linux_intel64_gcc_cc7.3.0_libc2.17_kernel3.10.0_release -> ../lib

While previously it was:

 ls -l /beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/tbb/2018_U5-GCCcore-7.3.0/build/linux_intel64_gcc_cc7.3.0_libc2.17_kernel3.10.0_release
lrwxr-xr-x 1 s3248973 zihforschung 107 11. Mai 11:06 /beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/tbb/2018_U5-GCCcore-7.3.0/build/linux_intel64_gcc_cc7.3.0_libc2.17_kernel3.10.0_release -> ../../../../../../../../../ws/1/s3248973-EasyBuild/easybuild-haswell/software/tbb/2018_U5-GCCcore-7.3.0/lib

@boegel boegel modified the milestones: 4.7.3, release after 4.7.3 Jul 5, 2023
akesandgren
akesandgren reviewed Sep 13, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@akesandgren akesandgren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does fix incorrect symlinks being generated for newer (20xx) versions, but the code for "if self.toolchain.is_system_toolchain():" both was and still is broken due to multiple layers of relative synlinking.

@Flamefire
Copy link
Copy Markdown
Contributor Author

Flamefire commented Sep 13, 2023

This does fix incorrect symlinks being generated for newer (20xx) versions, but the code for "if self.toolchain.is_system_toolchain():" both was and still is broken due to multiple layers of relative synlinking.

I see, we have

        # applications go looking into tbb/lib so we move what's in there to tbb/libs
        shutil.move(install_tbb_lib_path, os.path.join(self.installdir, 'tbb', 'libs'))
        # And add a symlink of the library folder to tbb/lib
        symlink(libpath, install_tbb_lib_path)

        root_lib_path = os.path.join(self.installdir, 'lib')
        move_file(libpath, root_lib_path)
        # Create a relative symlink at the original location as-if we didn't move it.
        # Note that the path must be relative from the folder where the symlink will be!
        symlink(os.path.relpath(root_lib_path, os.path.dirname(libpath)), libpath, use_abspath_source=False)

libpath is an actual folder, install_tbb_lib_path an absolute symlink to it. We move libpath and create a relative symlink to it

Now install_tbb_lib_path points to the relative symlink but is itself absolute, so shouldn't this work? Where do the multiple layers of relative symlinks come from?

@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Sep 14, 2023

From installdir of tbb/4.4.x (based under /home/a/ake/easybuild-amd64_ubuntu2004_bdw/software):

tbb -> compilers_and_libraries/linux/tbb/
compilers_and_libraries -> compilers_and_libraries_2016/
compilers_and_libraries_2016/linux/tbb -> ../../compilers_and_libraries_2016.3.210/linux/tbb/
compilers_and_libraries_2016.3.210/linux/tbb/lib -> /home/a/ake/easybuild-amd64_ubuntu2004_bdw/software/tbb/4.4.4.210/tbb/libs/intel64/gcc4.4
tbb/libs/intel64 -> ./intel64_lin/
tbb/libs/intel64_lin/gcc4.4 -> ../../../lib

But since that last one is really
compilers_and_libraries_2016.3.210/linux/tbb/libs/intel64_lin
the ../../../lib ends up as really being
compilers_and_libraries_2016.3.210/linux/lib
which doesn't exist.

@Flamefire
Copy link
Copy Markdown
Contributor Author

Flamefire commented Sep 14, 2023

@akesandgren I updated the easyblock to create the symlink with the resolved path.

I expect that to change tbb/libs/intel64_lin/gcc4.4 -> ../../../lib to tbb/libs/intel64_lin/gcc4.4 -> ../../../../../lib which should then work.

I can't test/debug this though as I can't find the 4.4 archive.

@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Sep 14, 2023

Test report by @akesandgren

Overview of tested easyconfigs (in order)

  • SUCCESS tbb-4.4.4.210.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
b-an02.hpc2n.umu.se - Linux Ubuntu 20.04, x86_64, Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, Python 3.8.10
See https://gist.github.com/akesandgren/fb6554572898944e2b388990f54475a0 for a full test report.

@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Sep 14, 2023

Test report by @akesandgren

Overview of tested easyconfigs (in order)

  • SUCCESS tbb-2017.2.132.eb
  • SUCCESS tbb-2019_U9-GCCcore-8.3.0.eb
  • SUCCESS tbb-2020.3-GCCcore-10.3.0.eb
  • SUCCESS tbb-2021.5.0-GCCcore-11.3.0.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total)
b-an02.hpc2n.umu.se - Linux Ubuntu 20.04, x86_64, Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, Python 3.8.10
See https://gist.github.com/akesandgren/101f823a9b91de63d6517601a9e490f8 for a full test report.

akesandgren
akesandgren approved these changes Sep 14, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@akesandgren akesandgren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Sep 14, 2023

Going in, thanks @Flamefire!

2 similar comments
@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Sep 14, 2023

Going in, thanks @Flamefire!

@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Sep 14, 2023

Going in, thanks @Flamefire!

@akesandgren akesandgren merged commit 6a421d5 into easybuilders:develop Sep 14, 2023
@Flamefire
Copy link
Copy Markdown
Contributor Author

Flamefire commented Sep 14, 2023

Test report by @akesandgren

Overview of tested easyconfigs (in order)

* **SUCCESS** _tbb-4.4.4.210.eb_

So I assume the symlinks are good there now?

@Flamefire Flamefire deleted the tbb-fix branch September 14, 2023 11:35
@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Sep 14, 2023

As far as I can tell at least.

@boegel boegel changed the title fix creation of symlink for libraries in TBB easyblock fix creation of symlink for libraries in tbb easyblock Oct 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@akesandgren akesandgren akesandgren approved these changes

Assignees

No one assigned

Labels

bug fix

Projects

None yet

Milestone

4.8.2

Development

Successfully merging this pull request may close these issues.

3 participants

@Flamefire @akesandgren @boegel