Skip to content

tweak CMakeMake easyblock apply CMake policy to select python based on $PATH order, instead of highest Python version#3088

Merged
boegel merged 1 commit intoeasybuilders:developfrom
casparvl:set_cmake_python_selection_policy
Jan 26, 2024
Merged

tweak CMakeMake easyblock apply CMake policy to select python based on $PATH order, instead of highest Python version#3088
boegel merged 1 commit intoeasybuilders:developfrom
casparvl:set_cmake_python_selection_policy

Conversation

@casparvl
Copy link
Copy Markdown
Contributor

@casparvl casparvl commented Jan 23, 2024

Set policy so that the python is selected based on location, i.e. the order in PATH, rather than the newest version (which might be an OS python).

This fixes issues like seen in e.g. EESSI/software-layer#370 (comment)

Set policy so that the python is selected based on location, i.e. the…
5a46079 
… order in PATH, rather than the newest version (which might be an OS python).
@casparvl casparvl mentioned this pull request Jan 23, 2024
Closed
@casparvl
Copy link
Copy Markdown
Contributor Author

casparvl commented Jan 23, 2024

With this PR, testing on casacore (where I undid the previous fix of specifying the Python exectuable as a configopt), I got:

-- PYTHON3_EXECUTABLE ......... = /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.3-GCCcore-12.3.0/bin/python3.11
-- PYTHON3_LIBRARIES .......... = /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.3-GCCcore-12.3.0/lib/libpython3
.11.so
-- PYTHON3_NUMPY_INCLUDE_DIRS . = /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3
.11/site-packages/numpy/core/include

and the build succesfully completes. So, looks like this does what it should do.

@casparvl
Copy link
Copy Markdown
Contributor Author

casparvl commented Jan 26, 2024

Test report by @casparvl

Overview of tested easyconfigs (in order)

  • SUCCESS pypmt-1.1.0-gfbf-2023a.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
gcn6.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/casparvl/d444c10fe10d9840a71827843facc53f for a full test report.

@casparvl
Copy link
Copy Markdown
Contributor Author

casparvl commented Jan 26, 2024

To check if the correct Python was selected, I checked the build log for pypmt:

-- Found Python3: /sw/arch/RHEL8/EB_production/2023/software/Python/3.11.3-GCCcore-12.3.0/bin/python3.11 (found version "3.11.3") found components: Interpreter Development Development.Module Development.Embed

As expected.

@casparvl
Copy link
Copy Markdown
Contributor Author

casparvl commented Jan 26, 2024

Test report by @casparvl

Overview of tested easyconfigs (in order)

  • SUCCESS libcint-5.4.0-gfbf-2023a.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
gcn6.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/casparvl/d0b2adea1fdae0696bd0e4086938ce85 for a full test report.

@casparvl
Copy link
Copy Markdown
Contributor Author

casparvl commented Jan 26, 2024

Likewise, correctly located for libcint:

-- Found PythonInterp: /scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/bin/python (found version "3.11.3")
-- Found python  /scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/bin/python

@casparvl
Copy link
Copy Markdown
Contributor Author

casparvl commented Jan 26, 2024 •
edited
Loading

Just to try a number of builds of a package that uses CMakeMake and Python...

@boegelbot please test @ generoso
EB_ARGS="Arrow-6.0.0-foss-2021a.eb Arrow-6.0.0-foss-2021b.eb Arrow-8.0.0-foss-2022a.eb Arrow-11.0.0-gfbf-2022b.eb Arrow-14.0.1-gfbf-2023a.eb --include-easyblocks-from-pr=3088"

@casparvl
Copy link
Copy Markdown
Contributor Author

casparvl commented Jan 26, 2024

Test report by @casparvl

Overview of tested easyconfigs (in order)

  • SUCCESS OpenMM-8.0.0-foss-2023a.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
gcn6.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/casparvl/5bf1e3d1b6def87f88abfeee804c2203 for a full test report.

@casparvl
Copy link
Copy Markdown
Contributor Author

casparvl commented Jan 26, 2024

OpenMM configure also looks as it should be:

-- Found PythonInterp: /scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/bin/python (found version "3.11.3")
CMake Warning (dev) at CMakeLists.txt:21 (PROJECT):
  cmake_minimum_required() should be called prior to this top-level project()
  call.  Please see the cmake-commands(7) manual for usage documentation of
  both commands.

@boegelbot
Copy link
Copy Markdown

boegelbot commented Jan 26, 2024

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=3088 EB_ARGS="Arrow-6.0.0-foss-2021a.eb Arrow-6.0.0-foss-2021b.eb Arrow-8.0.0-foss-2022a.eb Arrow-11.0.0-gfbf-2022b.eb Arrow-14.0.1-gfbf-2023a.eb --include-easyblocks-from-pr=3088" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3088 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 12747

Test results coming soon (I hope)...

Details

- notification for comment with ID 1912269105 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@casparvl
Copy link
Copy Markdown
Contributor Author

casparvl commented Jan 26, 2024

Test report by @casparvl

Overview of tested easyconfigs (in order)

  • SUCCESS RapidJSON-1.1.0-20230928-GCCcore-12.3.0.eb
  • SUCCESS utf8proc-2.8.0-GCCcore-12.3.0.eb
  • SUCCESS Arrow-14.0.1-gfbf-2023a.eb

Build succeeded for 3 out of 3 (1 easyconfigs in total)
gcn6.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/casparvl/a0fe189f50a54286409e9ba0c13934fc for a full test report.

@boegelbot
Copy link
Copy Markdown

boegelbot commented Jan 26, 2024

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS Arrow-6.0.0-foss-2021a.eb
  • SUCCESS Arrow-6.0.0-foss-2021b.eb
  • SUCCESS Arrow-8.0.0-foss-2022a.eb
  • SUCCESS Arrow-11.0.0-gfbf-2022b.eb
  • SUCCESS Arrow-14.0.1-gfbf-2023a.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/b9a6da0657aad13150b7f31c4a46b8c4 for a full test report.

@boegel boegel added the bug fix label Jan 26, 2024
@boegel boegel added this to the release after 4.9.0 milestone Jan 26, 2024
boegel
boegel approved these changes Jan 26, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel boegel merged commit 941c1eb into easybuilders:develop Jan 26, 2024
@boegel boegel changed the title Apply CMake policy to select python based on PATH order, instead of highest version Apply CMake policy to select python based on $PATH order, instead of highest Python version Jan 26, 2024
@casparvl
Copy link
Copy Markdown
Contributor Author

casparvl commented Jan 27, 2024

Test report by @casparvl

Overview of tested easyconfigs (in order)

  • SUCCESS GDAL-3.7.1-foss-2023a.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
gcn6.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/casparvl/b95ac33020ded0449240b60624e6ff33 for a full test report.

@boegel boegel changed the title Apply CMake policy to select python based on $PATH order, instead of highest Python version apply CMake policy to select python based on $PATH order, instead of highest Python version Jan 30, 2024
This was referenced Feb 14, 2024
Closed
Merged
@bedroge bedroge mentioned this pull request Feb 20, 2024
Merged
@boegel boegel mentioned this pull request Mar 13, 2024
Merged
@boegel boegel changed the title apply CMake policy to select python based on $PATH order, instead of highest Python version tweak CMakeMake easyblock apply CMake policy to select python based on $PATH order, instead of highest Python version Apr 4, 2024
@boegel boegel mentioned this pull request Jul 3, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@boegel boegel boegel approved these changes

Assignees

No one assigned

Labels

bug fix

Projects

None yet

Milestone

4.9.1

Development

Successfully merging this pull request may close these issues.

3 participants

@casparvl @boegelbot @boegel