Skip to content

make a project environment and manage LOAD_PATH for JuliaPackage#3239

Merged
branfosj merged 10 commits intoeasybuilders:developfrom
lexming:julia-env-load-path
Jul 3, 2024
Merged

make a project environment and manage LOAD_PATH for JuliaPackage#3239
branfosj merged 10 commits intoeasybuilders:developfrom
lexming:julia-env-load-path

Conversation

@lexming
Copy link
Copy Markdown
Contributor

@lexming lexming commented Mar 1, 2024 •
edited
Loading

This PR changes a bit how installations of JuliaPackages are handled to get the following improvements:

Fixes easybuilders/easybuild-easyconfigs#19281
Alternative fix for easybuilders/easybuild-easyconfigs#19650

  • Properly handle JuliaPackages that have other JuliaPackages in their dependencies. Currently those dependencies are not seen by Julia.Pkg which results in packages being re-installed even though they can be found in the deps. This is fixed by adding all Julia packages found across dependencies to the Julia environment of the installation.
  • Don't make assumptions about DEPOT_PATH and grab it from the Julia environment. This avoids needing JULIA_DEPOT_PATH in the shell environment.
  • Respect user environment: user depot and its shared environment for this version of Julia are kept as top paths of DEPOT_PATH and LOAD_PATH respectively. This ensures that the user can keep using its own environment after loading JuliaPackage modules, installing additional software on its personal depot while still using packages provided by the module:
    • Don't reset DEPOT_PATH or LOAD_PATH on module load unless they are not set at all
    • Append path to installation to DEPOT_PATH and path to installation environment to LOAD_PATH

@easybuilders easybuilders deleted a comment from boegelbot Mar 8, 2024
@easybuilders easybuilders deleted a comment from boegelbot Mar 8, 2024
@lexming
Copy link
Copy Markdown
Contributor Author

lexming commented Mar 8, 2024

@boegelbot please test @ generoso
EB_ARGS="IJulia-1.24.0-Julia-1.8.5.eb MultilevelEstimators-0.1.0-GCC-11.2.0-Julia-1.7.2.eb Circuitscape-5.12.3-Julia-1.9.2.eb Circuitscape-5.12.3-Julia-1.7.2.eb RCall-0.13.17-foss-2022a-R-4.2.1-Julia-1.9.2.eb"

@boegelbot
Copy link
Copy Markdown

boegelbot commented Mar 8, 2024

@lexming: Request for testing this PR well received on login1

PR test command 'EB_PR=3239 EB_ARGS="IJulia-1.24.0-Julia-1.8.5.eb MultilevelEstimators-0.1.0-GCC-11.2.0-Julia-1.7.2.eb Circuitscape-5.12.3-Julia-1.9.2.eb Circuitscape-5.12.3-Julia-1.7.2.eb RCall-0.13.17-foss-2022a-R-4.2.1-Julia-1.9.2.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3239 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 13053

Test results coming soon (I hope)...

Details

- notification for comment with ID 1985233802 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown

boegelbot commented Mar 8, 2024

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS IJulia-1.24.0-Julia-1.8.5.eb
  • SUCCESS MultilevelEstimators-0.1.0-GCC-11.2.0-Julia-1.7.2.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.7.2.eb
  • FAIL (build issue) RCall-0.13.17-foss-2022a-R-4.2.1-Julia-1.9.2.eb (partial log available at https://gist.github.com/boegelbot/d36b92e87518153280ec344c81135c49)

Build succeeded for 4 out of 5 (5 easyconfigs in total)
cns1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/119aea737428f95a9a8d32c93c5d15a1 for a full test report.

This was referenced Mar 13, 2024
Merged
Merged
@boegel boegel added this to the 4.x milestone Mar 13, 2024
@easybuilders easybuilders deleted a comment from boegelbot Mar 13, 2024
@easybuilders easybuilders deleted a comment from boegelbot Mar 13, 2024
@lexming
Copy link
Copy Markdown
Contributor Author

lexming commented Mar 13, 2024

@boegelbot please test @ generoso
EB_ARGS="IJulia-1.23.3-Julia-1.6.7.eb IJulia-1.24.0-Julia-1.8.5.eb MultilevelEstimators-0.1.0-GCC-11.2.0-Julia-1.7.2.eb Circuitscape-5.12.3-Julia-1.9.2.eb Circuitscape-5.12.3-Julia-1.7.2.eb"

@boegelbot
Copy link
Copy Markdown

boegelbot commented Mar 13, 2024

@lexming: Request for testing this PR well received on login1

PR test command 'EB_PR=3239 EB_ARGS="IJulia-1.23.3-Julia-1.6.7.eb IJulia-1.24.0-Julia-1.8.5.eb MultilevelEstimators-0.1.0-GCC-11.2.0-Julia-1.7.2.eb Circuitscape-5.12.3-Julia-1.9.2.eb Circuitscape-5.12.3-Julia-1.7.2.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3239 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 13096

Test results coming soon (I hope)...

Details

- notification for comment with ID 1994749688 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown

boegelbot commented Mar 13, 2024

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS IJulia-1.23.3-Julia-1.6.7.eb
  • SUCCESS IJulia-1.24.0-Julia-1.8.5.eb
  • SUCCESS MultilevelEstimators-0.1.0-GCC-11.2.0-Julia-1.7.2.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.7.2.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
cns3 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/7e0fbae412c57a421a187b7aa73523f1 for a full test report.

@branfosj branfosj modified the milestones: 4.x, release after 4.9.2 Jul 3, 2024
@branfosj
Copy link
Copy Markdown
Member

branfosj commented Jul 3, 2024

Test report by @branfosj

Overview of tested easyconfigs (in order)

  • SUCCESS IJulia-1.23.3-Julia-1.6.7.eb
  • SUCCESS IJulia-1.24.0-Julia-1.8.5.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.7.2.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total)
bear-pg0105u03a - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/branfosj/59be7c3f84039927157e18aa85d69f75 for a full test report.

@branfosj
Copy link
Copy Markdown
Member

branfosj commented Jul 3, 2024

Going in, thanks @lexming!

@branfosj
Copy link
Copy Markdown
Member

branfosj commented Jul 3, 2024

Going in, thanks @lexming!

@branfosj branfosj merged commit 640021d into easybuilders:develop Jul 3, 2024
@boegel boegel changed the title make a project environment and manage LOAD_PATH for JuliaPackage make a project environment and manage LOAD_PATH for JuliaPackage Jul 3, 2024
@lexming lexming mentioned this pull request Jul 4, 2024
Closed
@lexming lexming deleted the julia-env-load-path branch July 4, 2024 09:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@branfosj branfosj branfosj approved these changes

Assignees

No one assigned

Labels

enhancement

Projects

None yet

Milestone

4.9.3

Development

Successfully merging this pull request may close these issues.

Odd issue with JULIA_DEPOT_PATH makes ResistanceGA unusable

4 participants

@lexming @boegelbot @branfosj @boegel