update `Cargo` easyblock to extract crates into a vendor subdir and overwrite git repo URLs with local paths by lexming · Pull Request #3118 · easybuilders/easybuild-easyblocks

lexming · 2024-01-31T08:56:38Z

This evolved into a 2x1 PR, apologies for the added complexity.

Fix 1: extract crates into vendor subdir

Currently, the sources of the main Cargo package and the sources of all crates are all extracted into builddir. This is known to make some installations fail. The presence of the main sources alongside its dependencies can make cargo trip in identifying availability of vendored sources. See related comment below.

Not all Cargo packages are affected, but it seems to happen on more complex packaging. For instance, known failure is a package with a virtual workspace distributing multiple crates plus a another package in a subdir.

This PR solves this issue by extracting the list of crates into a separate vendor subdir without any sources of the main packages (i.e. items in the sources list of the easyconfig). This follows the same approach carried out by the cargo vendor command.

Fix 2: overwrite crates from git repos with local paths

I hit another issue where cargo was failing to locate the extracted sources of a dependency pulled from a git repo. The sources are properly downloaded and extracted from the git repo and the checksuming of the local sources is done properly. However, Cargo.lock of the main package being installed does not list any checksum for those dependencies pulled from a git repo, so cargo has no way to correlate the extracted folder to this dependency.

This PR solves this issue by overwriting the repo URL of each package pulled from a git repo with the local paths to the corresponding extracted folders. This is done in the generated .cargo/config.toml file by using the [patch.URL] directive as describe in the rust docs instead of [source.URL].

For instace,

[source."https://github.com/ritchie46/jsonpath"]
git = "https://github.com/ritchie46/jsonpath"
rev = "24eaf0b4416edff38a4d1b6b17bc4b9f3f047b4b"

becomes

[patch."https://github.com/ritchie46/jsonpath"]
jsonpath_lib = { path = "/user/brussel/101/vsc10122/easybuild/install/skylake/build/polars/0.19.19/gfbf-2023a/easybuild_vendor/jsonpath" }

Changelog

extract crates into self.builddir.easybuild_vendor
add static methods to generate filenames for downloaded tarballs and local tarballs
be smarter in setting finalpath of extracted sources by detecting
use global templates to generate .cargo/config.toml file
replace [source.URL] with [patch.URL] to overwrite git repos with local paths to extracted sources

…wnload_filename

lexming · 2024-01-31T14:14:54Z

@boegelbot please test @ generoso
EB_ARGS="maturin-1.1.0-GCCcore-12.3.0.eb maturin-1.4.0-GCCcore-12.3.0-Rust-1.75.0.eb cryptography-41.0.1-GCCcore-12.3.0.eb bcrypt-4.0.1-GCCcore-12.3.0.eb pydantic-2.5.3-GCCcore-12.3.0.eb"

boegelbot · 2024-01-31T14:20:05Z

@lexming: Request for testing this PR well received on login1

PR test command 'EB_PR=3118 EB_ARGS="maturin-1.1.0-GCCcore-12.3.0.eb maturin-1.4.0-GCCcore-12.3.0-Rust-1.75.0.eb cryptography-41.0.1-GCCcore-12.3.0.eb bcrypt-4.0.1-GCCcore-12.3.0.eb pydantic-2.5.3-GCCcore-12.3.0.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3118 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 12784

Test results coming soon (I hope)...

Details

- notification for comment with ID 1919188427 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-01-31T14:35:37Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS maturin-1.1.0-GCCcore-12.3.0.eb
SUCCESS maturin-1.4.0-GCCcore-12.3.0-Rust-1.75.0.eb
SUCCESS cryptography-41.0.1-GCCcore-12.3.0.eb
SUCCESS bcrypt-4.0.1-GCCcore-12.3.0.eb
SUCCESS pydantic-2.5.3-GCCcore-12.3.0.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
cns2 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/2154f0c6595af34a3e0feb6a1610a3b7 for a full test report.

…instead of [sources]

lexming · 2024-02-01T09:44:53Z

@boegelbot please test @ generoso
EB_ARGS="maturin-1.1.0-GCCcore-12.3.0.eb maturin-1.4.0-GCCcore-12.3.0-Rust-1.75.0.eb cryptography-41.0.1-GCCcore-12.3.0.eb bcrypt-4.0.1-GCCcore-12.3.0.eb pydantic-2.5.3-GCCcore-12.3.0.eb"

boegelbot · 2024-02-01T09:50:05Z

@lexming: Request for testing this PR well received on login1

PR test command 'EB_PR=3118 EB_ARGS="maturin-1.1.0-GCCcore-12.3.0.eb maturin-1.4.0-GCCcore-12.3.0-Rust-1.75.0.eb cryptography-41.0.1-GCCcore-12.3.0.eb bcrypt-4.0.1-GCCcore-12.3.0.eb pydantic-2.5.3-GCCcore-12.3.0.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3118 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 12797

Test results coming soon (I hope)...

Details

- notification for comment with ID 1920915926 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-02-01T10:08:45Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS maturin-1.1.0-GCCcore-12.3.0.eb
SUCCESS maturin-1.4.0-GCCcore-12.3.0-Rust-1.75.0.eb
SUCCESS cryptography-41.0.1-GCCcore-12.3.0.eb
SUCCESS bcrypt-4.0.1-GCCcore-12.3.0.eb
SUCCESS pydantic-2.5.3-GCCcore-12.3.0.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
cns1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/44d1ead4c616ace3bf7fa7da4143f94f for a full test report.

lexming · 2024-02-01T10:30:14Z

Test case for fix1 - DeltaLake:

Failed test with current develop: {data}[gfbf/2023a] DeltaLake v0.15.1 and fix Arrow v14.0.1 to install pyarrow as an extension easybuild-easyconfigs#19758 (comment)
Successful test with this PR: {data}[gfbf/2023a] DeltaLake v0.15.1 and fix Arrow v14.0.1 to install pyarrow as an extension easybuild-easyconfigs#19758 (comment)

lexming · 2024-02-01T15:45:39Z

Test case for fix2 - polars:

polars has jsonpath_lib pulled from a git repo

Failed test with current develop: {data}[gfbf/2023a] polars v0.19.19 easybuild-easyconfigs#19770 (comment)
Successful test with this PR: {data}[gfbf/2023a] polars v0.19.19 easybuild-easyconfigs#19770 (comment)

lexming · 2024-02-01T15:47:03Z

@Micket this is ready on my side, not touching it any more

Micket

lgtm

lexming added 5 commits January 30, 2024 23:54

add static methods to Cargo easyblock to define crate filename and do…

ad3d901

…wnload_filename

extract crates separate from main sources into vendor subdir

f5a7cee

use smarter detection of extracted dirs to define finalpath of crates

236cac3

rename cargo vendor dir to easybuild_vendor

e3b8383

add lexming to author list of Cargo easyblock

db984ac

lexming mentioned this pull request Jan 31, 2024

{data}[gfbf/2023a] DeltaLake v0.15.1 and fix Arrow v14.0.1 to install pyarrow as an extension easybuilders/easybuild-easyconfigs#19758

Merged

2 tasks

use Cargo.crate_src_filename with crates from git repos

a8147af

lexming requested a review from Micket January 31, 2024 16:28

template generation of config.toml and handle git repos with [patch] …

7e11856

…instead of [sources]

lexming changed the title ~~Place extracted dirs of crates into a vendor subdir without main package sources~~ Extract crates into a vendor subdir and overwrite git repo URLs with patch directive Feb 1, 2024

lexming changed the title ~~Extract crates into a vendor subdir and overwrite git repo URLs with patch directive~~ Extract crates into a vendor subdir and overwrite git repo URLs with local paths Feb 1, 2024

lexming mentioned this pull request Feb 1, 2024

{data}[gfbf/2023a] polars v0.19.19 easybuilders/easybuild-easyconfigs#19770

Merged

4 tasks

boegel added the bug fix label Feb 1, 2024

boegel added this to the 4.x milestone Feb 1, 2024

boegel changed the title ~~Extract crates into a vendor subdir and overwrite git repo URLs with local paths~~ update Cargo easyblock to extract crates into a vendor subdir and overwrite git repo URLs with local paths Feb 1, 2024

Micket approved these changes Feb 7, 2024

View reviewed changes

Micket merged commit 836f08d into easybuilders:develop Feb 7, 2024

Micket modified the milestones: 4.x, release after 4.9.0 Feb 7, 2024

lexming deleted the cargo-vendor branch February 8, 2024 00:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update `Cargo` easyblock to extract crates into a vendor subdir and overwrite git repo URLs with local paths#3118

update `Cargo` easyblock to extract crates into a vendor subdir and overwrite git repo URLs with local paths#3118
Micket merged 7 commits intoeasybuilders:developfrom
lexming:cargo-vendor

lexming commented Jan 31, 2024 •

edited

Loading

Uh oh!

lexming commented Jan 31, 2024

Uh oh!

boegelbot commented Jan 31, 2024

Uh oh!

boegelbot commented Jan 31, 2024

Uh oh!

lexming commented Feb 1, 2024

Uh oh!

boegelbot commented Feb 1, 2024

Uh oh!

boegelbot commented Feb 1, 2024

Uh oh!

lexming commented Feb 1, 2024 •

edited

Loading

Uh oh!

lexming commented Feb 1, 2024 •

edited

Loading

Uh oh!

lexming commented Feb 1, 2024

Uh oh!

Micket left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

lexming commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fix 1: extract crates into vendor subdir

Fix 2: overwrite crates from git repos with local paths

Changelog

Uh oh!

lexming commented Jan 31, 2024

Uh oh!

boegelbot commented Jan 31, 2024

Uh oh!

boegelbot commented Jan 31, 2024

Overview of tested easyconfigs (in order)

Uh oh!

lexming commented Feb 1, 2024

Uh oh!

boegelbot commented Feb 1, 2024

Uh oh!

boegelbot commented Feb 1, 2024

Overview of tested easyconfigs (in order)

Uh oh!

lexming commented Feb 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test case for fix1 - DeltaLake:

Uh oh!

lexming commented Feb 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test case for fix2 - polars:

Uh oh!

lexming commented Feb 1, 2024

Uh oh!

Micket left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lexming commented Jan 31, 2024 •

edited

Loading

lexming commented Feb 1, 2024 •

edited

Loading

lexming commented Feb 1, 2024 •

edited

Loading