{ai}[foss/2024a] TRL v0.22.0, PEFT v0.17.1, evaluate v0.4.6, HF-Datasets v4.0.0 by lexming · Pull Request #24302 · easybuilders/easybuild-easyconfigs

lexming · 2025-10-20T08:30:23Z

(created using eb --new-pr)

…fbf-2024a.eb, HF-Datasets-4.0.0-gfbf-2024a.eb, PEFT-0.17.1-foss-2024a.eb, TRL-0.22.0-foss-2024a.eb

github-actions · 2025-10-20T08:32:26Z

Updated software `evaluate-0.4.6-gfbf-2024a.eb`

Diff against evaluate-0.4.3-gfbf-2023a.eb

easybuild/easyconfigs/e/evaluate/evaluate-0.4.3-gfbf-2023a.eb

diff --git a/easybuild/easyconfigs/e/evaluate/evaluate-0.4.3-gfbf-2023a.eb b/easybuild/easyconfigs/e/evaluate/evaluate-0.4.6-gfbf-2024a.eb
index 37b624df7e..fb8d41c97a 100644
--- a/easybuild/easyconfigs/e/evaluate/evaluate-0.4.3-gfbf-2023a.eb
+++ b/easybuild/easyconfigs/e/evaluate/evaluate-0.4.6-gfbf-2024a.eb
@@ -1,28 +1,25 @@
 easyblock = 'PythonBundle'
 
 name = 'evaluate'
-version = '0.4.3'
+version = '0.4.6'
 
 homepage = 'https://huggingface.co/docs/evaluate/index'
 description = """
 A library for easily evaluating machine learning models and datasets.
 """
 
-toolchain = {'name': 'gfbf', 'version': '2023a'}
+toolchain = {'name': 'gfbf', 'version': '2024a'}
 
 dependencies = [
-    ('Python', '3.11.3'),
-    ('SciPy-bundle', '2023.07'),
-    ('Python-bundle-PyPI', '2023.06'),  # for requests
-    ('HF-Datasets', '2.18.0'),
-    ('dill', '0.3.7'),
-    ('tqdm', '4.66.1'),
-    ('multiprocess', '0.70.15'),
+    ('Python', '3.12.3'),
+    ('SciPy-bundle', '2024.05'),
+    ('HF-Datasets', '4.0.0'),
+    ('multiprocess', '0.70.17'),
 ]
 
 exts_list = [
     (name, version, {
-        'checksums': ['3a5700cf83aabee9549264e1e5666f116367c61dbd4d38352015e859a5e2098d'],
+        'checksums': ['e07036ca12b3c24331f83ab787f21cc2dbf3631813a1631e63e40897c69a3f21'],
     }),
 ]

Updated software `HF-Datasets-4.0.0-gfbf-2024a.eb`

Diff against HF-Datasets-3.1.0-gfbf-2023b.eb

easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-3.1.0-gfbf-2023b.eb

diff --git a/easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-3.1.0-gfbf-2023b.eb b/easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-4.0.0-gfbf-2024a.eb
index c1c51281a8..4b2c6996a4 100644
--- a/easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-3.1.0-gfbf-2023b.eb
+++ b/easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-4.0.0-gfbf-2024a.eb
@@ -1,7 +1,7 @@
 easyblock = "PythonBundle"
 
 name = 'HF-Datasets'
-version = '3.1.0'
+version = '4.0.0'
 
 homepage = 'https://github.com/huggingface/datasets'
 description = """
@@ -9,34 +9,30 @@ The largest hub of ready-to-use datasets for ML models with fast, easy-to-use an
 data manipulation tools.
 """
 
-toolchain = {'name': 'gfbf', 'version': '2023b'}
+toolchain = {'name': 'gfbf', 'version': '2024a'}
 
 dependencies = [
-    ('Python', '3.11.5'),
-    ('Python-bundle-PyPI', '2023.10'),  # for fsspec
-    ('SciPy-bundle', '2023.11'),
-    ('aiohttp', '3.9.5'),
-    ('dill', '0.3.8'),
-    ('Arrow', '16.1.0'),
-    ('PyYAML', '6.0.1'),
-    ('tqdm', '4.66.2'),
-    ('python-xxhash', '3.4.1'),
+    ('Python', '3.12.3'),
+    ('SciPy-bundle', '2024.05'),
+    ('aiohttp', '3.10.10'),
+    ('Arrow', '17.0.0'),
+    ('dill', '0.3.9'),
+    ('huggingface_hub', '0.34.4'),
+    ('multiprocess', '0.70.17'),
+    ('python-xxhash', '3.5.0'),
+    ('PyYAML', '6.0.2'),
+    ('tqdm', '4.66.5'),
 ]
 
-local_preinstallopts = "sed -i s'/tqdm>=4.66.3/tqdm/' setup.py && "
-local_preinstallopts += "sed -i s'/requests>=2.32.2/requests/' setup.py && "
+# fake pinning due to old issue, 'datasets' pushes the max pinned version of 'dill' on every release
+_fix_pins = """sed -i s'/dill>=0.3.0.*",/dill>=0.3.0",/' setup.py && """
+# unlocking dill allows to unlock multiprocess
+_fix_pins += """sed -i s'/multiprocess.*",/multiprocess",/' setup.py && """
 
 exts_list = [
-    ('multiprocess', '0.70.16', {
-        'checksums': ['161af703d4652a0e1410be6abccecde4a7ddffd19341be0a7011b94aeb171ac1'],
-    }),
-    ('huggingface-hub', '0.26.2', {
-        'sources': ['huggingface_hub-%(version)s.tar.gz'],
-        'checksums': ['b100d853465d965733964d123939ba287da60a547087783ddff8a323f340332b'],
-    }),
     ('datasets', version, {
-        'preinstallopts': local_preinstallopts,
-        'checksums': ['c92cac049e0f9f85b0dd63739c68e564c657b1624bc2b66b1e13489062832e27'],
+        'preinstallopts': _fix_pins,
+        'checksums': ['9657e7140a9050db13443ba21cb5de185af8af944479b00e7ff1e00a61c8dbf1'],
     }),
 ]

Diff against HF-Datasets-2.14.4-gfbf-2022b.eb

easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-2.14.4-gfbf-2022b.eb

diff --git a/easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-2.14.4-gfbf-2022b.eb b/easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-4.0.0-gfbf-2024a.eb
index 5fadc422e9..4b2c6996a4 100644
--- a/easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-2.14.4-gfbf-2022b.eb
+++ b/easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-4.0.0-gfbf-2024a.eb
@@ -1,7 +1,7 @@
 easyblock = "PythonBundle"
 
 name = 'HF-Datasets'
-version = '2.14.4'
+version = '4.0.0'
 
 homepage = 'https://github.com/huggingface/datasets'
 description = """
@@ -9,32 +9,30 @@ The largest hub of ready-to-use datasets for ML models with fast, easy-to-use an
 data manipulation tools.
 """
 
-toolchain = {'name': 'gfbf', 'version': '2022b'}
+toolchain = {'name': 'gfbf', 'version': '2024a'}
 
 dependencies = [
-    ('Python', '3.10.8'),
-    ('SciPy-bundle', '2023.02'),
-    ('aiohttp', '3.8.5'),
-    ('dill', '0.3.7'),
-    ('Arrow', '11.0.0'),
-    ('PyYAML', '6.0'),
-    ('tqdm', '4.64.1'),
-    ('python-xxhash', '3.2.0'),
+    ('Python', '3.12.3'),
+    ('SciPy-bundle', '2024.05'),
+    ('aiohttp', '3.10.10'),
+    ('Arrow', '17.0.0'),
+    ('dill', '0.3.9'),
+    ('huggingface_hub', '0.34.4'),
+    ('multiprocess', '0.70.17'),
+    ('python-xxhash', '3.5.0'),
+    ('PyYAML', '6.0.2'),
+    ('tqdm', '4.66.5'),
 ]
 
+# fake pinning due to old issue, 'datasets' pushes the max pinned version of 'dill' on every release
+_fix_pins = """sed -i s'/dill>=0.3.0.*",/dill>=0.3.0",/' setup.py && """
+# unlocking dill allows to unlock multiprocess
+_fix_pins += """sed -i s'/multiprocess.*",/multiprocess",/' setup.py && """
+
 exts_list = [
-    ('pyarrow_hotfix', '0.6', {
-        'checksums': ['79d3e030f7ff890d408a100ac16d6f00b14d44a502d7897cd9fc3e3a534e9945'],
-    }),
-    ('multiprocess', '0.70.15', {
-        'checksums': ['f20eed3036c0ef477b07a4177cf7c1ba520d9a2677870a4f47fe026f0cd6787e'],
-    }),
-    ('huggingface-hub', '0.15.1', {
-        'sources': ['huggingface_hub-%(version)s.tar.gz'],
-        'checksums': ['a61b7d1a7769fe10119e730277c72ab99d95c48d86a3d6da3e9f3d0f632a4081'],
-    }),
     ('datasets', version, {
-        'checksums': ['ef29c2b5841de488cd343cfc26ab979bff77efa4d2285af51f1ad7db5c46a83b'],
+        'preinstallopts': _fix_pins,
+        'checksums': ['9657e7140a9050db13443ba21cb5de185af8af944479b00e7ff1e00a61c8dbf1'],
     }),
 ]

Diff against HF-Datasets-2.18.0-gfbf-2023a.eb

easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-2.18.0-gfbf-2023a.eb

diff --git a/easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-2.18.0-gfbf-2023a.eb b/easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-4.0.0-gfbf-2024a.eb
index 3688a2eb40..4b2c6996a4 100644
--- a/easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-2.18.0-gfbf-2023a.eb
+++ b/easybuild/easyconfigs/h/HF-Datasets/HF-Datasets-4.0.0-gfbf-2024a.eb
@@ -1,7 +1,7 @@
 easyblock = "PythonBundle"
 
 name = 'HF-Datasets'
-version = '2.18.0'
+version = '4.0.0'
 
 homepage = 'https://github.com/huggingface/datasets'
 description = """
@@ -9,40 +9,30 @@ The largest hub of ready-to-use datasets for ML models with fast, easy-to-use an
 data manipulation tools.
 """
 
-toolchain = {'name': 'gfbf', 'version': '2023a'}
+toolchain = {'name': 'gfbf', 'version': '2024a'}
 
 dependencies = [
-    ('Python', '3.11.3'),
-    ('Python-bundle-PyPI', '2023.06'),  # for fsspec
-    ('SciPy-bundle', '2023.07'),
-    ('aiohttp', '3.8.5'),
-    ('dill', '0.3.7'),
-    ('Arrow', '14.0.1'),
-    ('PyYAML', '6.0'),
-    ('tqdm', '4.66.1'),
-    ('python-xxhash', '3.4.1'),
+    ('Python', '3.12.3'),
+    ('SciPy-bundle', '2024.05'),
+    ('aiohttp', '3.10.10'),
+    ('Arrow', '17.0.0'),
+    ('dill', '0.3.9'),
+    ('huggingface_hub', '0.34.4'),
+    ('multiprocess', '0.70.17'),
+    ('python-xxhash', '3.5.0'),
+    ('PyYAML', '6.0.2'),
+    ('tqdm', '4.66.5'),
 ]
 
+# fake pinning due to old issue, 'datasets' pushes the max pinned version of 'dill' on every release
+_fix_pins = """sed -i s'/dill>=0.3.0.*",/dill>=0.3.0",/' setup.py && """
+# unlocking dill allows to unlock multiprocess
+_fix_pins += """sed -i s'/multiprocess.*",/multiprocess",/' setup.py && """
+
 exts_list = [
-    ('multiprocess', '0.70.15', {
-        'checksums': ['f20eed3036c0ef477b07a4177cf7c1ba520d9a2677870a4f47fe026f0cd6787e'],
-    }),
-    ('huggingface-hub', '0.21.4', {
-        'sources': ['huggingface_hub-%(version)s.tar.gz'],
-        'checksums': ['e1f4968c93726565a80edf6dc309763c7b546d0cfe79aa221206034d50155531'],
-    }),
     ('datasets', version, {
-        'patches': [
-            'datasets-2.18.0_use-patched-pyarrow.patch',
-            'datasets-2.18.0_add-option-to-use-softfilelock.patch',
-        ],
-        'checksums': [
-            {'datasets-2.18.0.tar.gz': 'cdf8b8c6abf7316377ba4f49f9589a4c74556d6b481afd0abd2284f3d69185cb'},
-            {'datasets-2.18.0_use-patched-pyarrow.patch':
-             'a42f23c94f3f4935aae7a01aebd04bacae518537ecb674efeb94d5af7a296061'},
-            {'datasets-2.18.0_add-option-to-use-softfilelock.patch':
-             '18f3a3f3735afcbf9f4c19a2bd5bdb7221ca5ad13a36748df42404764a8cd0fc'},
-        ],
+        'preinstallopts': _fix_pins,
+        'checksums': ['9657e7140a9050db13443ba21cb5de185af8af944479b00e7ff1e00a61c8dbf1'],
     }),
 ]

lexming · 2025-10-20T09:13:15Z

@boegelbot please test @ jsc-zen3

boegelbot · 2025-10-20T09:20:21Z

@lexming: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=24302 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_24302 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 8480

Test results coming soon (I hope)...

Details

- notification for comment with ID 3421173291 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2025-10-20T09:26:25Z

Test report by @boegelbot
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/a21b5ee93df6d0e124f8b800f660ebf8 for a full test report.

Micket

lgtm

lexming · 2025-11-04T14:23:59Z

@Micket ping on this approved PR 🙂

adding easyconfigs: accelerate-1.10.1-foss-2024a.eb, evaluate-0.4.6-g…

70f9a71

…fbf-2024a.eb, HF-Datasets-4.0.0-gfbf-2024a.eb, PEFT-0.17.1-foss-2024a.eb, TRL-0.22.0-foss-2024a.eb

lexming added new update labels Oct 20, 2025

downgrade dependency on accelerate to existing v1.10.0

70cd22b

lexming changed the title ~~{ai}[foss/2024a] accelerate v1.10.1, evaluate v0.4.6, HF-Datasets v4.0.0, ...~~ {ai}[foss/2024a] TRL v0.22.0, PEFT v0.17.1, evaluate v0.4.6, HF-Datasets v4.0.0 Oct 20, 2025

Micket approved these changes Oct 28, 2025

View reviewed changes

Micket added this to the next release (5.2.0?) milestone Oct 28, 2025

Micket merged commit ca37068 into easybuilders:develop Nov 4, 2025
8 checks passed

lexming deleted the 20251020103019_new_pr_accelerate1101 branch November 5, 2025 08:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{ai}[foss/2024a] TRL v0.22.0, PEFT v0.17.1, evaluate v0.4.6, HF-Datasets v4.0.0#24302

{ai}[foss/2024a] TRL v0.22.0, PEFT v0.17.1, evaluate v0.4.6, HF-Datasets v4.0.0#24302
Micket merged 2 commits intoeasybuilders:developfrom
lexming:20251020103019_new_pr_accelerate1101

lexming commented Oct 20, 2025

Uh oh!

github-actions Bot commented Oct 20, 2025 •

edited

Loading

Uh oh!

lexming commented Oct 20, 2025

Uh oh!

boegelbot commented Oct 20, 2025

Uh oh!

boegelbot commented Oct 20, 2025

Uh oh!

Micket left a comment

Uh oh!

lexming commented Nov 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

lexming commented Oct 20, 2025

Uh oh!

github-actions Bot commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Updated software evaluate-0.4.6-gfbf-2024a.eb

Updated software HF-Datasets-4.0.0-gfbf-2024a.eb

Uh oh!

lexming commented Oct 20, 2025

Uh oh!

boegelbot commented Oct 20, 2025

Uh oh!

boegelbot commented Oct 20, 2025

Uh oh!

Micket left a comment

Choose a reason for hiding this comment

Uh oh!

lexming commented Nov 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Oct 20, 2025 •

edited

Loading

Updated software `evaluate-0.4.6-gfbf-2024a.eb`

Updated software `HF-Datasets-4.0.0-gfbf-2024a.eb`