Skip to content

use PyYAML 6.0.1 instead of 6.0 for recent ReFrame versions to fix problem with Cython 3.x#20879

Merged
boegel merged 3 commits intoeasybuilders:developfrom
bedroge:fix_reframe433_dep_pyyaml
Jul 3, 2024
Merged

use PyYAML 6.0.1 instead of 6.0 for recent ReFrame versions to fix problem with Cython 3.x#20879
boegel merged 3 commits intoeasybuilders:developfrom
bedroge:fix_reframe433_dep_pyyaml

Conversation

@bedroge
Copy link
Copy Markdown
Contributor

@bedroge bedroge commented Jun 21, 2024

I ran into this when installing ReFrame on a system with Cython 3:

==> [+pygelf] python3 -m pip install --no-cache-dir -q -r /tmp/eb-myh20jsw/tmp.V9e2PYU6tR --target=external/ --upgrade
DEPRECATION: Loading egg at /cvmfs/riscv.eessi.io/versions/20240402/software/linux/riscv64/generic/software/ReFrame/4.3.3/lib/python3.11/site-packages/pip-21.3.1-py3.11.egg is deprecated. pi
p 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
  error: subprocess-exited-with-error
  
   Getting requirements to build wheel did not run successfully.
   exit code: 1
  > [54 lines of output]
      running egg_info
      writing lib/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/tmp/eb/easybuild/build/ReFrame/4.3.3/system-system/reframe/reframe-4.3.3/external/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/tmp/eb/easybuild/build/ReFrame/4.3.3/system-system/reframe/reframe-4.3.3/external/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/eb/easybuild/build/ReFrame/4.3.3/system-system/reframe/reframe-4.3.3/external/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
          self.run_setup()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 288, in <module>
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 976, in run_command
          super().run_command(command)
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 321, in run
          self.find_sources()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
          mm.run()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 550, in run
          self.add_defaults()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 588, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 102, in add_defaults
          super().add_defaults()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 250, in add_defaults
          self._add_defaults_ext()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 335, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 204, in get_source_files
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

 Getting requirements to build wheel did not run successfully.
 exit code: 1

Apparently PyYAML 6.0 has a known issue regarding Cython 3, and is not compatible with this version:
yaml/pyyaml#736
They worked around it by pinning the Cython build constraint to < 3.0:
yaml/pyyaml#702
And they released 6.0.1 with (only) this workaround: https://github.com/yaml/pyyaml/blob/6.0.1/announcement.msg

This PR replaces 6.0 in the requirements.txt by 6.0.1 for recent ReFrame version. Since nothing else is changed in PyYAML, this should be a safe change.

ReFrame 4.4.x and newer already use 6.0.1, so this fix can be removed for those versions.

@bedroge bedroge added bug fix EESSI Related to EESSI project labels Jun 21, 2024
@bedroge
Copy link
Copy Markdown
Contributor Author

bedroge commented Jun 21, 2024

@boegelbot please test @ generoso

@bedroge bedroge mentioned this pull request Jun 21, 2024
Open
@boegelbot
Copy link
Copy Markdown
Collaborator

boegelbot commented Jun 21, 2024

@bedroge: Request for testing this PR well received on login1

PR test command 'EB_PR=20879 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_20879 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 13796

Test results coming soon (I hope)...

Details

- notification for comment with ID 2182662114 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown
Collaborator

boegelbot commented Jun 21, 2024

Test report by @boegelbot
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
cns2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/117eee9aac35a9f844a8afba0acc2f5c for a full test report.

@bedroge
Copy link
Copy Markdown
Contributor Author

bedroge commented Jun 21, 2024

Test report by @bedroge
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
interactive1 - Linux Rocky Linux 8.9, x86_64, AMD EPYC-Milan Processor (zen2), Python 3.6.8
See https://gist.github.com/bedroge/3f8321df1b5f0d9edea0b85258e3329b for a full test report.

@bedroge
Copy link
Copy Markdown
Contributor Author

bedroge commented Jun 21, 2024

@boegelbot please test @ jsc-zen3

@boegelbot
Copy link
Copy Markdown
Collaborator

boegelbot commented Jun 21, 2024

@bedroge: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=20879 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_20879 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 4442

Test results coming soon (I hope)...

Details

- notification for comment with ID 2182687643 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown
Collaborator

boegelbot commented Jun 21, 2024

Test report by @boegelbot
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/9c6a4f21f69bc8e97596787bb0bc7047 for a full test report.

@boegel boegel added this to the release after 4.9.2 milestone Jul 3, 2024
boegel
boegel approved these changes Jul 3, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel
Copy link
Copy Markdown
Member

boegel commented Jul 3, 2024

Test report by @boegel
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
node3158.skitty.os - Linux RHEL 8.8, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Python 3.6.8
See https://gist.github.com/boegel/5a90b66337fffe8d6253fa184122ce54 for a full test report.

@boegel
Copy link
Copy Markdown
Member

boegel commented Jul 3, 2024

Going in, thanks @bedroge!

@boegel boegel merged commit 83a74ba into easybuilders:develop Jul 3, 2024
@bedroge bedroge deleted the fix_reframe433_dep_pyyaml branch July 4, 2024 09:20
@boegel boegel changed the title use PyYAML 6.0.1 instead of 6.0 for recent ReFrame versions use PyYAML 6.0.1 instead of 6.0 for recent ReFrame versions to fix problem with Cython 3.x Sep 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@boegel boegel boegel approved these changes

Assignees

No one assigned

Labels

bug fix EESSI Related to EESSI project

Projects

None yet

Milestone

4.9.3

Development

Successfully merging this pull request may close these issues.

3 participants

@bedroge @boegelbot @boegel