Skip to content

fix derive_alt_pypi_url after PyPI switching to sha256 in package URLs + fix broken test for pypi_source_urls + fix bootstrap script#2471

Merged
wpoely86 merged 5 commits intoeasybuilders:developfrom
boegel:pypi_sha256
Apr 12, 2018
Merged

fix derive_alt_pypi_url after PyPI switching to sha256 in package URLs + fix broken test for pypi_source_urls + fix bootstrap script#2471
wpoely86 merged 5 commits intoeasybuilders:developfrom
boegel:pypi_sha256

Conversation

@boegel
Copy link
Copy Markdown
Member

@boegel boegel commented Apr 12, 2018

Seems like PyPI is switching away from MD5 checksums in favor of SHA256 checksums, probably related to the switch to the new "Warehouse" PyPI.

@boegel boegel added the bug fix label Apr 12, 2018
@boegel boegel added this to the 3.6.0 milestone Apr 12, 2018
@easybuilders easybuilders deleted a comment from boegelbot Apr 12, 2018
@boegel boegel changed the title fix derive_alt_pypi_url after PyPI switching to sha256 in package URLs + fix broken test for pypi_source_urls fix derive_alt_pypi_url after PyPI switching to sha256 in package URLs + fix broken test for pypi_source_urls + fix bootstrap script Apr 12, 2018
wpoely86
wpoely86 reviewed Apr 12, 2018
View reviewed changes
Comment thread easybuild/tools/filetools.py Outdated

if not alt_pypi_url:
_log.debug("Failed to extract hash using pattern '%s' from list of URLs: %s", regex.pattern, cand_urls)
_log.debug("Failed to extract hash using pattern '%s' or '%s' from list of URLs: %s", regex.pattern, cand_urls)
Copy link
Copy Markdown
Member

@wpoely86 wpoely86 Apr 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3x %s, only two arguments?

Copy link
Copy Markdown
Member Author

@boegel boegel Apr 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch, thanks, fix pushed.

boegel
boegel commented Apr 12, 2018
View reviewed changes
Comment thread easybuild/scripts/bootstrap_eb.py Outdated
handle = open(ebfile, 'w')
# need to initialise build options before pypi_source_urls function can be used
from test.framework.utilities import init_config
init_config()
Copy link
Copy Markdown
Member Author

@boegel boegel Apr 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not entirely happy with this bit, but I consider the stuff below a temporary fix.

Once EasyBuild v3.6.0 is out that includes the fix for derive_alt_pypi_url, we can roll back this change.

wpoely86
wpoely86 previously approved these changes Apr 12, 2018
View reviewed changes
@MisterFruits
Copy link
Copy Markdown

MisterFruits commented Apr 12, 2018

wierd behaviour of the new bootstrap script, it looks like it didn't found the modules tool (I'm pretty sure it found it automaticaly before that)

$ python bootstrap_eb.py testeb
[[INFO]] EasyBuild bootstrap script (version 20180412.01, MD5: eb5cac2c956551686ccf883c8f8cde2f)
[[INFO]] Found Python 2.7.5 (default, May  3 2017, 07:55:04) ; [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)]

[[INFO]] Installation [...]/test/testeb
[[INFO]] Found module command 'modulecmd' (EnvironmentModulesC), so using it.
[[INFO]] Suitable setuptools installation already found, skipping stage 0...


[[INFO]] +++ STAGE 1: installing EasyBuild in temporary dir with easy_install...

[[INFO]] installing EasyBuild with 'easy_install --quiet --upgrade --prefix=/tmp/tmpUwMk8o/eb_stage1 easybuild'
[[INFO]] running post install command 'easy_install --upgrade --prefix=/tmp/tmpUwMk8o/eb_stage1 vsc-base'


[[INFO]] +++ STAGE 2: installing EasyBuild in[...]/test/testeb with EasyBuild from stage 1...

== temporary log file in case of crash /tmp/eb-A9U9At/eb-nouMGM/eb-id1xWA/easybuild-fuPhfq.log
Traceback (most recent call last):
  File "bootstrap_eb.py", line 1069, in <module>
    main()
  File "bootstrap_eb.py", line 872, in main
    stage2(tmpdir, templates, install_path, distribute_egg_dir, sourcepath)
  File "bootstrap_eb.py", line 739, in stage2
    easybuild_main()
  File "/tmp/tmpUwMk8o/eb_stage1/lib/python2.7/site-packages/easybuild_framework-3.5.3-py2.7.egg/easybuild/main.py", line 259, in main
    modtool = modules_tool(testing=testing)
  File "/tmp/tmpUwMk8o/eb_stage1/lib/python2.7/site-packages/easybuild_framework-3.5.3-py2.7.egg/easybuild/tools/modules.py", line 1354, in modules_tool
    return modules_tool_class(mod_paths=mod_paths, testing=testing)
  File "/tmp/tmpUwMk8o/eb_stage1/lib/python2.7/site-packages/easybuild_framework-3.5.3-py2.7.egg/easybuild/tools/modules.py", line 1118, in __init__
    super(Lmod, self).__init__(*args, **kwargs)
  File "/tmp/tmpUwMk8o/eb_stage1/lib/python2.7/site-packages/easybuild_framework-3.5.3-py2.7.egg/easybuild/tools/modules.py", line 196, in __init__
    self.check_cmd_avail()
  File "/tmp/tmpUwMk8o/eb_stage1/lib/python2.7/site-packages/easybuild_framework-3.5.3-py2.7.egg/easybuild/tools/modules.py", line 274, in check_cmd_avail
    raise EasyBuildError(error_msg)
easybuild.tools.build_log.EasyBuildError: "Lmod modules tool can not be used, 'lmod' command is not available; use --modules-tool to specify a different modules tool to use (EnvironmentModules, Lmod, EnvironmentModulesTcl, EnvironmentModulesC)"
```

@boegel
don't use pypi_source_urls in bootstrap script (since that requires i…
e5b73b0 
…nitialised build options), implement lightweight version instead
@boegel
Copy link
Copy Markdown
Member Author

boegel commented Apr 12, 2018

@MisterFruits I indeed introduced a problem in the bootstrap script, the tests also caught it...

I've pushed another updated (version 20180412.02, see e5b73b0) which fixes that, please re-test?

@wpoely86 wpoely86 merged commit c906b6c into easybuilders:develop Apr 12, 2018
@boegel boegel deleted the pypi_sha256 branch April 12, 2018 20:09
This was referenced Apr 20, 2018
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wpoely86 wpoely86 wpoely86 approved these changes

Assignees

No one assigned

Labels

bug fix

Projects

None yet

Milestone

3.6.0

Development

Successfully merging this pull request may close these issues.

3 participants

@boegel @MisterFruits @wpoely86