Skip to content

make sure setuptools installed in stage0 is still available at end of stage1#1727

Merged
boegel merged 1 commit intoeasybuilders:developfrom
boegel:fix_bootstrap_setuptools_stage1
Apr 19, 2016
Merged

make sure setuptools installed in stage0 is still available at end of stage1#1727
boegel merged 1 commit intoeasybuilders:developfrom
boegel:fix_bootstrap_setuptools_stage1

Conversation

@boegel
Copy link
Copy Markdown
Member

@boegel boegel commented Apr 19, 2016

fix for problem reported by @andreas-h on EasyBuild mailing list, which manifests itself as:

[[DEBUG]] Determining EasyBuild version using command '/usr/bin/python -c 'from easybuild.tools.version import this_is_easybuild; print(this_is_easybuild())' > /tmpx/jtmp.hilboll/tmpYppHcj/eb_version.out 2>&1'
[[ERROR]] Stage 1 failed, could not determine EasyBuild version (txt: Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmpx/jtmp.hilboll/tmpYppHcj/eb_stage1/lib/python2.7/site-packages/easybuild_easyblocks-2.7.0-py2.7.egg/easybuild/__init__.py", line 31, in <module>
    import pkg_resources
ImportError: No module named pkg_resources
).

The problem is that easybuild is being imported without having setuptools (which provides pkg_resources) available at all.

The fix proposed in this patch makes sure that the distribute (basically an old version of setuptools) being installed in stage0 remains available.

@boegel boegel added this to the v2.8.0 milestone Apr 19, 2016
@boegel
Copy link
Copy Markdown
Member Author

boegel commented Apr 19, 2016

@andreas-h: please test?

You can download the patched bootstrap script via curl -O https://raw.githubusercontent.com/boegel/easybuild-framework/fix_bootstrap_setuptools_stage1/easybuild/scripts/bootstrap_eb.py

@andreas-h
Copy link
Copy Markdown

andreas-h commented Apr 19, 2016 via email

@andreas-h
Copy link
Copy Markdown

andreas-h commented Apr 19, 2016 via email

@boegel
Copy link
Copy Markdown
Member Author

boegel commented Apr 19, 2016

@andreas-h $HOME/.local/lib/python2.7/site-packages is not picked up by default, it needs to be listed in $PYTHONPATH.

It *will8 be picked up when installing things in that location through --user, I think...

@andreas-h
Copy link
Copy Markdown

andreas-h commented Apr 19, 2016

Not so sure about that, @boegel , as on my Debian system, I have the following behaviour:

hilboll@shaula:~$ ls .local/lib/python2.7/site-packages/ltc_scrypt-1.0-py2.7-linux-x86_64.egg 
.local/lib/python2.7/site-packages/ltc_scrypt-1.0-py2.7-linux-x86_64.egg
hilboll@shaula:~$ find /usr/lib/python2.7/ -name 'ltc*'
hilboll@shaula:~$ echo $PYTHONPATH

hilboll@shaula:~$ python -c 'import ltc_scrypt'
hilboll@shaula:~$ python -c 'import ltc_scrypt; print ltc_scrypt.__file__'
/home2/hilboll/.python-eggs/ltc_scrypt-1.0-py2.7-linux-x86_64.egg-tmp/ltc_scrypt.so

On the contrary, some time ago (on a Ubuntu machine) I had the problem that $HOME/.local/lib/python2.7/site-packages was always picked up, even in a clean virtualenv with --no-site-packages.

@boegel
Copy link
Copy Markdown
Member Author

boegel commented Apr 19, 2016

@andreas-h Is that a system where setuptools is available centrally? If so, magic happens...

Anyway, the whole Python packaging ecosystem is a huge mess, and it's not getting better. The fact that we need a bootstrap script to get EasyBuild installed easily is just a little bit too ironic...

@andreas-h
Copy link
Copy Markdown

andreas-h commented Apr 19, 2016

@andreas-h Is that a system where setuptools is available centrally? If so, magic happens...

Maybe it's time to start believing in magic, then ;-)

hilboll@heron:~> python
Python 2.7.9 (default, Dec 21 2014, 11:02:59) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for p in sys.path: print p
... 

/usr/lib/python27.zip
/usr/lib64/python2.7
/usr/lib64/python2.7/plat-linux2
/usr/lib64/python2.7/lib-tk
/usr/lib64/python2.7/lib-old
/usr/lib64/python2.7/lib-dynload
/home/hilboll/.local/lib/python2.7/site-packages
/usr/lib64/python2.7/site-packages
/usr/local/lib64/python2.7/site-packages
/usr/local/lib/python2.7/site-packages
/usr/lib/python2.7/site-packages

So my local site-packages are in sys.path even though $PYTHONPATH
is empty.

But let's remove that and see what happens:

>>> sys.path = ['', '/usr/lib/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/local/lib64/python2.7/site-packages', '/usr/local/lib/python2.7/site-packages', '/usr/lib/python2.7/site-packages']
>>> for p in sys.path: print p
... 

/usr/lib/python27.zip
/usr/lib64/python2.7
/usr/lib64/python2.7/plat-linux2
/usr/lib64/python2.7/lib-tk
/usr/lib64/python2.7/lib-old
/usr/lib64/python2.7/lib-dynload
/usr/lib64/python2.7/site-packages
/usr/local/lib64/python2.7/site-packages
/usr/local/lib/python2.7/site-packages
/usr/lib/python2.7/site-packages
>>> import setuptools
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named setuptools

Anyways, the problem at hand is solved, so maybe that should be all to
wish for today ;-)

@hpcugentbot
Copy link
Copy Markdown

hpcugentbot commented Apr 19, 2016

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2942/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

@boegel
Copy link
Copy Markdown
Member Author

boegel commented Apr 19, 2016

@andreas-h The problem in the bootstrap script was a little bit different, there $PYTHONPATH was defined, but empty. Not sure if that makes a difference.

Anyway, the change done here should be airtight, regardless of whether setuptools is available on the system or not.

So, going in, thanks for the feedback!

@boegel boegel merged commit 7e77773 into easybuilders:develop Apr 19, 2016
@boegel boegel deleted the fix_bootstrap_setuptools_stage1 branch April 19, 2016 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

v2.8.0

Development

Successfully merging this pull request may close these issues.

3 participants

@boegel @andreas-h @hpcugentbot