Skip to content

Add option to disable pip connecting to PyPi (enable use of --no-index)#2390

Merged
smoors merged 1 commit intoeasybuilders:developfrom
Flamefire:pipNoIndex
Apr 27, 2021
Merged

Add option to disable pip connecting to PyPi (enable use of --no-index)#2390
smoors merged 1 commit intoeasybuilders:developfrom
Flamefire:pipNoIndex

Conversation

@Flamefire
Copy link
Copy Markdown
Contributor

@Flamefire Flamefire commented Apr 15, 2021

Pip does periodically check for new versions of pip on PyPi which can be disabled with --disable-pip-version-check
We certainly want to disable that as it is of no use for us.

However we want even more: pip should not connect to PyPi at all to e.g. download dependencies. This can be done with --no-index which implies --disable-pip-version-check.

To be safe I added an option which is by default only enabled when download_dep_fail is enabled, although I think enabling always by default makes more sense. But I guess that's for EB 5.

--> Faster installations of Python packages and no unwanted network traffic.

@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Apr 15, 2021

Will that still enable us to see what it would have tried to download so that we can easily see what deps are really missing?

@Flamefire
Copy link
Copy Markdown
Contributor Author

Flamefire commented Apr 15, 2021

Well at least the pip check at the end will tell us, but generally we have use_pip_for_deps set to False by default which means --no-deps is added so it shouldn't even try to download those anyway

@smoors
Copy link
Copy Markdown
Contributor

smoors commented Apr 27, 2021

Test report by @smoors

Overview of tested easyconfigs (in order)

  • SUCCESS PySAT-0.1.7.dev1-GCC-10.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node374.hydra.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz (skylake_avx512), Python 2.7.5
See https://gist.github.com/0390235f814090451dcfefc8d36b3feb for a full test report.

@smoors
Copy link
Copy Markdown
Contributor

smoors commented Apr 27, 2021

looks fine to me, --no-index added by default for PythonBundle easyconfigs:

== installing extension pypblib 0.0.4 (1/2)...
  >> defining build environment for GCC/10.2.0 toolchain
  >> running command:
        [started at: 2021-04-27 10:32:10]
        [working dir: /tmp/3871048.master01.hydra.brussel.vsc/PySAT/0.1.7.dev1/GCC-10.2.0/pypblib]
        [output logged in /dev/shm/vsc10009/eb-vVY3wq/easybuild-run_cmd-nUv2yN.log]
        tar xzf /apps/brussel/sources/p/PySAT/extensions/pypblib-0.0.4.tar.gz
  >> command completed: exit 0, ran in < 1s
==      configuring...
==      building...
==      testing...
==      installing...
  >> running command:
        [started at: 2021-04-27 10:32:13]
        [working dir: /tmp/3871048.master01.hydra.brussel.vsc/PySAT/0.1.7.dev1/GCC-10.2.0/pypblib/pypblib-0.0.4]
        [output logged in /dev/shm/vsc10009/eb-vVY3wq/easybuild-run_cmd-EELNYJ.log]
        pip install --prefix=/scratch/brussel/vo/000/bvo00005/vsc10009/ebtest/pysat/software/PySAT/0.1.7.dev1-GCC-10.2.0  --no-deps  --ignore-installed  --no-index  --no-build-isolation  .
  >> command completed: exit 0, ran in 00h01m00s
== installing extension python-sat 0.1.7.dev1 (2/2)...
  >> defining build environment for GCC/10.2.0 toolchain
  >> running command:
        [started at: 2021-04-27 10:33:14]
        [working dir: /tmp/3871048.master01.hydra.brussel.vsc/PySAT/0.1.7.dev1/GCC-10.2.0/pythonsat]
        [output logged in /dev/shm/vsc10009/eb-vVY3wq/easybuild-run_cmd-vWe_2U.log]
        tar xzf /scratch/brussel/100/vsc10009/sources/p/PySAT/extensions/python-sat-0.1.7.dev1.tar.gz
  >> command completed: exit 0, ran in < 1s
==      configuring...
==      building...
==      testing...
==      installing...
  >> running command:
        [started at: 2021-04-27 10:33:15]
        [working dir: /tmp/3871048.master01.hydra.brussel.vsc/PySAT/0.1.7.dev1/GCC-10.2.0/pythonsat/python-sat-0.1.7.dev1]
        [output logged in /dev/shm/vsc10009/eb-vVY3wq/easybuild-run_cmd-gKhGEC.log]
        pip install --prefix=/scratch/brussel/vo/000/bvo00005/vsc10009/ebtest/pysat/software/PySAT/0.1.7.dev1-GCC-10.2.0  --no-deps  --ignore-installed  --no-index  --no-build-isolation  .
  >> command completed: exit 0, ran in 00h02m49s

@smoors smoors added this to the next release (4.3.5?) milestone Apr 27, 2021
smoors
smoors approved these changes Apr 27, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@smoors smoors left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@smoors
Copy link
Copy Markdown
Contributor

smoors commented Apr 27, 2021

Going in, thanks @Flamefire!

@smoors smoors merged commit 1a6abe2 into easybuilders:develop Apr 27, 2021
@Flamefire Flamefire deleted the pipNoIndex branch April 27, 2021 09:03
@boegel boegel changed the title Add option to disable connecting to PyPi from pip Add option to disable connecting to PyPi from pip via --no-index Apr 28, 2021
@boegel boegel changed the title Add option to disable connecting to PyPi from pip via --no-index Add option to disable pip connecting to PyPi (enable use of --no-index) Apr 28, 2021
@boegel boegel mentioned this pull request May 5, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@smoors smoors smoors approved these changes

Assignees

No one assigned

Labels

change enhancement

Projects

None yet

Milestone

4.4.0

Development

Successfully merging this pull request may close these issues.

3 participants

@Flamefire @akesandgren @smoors