sacrifice download progress report in log to restore support for downloading over a proxy (i.e. use urllib2 rather than urllib)#1158
Conversation
This add support for downloading with proxies as often found in corporate settings. Pythons urllib2 will do "the right thing" when common environment variables such as `http_proxy` or `https_proxy` are set. Most noteably this also removes the `reporthook` nested function and removes a lot of code from the `download_file` function.
This add support for downloading with proxies as often found in corporate settings. Pythons urllib2 will do "the right thing" when common environment variables such as `http_proxy` or `https_proxy` are set. Most noteably this also removes the `reporthook` nested function and removes a lot of code from the `download_file` function.
…ure-proxy-download-support Conflicts: easybuild/tools/filetools.py
…oxies into account
…tion for specifying different timeout
|
@wpoely86: up for reviewing this? |
|
Refer to this link for build results (access rights to CI server needed): |
… reporting, fix broken unit tests
|
Refer to this link for build results (access rights to CI server needed): |
|
Refer to this link for build results (access rights to CI server needed): |
|
This has been well-tested by re-downloading all required sources for an R installation with batteries included (300+ extensions). As far as I can tell, the proxy support should now be restored, and the unit test that's in place now should prevent us from breaking it again without being aware... Thanks a lot for initiating this @serverhorror, going in! |
sacrifice download progress report in log to restore support for downloading over a proxy (i.e. use urllib2 rather than urllib)
This is a fork of @serverhorror's PR #1150. I took things over after @serverhorror informed me he wouldn't be able to follow up on this in the coming days/weeks.
On top of PR #1150, this includes a unit test that attempts to verify that
download_fileis taking proxy settings into account, and further cleans up the mess indownload_fileitself. It also adds a dedicated configuration option for specifying the timeout used byurllib2.urlopen.I was informed that downloading over a proxy was supported in an earlier version of EasyBuild, sometime around v1.13, but it was unintentionally broken recently.
Just like PR #1150, this thrashes the download progress report that was added in #1066 by @JensTimmerman, since
urllib2doesn't provide a report hook.Although it would be possible to implement a progress report, it's probably not worth the hassle (cfr. http://stackoverflow.com/questions/2028517/python-urllib2-progress-hook/2030027#2030027).
@JensTimmerman already mentioned he was OK with dropping it in #1150.