{phys}[foss/2019a] TRIQS v2.2.0 w/ Python 2.7.15#8835
{phys}[foss/2019a] TRIQS v2.2.0 w/ Python 2.7.15#8835boegel merged 8 commits intoeasybuilders:developfrom rokzitko:TRIQS
Conversation
boegel
left a comment
There was a problem hiding this comment.
Thanks a lot for your contribution @rokzitko!
There are a couple of minor improvements that should be made to make the tests happy, and to ensure your easyconfig file can be maintained/updated easily in the future.
Please see my comments & suggestions, do let us know if you have any questions!
Fixed order of settings Fixed too long line Fixed indentation Python is now included in versionsuffix Library suffix using SHLIB_EXT
|
Thank you for the comments and explanations, much appreciated! I’ve updated the easyconfig accordingly.
Kind regards,
Rok
… On 26 Aug 2019, at 15:56, Kenneth Hoste ***@***.***> wrote:
@boegel requested changes on this pull request.
Thanks a lot for your contribution @rokzitko <https://github.com/rokzitko>!
There are a couple of minor improvements that should be made to make the tests happy, and to ensure your easyconfig file can be maintained/updated easily in the future.
Please see my comments & suggestions, do let us know if you have any questions!
In easybuild/easyconfigs/t/TRIQS/TRIQS-2.1.1-foss-2018b.eb <#8835 (comment)>:
> +
+homepage = 'https://triqs.github.io/'
+description = """
+ TRIQS (Toolbox for Research on Interacting Quantum Systems) is a
+ scientific project providing a set of C++ and Python libraries to
+ develop new tools for the study of interacting quantum systems.
+"""
+
+docurls = ['https://triqs.github.io/triqs/2.1.x/reference.html']
+software_license = 'LicenseGPLv3'
+
+toolchain = {'name': 'foss', 'version': '2018b'}
+toolchainopts = {'pic': True, 'usempi': True}
+
+sources = ['%(version)s.tar.gz']
+source_urls = ['https://github.com/TRIQS/triqs/archive/']
@rokzitko <https://github.com/rokzitko> Please add a sha256 checksum; you can let the eb command take care of that for you using eb TRIQS-2.1.1-foss-2018b.eb --inject-checksums
In easybuild/easyconfigs/t/TRIQS/TRIQS-2.1.1-foss-2018b.eb <#8835 (comment)>:
> +software_license = 'LicenseGPLv3'
+
+toolchain = {'name': 'foss', 'version': '2018b'}
+toolchainopts = {'pic': True, 'usempi': True}
+
+sources = ['%(version)s.tar.gz']
+source_urls = ['https://github.com/TRIQS/triqs/archive/']
+
+dependencies = [
+ ('Python', '2.7.15'),
+ ('Boost', '1.68.0', '-Python-2.7.15'),
+ ('Clang', '7.0.1'),
+ ('GMP', '6.1.2'),
+ ('Mako', '1.0.7', '-Python-2.7.15'),
+ ('h5py', '2.8.0', '-Python-2.7.15'),
+ ('CMake', '3.12.1')
CMake is most likely only required to build TRIQS, it's not required to use it?
If so, make CMake a build-only dependency:
builddependencies = [
('CMake', '3.12.1'),
]
In easybuild/easyconfigs/t/TRIQS/TRIQS-2.1.1-foss-2018b.eb <#8835 (comment)>:
> + ('Boost', '1.68.0', '-Python-2.7.15'),
+ ('Clang', '7.0.1'),
+ ('GMP', '6.1.2'),
+ ('Mako', '1.0.7', '-Python-2.7.15'),
+ ('h5py', '2.8.0', '-Python-2.7.15'),
+ ('CMake', '3.12.1')
+]
+
+sanity_check_paths = {
+ 'files': ['lib/libtriqs.so'],
+ 'dirs': ['bin', 'include', 'lib', 'share'],
+}
+
+separate_build_dir = True
+
+runtest = 'test'
Style nitpicking: separate_build_dir = True and runtest = 'test' usually go above sanity_check_paths
In easybuild/easyconfigs/t/TRIQS/TRIQS-2.1.1-foss-2018b.eb <#8835 (comment)>:
> + ('GMP', '6.1.2'),
+ ('Mako', '1.0.7', '-Python-2.7.15'),
+ ('h5py', '2.8.0', '-Python-2.7.15'),
+ ('CMake', '3.12.1')
+]
+
+sanity_check_paths = {
+ 'files': ['lib/libtriqs.so'],
+ 'dirs': ['bin', 'include', 'lib', 'share'],
+}
+
+separate_build_dir = True
+
+runtest = 'test'
+
+modextrapaths = {'PYTHONPATH': 'lib/python2.7/site-packages', 'CMAKE_PREFIX_PATH': ['lib/cmake/triqs', 'lib/cmake/cpp2py']}
This line is too long, and we like to have each key on a separate line anyway, so:
modextrapaths = {
'PYTHONPATH': 'lib/python%(pyshortver)s/site-packages',
'CMAKE_PREFIX_PATH': ['lib/cmake/triqs', 'lib/cmake/cpp2py'],
}
In easybuild/easyconfigs/t/TRIQS/TRIQS-2.1.1-foss-2018b.eb <#8835 (comment)>:
> + TRIQS (Toolbox for Research on Interacting Quantum Systems) is a
+ scientific project providing a set of C++ and Python libraries to
+ develop new tools for the study of interacting quantum systems.
+"""
+
+docurls = ['https://triqs.github.io/triqs/2.1.x/reference.html']
+software_license = 'LicenseGPLv3'
+
+toolchain = {'name': 'foss', 'version': '2018b'}
+toolchainopts = {'pic': True, 'usempi': True}
+
+sources = ['%(version)s.tar.gz']
+source_urls = ['https://github.com/TRIQS/triqs/archive/']
+
+dependencies = [
+ ('Python', '2.7.15'),
Style check fails here, probably you're not indenting with 4 spaces (but just two), so please fix?
In easybuild/easyconfigs/t/TRIQS/TRIQS-2.1.1-foss-2018b.eb <#8835 (comment)>:
> + scientific project providing a set of C++ and Python libraries to
+ develop new tools for the study of interacting quantum systems.
+"""
+
+docurls = ['https://triqs.github.io/triqs/2.1.x/reference.html']
+software_license = 'LicenseGPLv3'
+
+toolchain = {'name': 'foss', 'version': '2018b'}
+toolchainopts = {'pic': True, 'usempi': True}
+
+sources = ['%(version)s.tar.gz']
+source_urls = ['https://github.com/TRIQS/triqs/archive/']
+
+dependencies = [
+ ('Python', '2.7.15'),
+ ('Boost', '1.68.0', '-Python-2.7.15'),
If Python is included as a dependency, please indicate so by using the following versionsuffix (right below where you define version), so:
version = '2.1.1'
versionsuffix = '-Python-%(pyver)s'
The %(pyver)s part is a template value, which gets automatically replaced with the Python version included as a dependency.
Any dependencies that have the same versionsuffix can then leverage versionsuffix, so:
dependencies = [
('Python', '2.7.15'),
('Boost', '1.68.0', versionsuffix),
...
In general, you should avoid hardcoding things like the Python version in multiple places: just specify the version once in dependencies (like you did above), and then use the template values like %(pyver)s and %(pyshortver)s, see https://easybuild.readthedocs.io/en/latest/version-specific/easyconfig_templates.html#avail-easyconfig-templates <https://easybuild.readthedocs.io/en/latest/version-specific/easyconfig_templates.html#avail-easyconfig-templates>
In easybuild/easyconfigs/t/TRIQS/TRIQS-2.1.1-foss-2018b.eb <#8835 (comment)>:
> +
+sources = ['%(version)s.tar.gz']
+source_urls = ['https://github.com/TRIQS/triqs/archive/']
+
+dependencies = [
+ ('Python', '2.7.15'),
+ ('Boost', '1.68.0', '-Python-2.7.15'),
+ ('Clang', '7.0.1'),
+ ('GMP', '6.1.2'),
+ ('Mako', '1.0.7', '-Python-2.7.15'),
+ ('h5py', '2.8.0', '-Python-2.7.15'),
+ ('CMake', '3.12.1')
+]
+
+sanity_check_paths = {
+ 'files': ['lib/libtriqs.so'],
Please avoid hardcoding .so (since on macOS it's actually .dylib), use the SHLIB_EXT constant instead:
sanity_check_paths = {
'files': ['lib/libtriqs.%s' % SHLIB_EXT],
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#8835?email_source=notifications&email_token=AG57NBD72BK6FLNA4RYQGXLQGPOJFA5CNFSM4IPOUZWKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCCVGBEY#pullrequestreview-279601299>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG57NBEJRASUHOME2CCNJGTQGPOJFANCNFSM4IPOUZWA>.
|
|
I think all the issues have been address and that the easyconfig for TRIQS is ready to be merged. |
|
Test report by @boegel |
2. switched to %(pyshortver)s 3. added sanity_check_commands = ["python -c 'import pytriqs'"]
|
Test report by @boegel |
|
Test report by @boegel |
|
@rokzitko Latest test reports show that the tests work just fine on Intel Haswell and Broadwell, so the problem seems specific to Intel Skylake. If this is a known problem with TRIQS and there's an open issue to track it, I guess this doesn't need to block this PR... |
|
I’ve tested on our Sandy Bridge, as well as on Opteron 63xx and Epyc 1 machines, and the tests work fine. Furthermore, Nils Wentzell reports that he was not able to reproduce the problems on his Intel Skylake.
Could it be specific to the configuration of that particular Skylake box? Can you rerun it to see if it is reproducible? Could it be a one-time quirk?
… On 10 Sep 2019, at 09:08, Kenneth Hoste ***@***.***> wrote:
@rokzitko <https://github.com/rokzitko> Latest test reports show that the tests work just fine on Intel Haswell and Broadwell, so the problem seems specific to Intel Skylake.
Any idea what the cause could be?
If this is a known problem with TRIQS and there's an open issue to track it, I guess this doesn't need to block this PR...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#8835?email_source=notifications&email_token=AG57NBDXZMZIQB54A3CDL7TQI5BVJA5CNFSM4IPOUZWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6KCOLQ#issuecomment-529803054>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG57NBB7YJKHDVOFRKO5P53QI5BVJANCNFSM4IPOUZWA>.
|
|
@boegel the same two tests failed for me on skylake |
|
ok, this should be investigated then.
… On 10 Sep 2019, at 13:48, Miguel Dias Costa ***@***.***> wrote:
@boegel <https://github.com/boegel> the same two tests failed for me on skylake
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#8835?email_source=notifications&email_token=AG57NBE2MVI6O2L7EDP4EB3QI6CRJA5CNFSM4IPOUZWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6KZXUQ#issuecomment-529898450>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG57NBGW3RZTMADZMZ7VWILQI6CRJANCNFSM4IPOUZWA>.
|
|
This should indeed be investigated! Unfortunately I have no experience with EasyBuild and cannot reproduce the problem on my machine. Could you trigger the TRIQS library a build with |
|
@Wentzell Tried that, is this the information you're looking for? Output for two failing tests: |
|
@boegel Thank you for rerunning the build! |
|
We were able to trace this problem back to hdf5 easyblock = 'CMakeMake'
name = 'TRIQS'
version = '2.2-rc1'
versionsuffix = '-Python-%(pyver)s'
homepage = 'https://triqs.github.io/'
description = """
TRIQS (Toolbox for Research on Interacting Quantum Systems) is a
scientific project providing a set of C++ and Python libraries to
develop new tools for the study of interacting quantum systems.
"""
docurls = ['https://triqs.github.io/triqs/%(version_major_minor)s.x/reference.html']
software_license = 'LicenseGPLv3'
toolchain = {'name': 'foss', 'version': '2019a'}
toolchainopts = {'pic': True, 'usempi': True}
sources = ['%(version)s.tar.gz']
source_urls = ['https://github.com/TRIQS/triqs/archive/']
checksums = ['cf4ce7ea7ba873a5bbb3fd53aa544106dc7e76970d5962ba19e27c240b5b1641']
dependencies = [
('Python', '2.7.15'),
('Boost', '1.70.0'),
('Clang', '8.0.0'),
('GMP', '6.1.2'),
('HDF5', '1.10.5'),
('Mako', '1.0.8'),
('h5py', '2.9.0')
]
builddependencies = [
('CMake', '3.13.3')
]
separate_build_dir = True
runtest = 'test'
sanity_check_paths = {
'files': ['lib/libtriqs.%s' % SHLIB_EXT],
'dirs': ['bin', 'include', 'lib', 'share'],
}
sanity_check_commands = ["python -c 'import pytriqs'"]
modextrapaths = {
'PYTHONPATH': 'lib/python%(pyshortver)s/site-packages',
'CMAKE_PREFIX_PATH': ['lib/cmake/triqs', 'lib/cmake/cpp2py']
}
modextravars = {'TRIQS_ROOT': '%(installdir)s'}
moduleclass = 'phys'The reason we are no longer building the triqs |
|
We have just tagged the @rokzitko Could you update the pull request with the following foss2019a based eb script easyblock = 'CMakeMake'
name = 'TRIQS'
version = '2.2.0'
versionsuffix = '-Python-%(pyver)s'
homepage = 'https://triqs.github.io/'
description = """
TRIQS (Toolbox for Research on Interacting Quantum Systems) is a
scientific project providing a set of C++ and Python libraries to
develop new tools for the study of interacting quantum systems.
"""
docurls = ['https://triqs.github.io/triqs/%(version_major_minor)s.x/reference.html']
software_license = 'LicenseGPLv3'
toolchain = {'name': 'foss', 'version': '2019a'}
toolchainopts = {'pic': True, 'usempi': True}
sources = ['%(version)s.tar.gz']
source_urls = ['https://github.com/TRIQS/triqs/archive/']
checksums = ['5f7fca3dd140d86e464ab18ff91fa16febad865344ec2c5b924434f507ef4038']
dependencies = [
('Python', '2.7.15'),
('Boost', '1.70.0'),
('Clang', '8.0.0'),
('GMP', '6.1.2'),
('HDF5', '1.10.5'),
('Mako', '1.0.8'),
('h5py', '2.9.0')
]
builddependencies = [
('CMake', '3.13.3')
]
separate_build_dir = True
runtest = 'test'
sanity_check_paths = {
'files': ['lib/libtriqs.%s' % SHLIB_EXT],
'dirs': ['bin', 'include', 'lib', 'share'],
}
sanity_check_commands = ["python -c 'import pytriqs'"]
modextrapaths = {
'PYTHONPATH': 'lib/python%(pyshortver)s/site-packages',
'CMAKE_PREFIX_PATH': ['lib/cmake/triqs', 'lib/cmake/cpp2py']
}
modextravars = {'TRIQS_ROOT': '%(installdir)s'}
moduleclass = 'phys' |
|
Test report by @boegel |
boegel
left a comment
There was a problem hiding this comment.
Just a minor suggestion to make the sanity check a bit more specific...
Other than that, this looks good to go, last test report shows that tests pass on Intel Skylake now with TRIQS 2.2.0 :)
|
|
||
| sanity_check_paths = { | ||
| 'files': ['lib/libtriqs.%s' % SHLIB_EXT], | ||
| 'dirs': ['bin', 'include', 'lib', 'share'], |
There was a problem hiding this comment.
@rokzitko I think it's worth enhancing this a little bit, to check for some specific binaries, and also check for lib/python%(pyshortver)s/site-packages rather than just lib:
sanity_check_paths = {
'files': ['bin/triqs++', 'lib/libtriqs.%s' % SHLIB_EXT],
'dirs': ['include/triqs', 'lib/python%(pyshortver)s/site-packages', 'share'],
}Maybe it's also worth adding triqs++ --help as an additional sanity check command?
sanity_check_commands = [
"triqs++ --help",
"python -c 'import pytriqs'",
]|
On 3 Oct 2019, at 17:03, Nils Wentzell ***@***.***> wrote:
I have updated the easyconfig for TRIQS. What is the status with various TRIQS apps? Will there be 2.2.0 releases soon? I have easyconfigs for 2.1.1 releases of CTHYB, DFTTools and TPRF (as well as NFFT as dependency) that I can easily update as well.
We hope to tag the applications as soon as possible!
Do you already have easybuild scripts for those that we could add additional pull requests for once the tags are down?
Yes, I have the easyconfigs for 2.1.1 and they work fine. I’ll wait for 2.2.0 tags, update the easyconfigs, and make the pull requests.
|
|
I will add some more specific sanity checks, but more importantly, I now realized that there is a hidden run-time dependency, namely c++2py requires clang.cindex (typically installed via pip install clang). What is the best way to handle this dependency?
|
|
@rokzitko The best way to handle that is to create an additional easyconfig file for the Let us know if you need help with that. It's probably best to add that easyconfig through a separate PR... |
|
Yes, I think I’ll need help with this one. It turns out that “pip install clang” fetches python bindings that were bundled with clang 6.0, but were repackaged as a stand-alone package (here: https://pypi.org/project/clang/ <https://pypi.org/project/clang/>). It’s an unofficial package, which does seem to work with more recent versions of clang, too.
Generally, one should use the bindings which are bundled with the clang sources, but are typically not installed during clang builds, it seems, and this holds true for Clang-8.0.0 easyconfig in EB. One solution would be to fix the clang easyconfig so that it installs the binding as well.
… On 4 Oct 2019, at 11:24, Kenneth Hoste ***@***.***> wrote:
@rokzitko <https://github.com/rokzitko> The best way to handle that is to create an additional easyconfig file for the clang Python bindings (obviously with Clang as a dependency in there), using the PythonPackage easyblock (or PythonBundle if clang has additional dependencies).
Let us know if you need help with that. It's probably best to add that easyconfig through a separate PR...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#8835?email_source=notifications&email_token=AG57NBF4EG6UIGOGCVZ5MBDQM4DWDA5CNFSM4IPOUZWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEALCOHA#issuecomment-538322716>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG57NBEFFPI63YCYWBOYAJ3QM4DWDANCNFSM4IPOUZWA>.
|
|
@rokzitko We can certainly look into enabling the Python bindings in the Is there any documentation on how to build the |
|
@boegel The libclang python bindings do not need any compilation. When it comes to compatibility I am not quite sure which versions of llvm/clang the clang pypi-package is compatible against. I just compared the latest sources at https://github.com/llvm/llvm-project/tree/llvmorg-9.0.0/clang/bindings/python/clang against the ones contained in the pypi-package (6.0.0) and they seem to be largely identical. Maybe you could just package the python bindings together with |
|
I believe the installation of python bindings can be enabled during the llvm build by adding e.g. |
|
@Wentzell The fact that a specific Python version needs to be specified implies that a I wonder if there's an easy way to only install the Python bindings from a Clang source tarball (not using the unofficial PyPI package, it seems like that is likely to end in tears sooner or later)... It seems like just copying the contents of |
|
Yes, I think this would be a sufficient (and possibly cleanest) solution. Which easyblock is the most appropriate for doing such things (copying specific files from tarballs)?
… On 5 Oct 2019, at 12:08, Kenneth Hoste ***@***.***> wrote:
@Wentzell <https://github.com/Wentzell> The fact that a specific Python version needs to be specified implies that a Python dependency should be added, which I'm not a big fan of for Clang, for hopefully obvious reasons.
I wonder if there's an easy way to only install the Python bindings from a Clang source tarball (not using the unofficial PyPI package, it seems like that is likely to end in tears sooner or later)...
It seems like just copying the contents of /clang/bindings/python and updating $PYTHONPATH + setting $CLANG_LIBRARY_PATH could do it...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#8835?email_source=notifications&email_token=AG57NBAHYLC4MEQIMF3RCBLQNBRTRA5CNFSM4IPOUZWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEANPBGA#issuecomment-538636440>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG57NBEOTIQ6KNTG3LKWGOLQNBRTRANCNFSM4IPOUZWA>.
|
|
@rokzitko Probably For the environment variables, you can use |
|
The following seems to work (Clang-Python-bindings-8.0.0-GCCcore-8.2.0-Python-2.7.15.eb):
easyblock = 'Tarball'
name = 'Clang-Python-bindings'
version = '8.0.0'
versionsuffix = '-Python-%(pyver)s'
homepage = 'https://clang.llvm.org'
description = """Python bindings for libclang"""
toolchain = {'name': 'GCCcore', 'version': '8.2.0'}
source_urls = ["https://llvm.org/releases/%(version)s"]
sources = ['cfe-%(version)s.src.tar.xz']
dependencies = [
('Clang', version),
('Python', '2.7.15')
]
start_dir = 'bindings/python'
sanity_check_paths = {
'files': ['clang/cindex.py'],
'dirs': ['clang']
}
modextrapaths = {
'PYTHONPATH': ''
}
moduleclass = 'lib'
If this looks acceptable, I'll make a separate PR. This is then used as a dependency in TRIQS:
dependencies = [
('Python', '2.7.15'),
('SciPy-bundle', '2019.03'),
('Boost', '1.70.0'),
('Clang', '8.0.0'),
('Clang-Python-bindings', '8.0.0', '-Python-2.7.15', ('GCCcore', '8.2.0')),
('GMP', '6.1.2'),
('HDF5', '1.10.5'),
('Mako', '1.0.8'),
('h5py', '2.9.0')
]
(I've also added SciPy-bundle, since TRIQS requires numpy.)
I've also added a test for c++2py:
sanity_check_commands = [
"triqs++ --help",
"c++2py --help",
"python -c 'import pytriqs'"
]
|
|
@rokzitko No need to set I think You should also add an sanity_check_commands = ["python -c 'import clang'"]For the dependency specification, there's no need to hardcode the toolchain, this will work just fine: |
|
On 5 Oct 2019, at 18:22, Kenneth Hoste ***@***.***> wrote:
@rokzitko <https://github.com/rokzitko> No need to set $CLANG_LIBRARY_PATH to make the Clang Python bindings work as expected?
I think the path to libclang.so in LD_LIBRARY_PATH (set in Clang module) is sufficient.
I think clang as name should suffice, the -Python-* versionsuffix will make it clear that it's the Python bindings.
But wouldn't this suggest that this is clang compiler with Python bindings, rather than solely Python bindings for clang from another module?
Also, if I rename it to clang alone, I get the following error:
ERROR:150: Module 'Clang/8.0.0-GCCcore-8.2.0-Python-2.7.15' conflicts with the currently loaded module(s) 'Clang/8.0.0-GCCcore-8.2.0'
I thus propose to keep the clang-Python-bindings name, and reserve the name clang-x.x.x-Python-* for possible future easyconfig for clang with bundled bindings.
You should also add an import test:
sanity_check_commands = ["python -c 'import clang'"]
Ok.
For the dependency specification, there's no need to hardcode the toolchain, this will work just fine:
Ok.
|
|
@rokzitko I overlooked the name clash issue, so OK, let's go with |
|
Yes, I have just pushed the update.
|
|
Test report by @boegel |
|
Great. Thanks a lot @rokzitko for making this work! |
|
Test report by @boegel |
|
Going in, thanks @rokzitko! |
|
Test report by @boegel |
Adding support for a new package: TRIQS, Toolbox for Research on Interacting Quantum Systems
https://triqs.github.io/triqs/2.1.x/