enhance TensorFlow easyblock to take into account provided OpenSSL dependency#2575
Conversation
|
Test report by @SebastianAchilles Overview of tested easyconfigs (in order)
Build succeeded for 0 out of 1 (1 easyconfigs in total) |
| if openssl_root: | ||
| if 'boringssl' not in system_libs: | ||
| system_libs.append('boringssl') | ||
| incpath = os.path.join(openssl_root, 'include') |
There was a problem hiding this comment.
I think the path should be
| incpath = os.path.join(openssl_root, 'include') | |
| incpath = os.path.join(openssl_root, 'include','openssl') |
There was a problem hiding this comment.
This seems to be wrong. I get a similar error with this include path.
There was a problem hiding this comment.
No, the path is correct: #include <openssl/bio.h>
I see /p/project/cswmanage/jurecadc/stages/Upstream/software/OpenSSL/1.1/include in the CPATH. So doesn't that contain the above file?
There was a problem hiding this comment.
Yes the path os.path.join(openssl_root, 'include') is correct. And all the header files are there:
$ ls /p/project/cswmanage/jurecadc/stages/Upstream/software/OpenSSL/1.1/include/openssl
aes.h blowfish.h cms.h ct.h ecdsa.h fips.h md5.h opensslconf-x86_64.h pkcs7.h rsa.h ssl.h uierr.h
asn1err.h bnerr.h comperr.h des.h ecerr.h fips_rand.h mdc2.h opensslv.h rand_drbg.h safestack.h stack.h ui.h
asn1.h bn.h comp.h dherr.h ec.h hmac.h modes.h ossl_typ.h randerr.h seed.h storeerr.h whrlpool.h
asn1_mac.h buffererr.h conf_api.h dh.h engineerr.h idea.h objectserr.h pem2.h rand.h sha.h store.h x509err.h
asn1t.h buffer.h conferr.h dsaerr.h engine.h kdferr.h objects.h pemerr.h rc2.h srp.h symhacks.h x509.h
asyncerr.h camellia.h conf.h dsa.h e_os2.h kdf.h obj_mac.h pem.h rc4.h srtp.h tls1.h x509v3err.h
async.h cast.h cryptoerr.h dtls1.h err.h lhash.h ocsperr.h pkcs12err.h rc5.h ssl2.h tserr.h x509v3.h
bioerr.h cmac.h crypto.h ebcdic.h evperr.h md2.h ocsp.h pkcs12.h ripemd.h ssl3.h ts.h x509_vfy.h
bio.h cmserr.h cterr.h ecdh.h evp.h md4.h opensslconf.h pkcs7err.h rsaerr.h sslerr.h txt_db.h
Also openssl/bio.h is in there. So I do not understand yet why I am still getting this error.
There was a problem hiding this comment.
No idea either. Something with access rights maybe? Note that you can rerun the failing commands if you still have the build tree. E.g. from the log take EVERYTHING inside the parens and run it in a shell (or shell script if it doesn't past properly):
(cd /dev/shm/achilles4/TensorFlow/2.6.0/foss-2021a/tmpm93y4b56-bazel-tf/40866da151b2fe950b6c8b274be00c2f/execroot/org_tensorflow && \
exec env - \
CPATH=/p/project/cswmanage/jurecadc/stages/Upstream/software/cURL/7.76.0-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/double-conversion/3.1.5-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/flatbuffers/2.0.0-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/giflib/5.2.1-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/hwloc/2.4.1-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/ICU/69.1-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/JsonCpp/1.9.4-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/libjpeg-turbo/2.0.6-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/libpng/1.6.37-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/LMDB/0.9.28-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/nsync/1.24.0-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/protobuf/3.17.3-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/pybind11/2.6.2-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/snappy/1.1.8-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/SQLite/3.35.4-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/zlib/1.2.11-GCCcore-10.3.0/include:/p/project/cswmanage/jurecadc/stages/Upstream/software/OpenSSL/1.1/include \
LD_LIBRARY_PATH=/p/project/cswmanage/jurecadc/stages/Upstream/software/snappy/1.1.8-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/libpng/1.6.37-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/nsync/1.24.0-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/LMDB/0.9.28-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/libjpeg-turbo/2.0.6-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/JsonCpp/1.9.4-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/ICU/69.1-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/giflib/5.2.1-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/flatbuffers/2.0.0-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/double-conversion/3.1.5-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/HDF5/1.10.7-gompi-2021a/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/Szip/2.1.1-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/SciPy-bundle/2021.05-foss-2021a/lib/python3.9/site-packages/numpy/core/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/Python/3.9.5-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/libffi/3.3-GCCcore-10.3.0/lib64:/p/project/cswmanage/jurecadc/stages/Upstream/software/GMP/6.2.1-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/SQLite/3.35.4-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/Tcl/8.6.11-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/bzip2/1.0.8-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/DB/18.1.40-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/libreadline/8.1-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/gettext/0.21-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/ncurses/6.2-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/expat/2.2.9-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/cURL/7.76.0-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/protobuf/3.17.3-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/Java/11.0.2/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/ScaLAPACK/2.1.0-gompi-2021a-fb/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/FFTW/3.3.9-gompi-2021a/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/FlexiBLAS/3.0.4-GCC-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/OpenBLAS/0.3.15-GCC-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/OpenMPI/4.1.1-GCC-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/PMIx/3.2.3-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/libfabric/1.12.1-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/UCX/1.10.0-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/libevent/2.1.12-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/OpenSSL/1.1/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/hwloc/2.4.1-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/libpciaccess/0.16-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/libxml2/2.9.10-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/XZ/5.2.5-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/numactl/2.0.14-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/binutils/2.36.1-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/zlib/1.2.11-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/GCCcore/10.3.0/lib64 \
LIBRARY_PATH=/p/project/cswmanage/jurecadc/stages/Upstream/software/cURL/7.76.0-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/double-conversion/3.1.5-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/flatbuffers/2.0.0-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/giflib/5.2.1-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/hwloc/2.4.1-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/ICU/69.1-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/JsonCpp/1.9.4-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/libjpeg-turbo/2.0.6-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/libpng/1.6.37-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/LMDB/0.9.28-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/nsync/1.24.0-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/protobuf/3.17.3-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/pybind11/2.6.2-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/snappy/1.1.8-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/SQLite/3.35.4-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/zlib/1.2.11-GCCcore-10.3.0/lib:/p/project/cswmanage/jurecadc/stages/Upstream/software/OpenSSL/1.1/lib \
PATH=/p/project/cswmanage/jurecadc/stages/Upstream/software/libpng/1.6.37-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/LMDB/0.9.28-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/libjpeg-turbo/2.0.6-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/NASM/2.15.05-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/ICU/69.1-GCCcore-10.3.0/sbin:/p/project/cswmanage/jurecadc/stages/Upstream/software/ICU/69.1-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/giflib/5.2.1-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/flatbuffers/2.0.0-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/HDF5/1.10.7-gompi-2021a/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/SciPy-bundle/2021.05-foss-2021a/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/UnZip/6.0-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/pybind11/2.6.2-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/Python/3.9.5-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/SQLite/3.35.4-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/Tcl/8.6.11-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/bzip2/1.0.8-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/git/2.32.0-GCCcore-10.3.0-nodocs/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/Perl/5.32.1-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/DB/18.1.40-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/gettext/0.21-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/ncurses/6.2-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/expat/2.2.9-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/cURL/7.76.0-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/protobuf/3.17.3-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/Bazel/3.7.2-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/Java/11.0.2:/p/project/cswmanage/jurecadc/stages/Upstream/software/Java/11.0.2/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/EasyBuild/20210914-dev/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/FFTW/3.3.9-gompi-2021a/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/FlexiBLAS/3.0.4-GCC-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/OpenMPI/4.1.1-GCC-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/PMIx/3.2.3-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/libfabric/1.12.1-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/UCX/1.10.0-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/libevent/2.1.12-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/OpenSSL/1.1/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/hwloc/2.4.1-GCCcore-10.3.0/sbin:/p/project/cswmanage/jurecadc/stages/Upstream/software/hwloc/2.4.1-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/libxml2/2.9.10-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/XZ/5.2.5-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/numactl/2.0.14-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/binutils/2.36.1-GCCcore-10.3.0/bin:/p/project/cswmanage/jurecadc/stages/Upstream/software/GCCcore/10.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ddn/ime/bin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/p/software/jurecadc/bin:/p/home/jusers/achilles4/jureca/.local/bin:/p/home/jusers/achilles4/jureca/achilles4/Repos/JSC/easybuild-repository/bin:/opt/ddn/ime/bin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/p/software/jurecadc/bin:/p/home/jusers/achilles4/jureca/.local/bin:/p/home/jusers/achilles4/jureca/achilles4/Repos/JSC/easybuild-repository/bin:/opt/ddn/ime/bin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/p/software/jurecadc/bin:/opt/parastation/bin:/p/home/jusers/achilles4/jureca/.local/bin:/p/home/jusers/achilles4/jureca/achilles4/Repos/JSC/easybuild-repository/bin \
PWD=/proc/self/cwd \
PYTHONNOUSERSITE=1 \
PYTHONPATH=/p/project/cswmanage/jurecadc/stages/Upstream/software/TensorFlow/2.6.0-foss-2021a/lib/python3.9/site-packages:/p/project/cswmanage/jurecadc/stages/Upstream/software/TensorFlow/2.6.0-foss-2021a/lib/python3.9/site-packages:/p/project/cswmanage/jurecadc/stages/Upstream/software/typing-extensions/3.10.0.0-GCCcore-10.3.0/lib/python3.9/site-packages:/p/project/cswmanage/jurecadc/stages/Upstream/software/flatbuffers-python/2.0-GCCcore-10.3.0/lib/python3.9/site-packages:/p/project/cswmanage/jurecadc/stages/Upstream/software/protobuf-python/3.17.3-GCCcore-10.3.0/lib/python3.9/site-packages:/p/project/cswmanage/jurecadc/stages/Upstream/software/h5py/3.2.1-foss-2021a/lib/python3.9/site-packages:/p/project/cswmanage/jurecadc/stages/Upstream/software/SciPy-bundle/2021.05-foss-2021a/lib/python3.9/site-packages:/p/project/cswmanage/jurecadc/stages/Upstream/software/pybind11/2.6.2-GCCcore-10.3.0/lib/python3.9/site-packages:/p/project/cswmanage/jurecadc/stages/Upstream/software/Python/3.9.5-GCCcore-10.3.0/easybuild/python:/p/project/cswmanage/jurecadc/stages/Upstream/software/EasyBuild/20210914-dev/lib/python3.6/site-packages \
PYTHON_BIN_PATH=/p/project/cswmanage/jurecadc/stages/Upstream/software/Python/3.9.5-GCCcore-10.3.0/bin/python \
PYTHON_LIB_PATH=/p/project/cswmanage/jurecadc/stages/Upstream/software/TensorFlow/2.6.0-foss-2021a/lib/python3.9/site-packages \
TF2_BEHAVIOR=1 \
TF_SYSTEM_LIBS=absl_py,astor_archive,astunparse_archive,boringssl,com_google_protobuf,curl,cython,dill_archive,double_conversion,enum34_archive,flatbuffers,functools32_archive,gast_archive,gif,hwloc,icu,jsoncpp_git,libjpeg_turbo,lmdb,nasm,nsync,opt_einsum_archive,org_sqlite,pasta,png,pybind11,six_archive,snappy,tblib_archive,termcolor_archive,typing_extensions_archive,wrapt,zlib \
/p/project/cswmanage/jurecadc/stages/Upstream/software/GCCcore/10.3.0/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++11' -MD -MF bazel-out/k8-opt/bin/external/com_github_grpc_grpc/_objs/alts_frame_protector/aes_gcm.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/com_github_grpc_grpc/_objs/alts_frame_protector/aes_gcm.pic.o' -fPIC '-DGRPC_ARES=0' -iquoteexternal/com_github_grpc_grpc -iquotebazel-out/k8-opt/bin/external/com_github_grpc_grpc -iquoteexternal/com_google_absl -iquotebazel-out/k8-opt/bin/external/com_google_absl -iquoteexternal/zlib -iquotebazel-out/k8-opt/bin/external/zlib -iquoteexternal/boringssl -iquotebazel-out/k8-opt/bin/external/boringssl -isystem external/com_github_grpc_grpc/include -isystem bazel-out/k8-opt/bin/external/com_github_grpc_grpc/include -isystem external/com_github_grpc_grpc/src/core/ext/upb-generated -isystem bazel-out/k8-opt/bin/external/com_github_grpc_grpc/src/core/ext/upb-generated -w -DAUTOLOAD_DYNAMIC_KERNELS -O2 -ftree-vectorize '-march=native' -fno-math-errno -fPIC -fPIC '-std=c++14' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/com_github_grpc_grpc/src/core/tsi/alts/crypt/aes_gcm.cc -o bazel-out/k8-opt/bin/external/com_github_grpc_grpc/_objs/alts_frame_protector/aes_gcm.pic.o)
|
@akesandgren As there is no further update, can we merge this? |
|
I think you should ask @SebastianAchilles , I was just trying to get him to react... |
|
Well this was 5 months ago. And as the code is correct as far as I can tell (add the include/library paths for OpenSSL when used) I'd suggest to merge this and improve on it if it later turns out to be required. But I don't think so according to the tests I've done at the time. |
|
Going in, thanks @Flamefire! |
(created using
eb --new-pr)