build: Propagate well-known vars into depends#18820
Conversation
|
Ref #15914. |
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsNo conflicts as of last run. |
|
Concept ACK Having this merged would allow enabling MemorySanitizer (MSan) more easily (#18288). |
|
@dongcarl It seems like the Travis macOS 10.12 job failed :) |
|
Concept ACK |
As of Boost 1.72.0, the darwin toolset has been removed from Boost Build, in favour of the Clang toolset; see this commit: boostorg/build@eb8da52. The reasoning (not given in the commit) being that darwin was basically a copy of the clang toolset anyways. Some comments from the commit author here: boostorg/build#528. This change may also resolve the macOS issues seen in bitcoin#18820.
|
Tested a0e7fe7e11f061025396da6e588f6f3828a205e7 on Linux Mint 19.3: |
|
Concept ACK (assuming it solves my problems) |
|
Concept ACK. |
|
Seems like the issue is that we need to separate the |
a0e7fe7 to
645e080
Compare
|
Done, should work now. Glad to know that our build system is not the only one with weird idiosyncrasies 😬 |
|
ACK 645e0803d8b5e0a4e101649519ca7588852f9d3d -- patch looks correct and Travis is happy :) |
hebasto
left a comment
There was a problem hiding this comment.
Tested 645e0803d8b5e0a4e101649519ca7588852f9d3d on Linux Mint 19.3 (x86_64):
$ make -C depends CC=clang CXX=clang++
$ CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure
...
CC = /usr/bin/ccache clang
...
CXX = /usr/bin/ccache clang++ -std=c++11
...
$ make clean && make
...
/usr/bin/ld: /home/hebasto/GitHub/bitcoin/depends/x86_64-pc-linux-gnu/share/../lib/libdb_cxx-4.8.a(cxx_db.o): relocation R_X86_64_32S against symbol `_ZTV2Db' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: /home/hebasto/GitHub/bitcoin/depends/x86_64-pc-linux-gnu/share/../lib/libdb_cxx-4.8.a(cxx_env.o): relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: /home/hebasto/GitHub/bitcoin/depends/x86_64-pc-linux-gnu/share/../lib/libdb_cxx-4.8.a(cxx_except.o): relocation R_X86_64_32S against symbol `_ZTV11DbException' can not be used when making a PIE object; recompile with -fPIC
# loads of "recompile with -fPIC" link errors
|
Maybe instead of backporting patch just bump FreeType version to the latest 2.10.2 where that patch is included? |
Can you check your logs to see the configuration logs of |
output |
|
@dongcarl |
|
REMOVED |
|
Please don't merge this just yet. There's at least one build failure that will need addressing (likely in a follow up, but I want to understand it before merging). |
|
There is a at least one failure building make -C depends/ -j6 V=1 HOST=x86_64-apple-darwin16 CC=clang CXX=clang++
....
echo Configuring zlib...
Configuring zlib...
rm -rf /bitcoin/depends/x86_64-apple-darwin16; mkdir -p /bitcoin/depends/x86_64-apple-darwin16/lib; cd /bitcoin/depends/x86_64-apple-darwin16; tar --no-same-owner -xf /bitcoin/depends/built/x86_64-apple-darwin16/native_cctools/native_cctools-3764b223c011574971ee3ae09ce968ba5dc2f00f-d41d83b91be.tar.gz;
mkdir -p /bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c/.
cd /bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c/.; PKG_CONFIG_LIBDIR=/bitcoin/depends/x86_64-apple-darwin16/lib/pkgconfig PKG_CONFIG_PATH=/bitcoin/depends/x86_64-apple-darwin16/share/pkgconfig CMAKE_MODULE_PATH=/bitcoin/depends/x86_64-apple-darwin16/lib/cmake PATH=/bitcoin/depends/x86_64-apple-darwin16/native/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin env CC="clang" CFLAGS="-pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC" RANLIB="x86_64-apple-darwin16-ranlib" AR="x86_64-apple-darwin16-ar" AR="x86_64-apple-darwin16-libtool" ARFLAGS="-o" ./configure --static --prefix=/bitcoin/depends/x86_64-apple-darwin16
Building static library libz.a version 1.2.11 with clang.
Checking for size_t... Yes.
Checking for off64_t... Yes.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... Yes.
touch /bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c/./.stamp_configured
echo Building zlib...
Building zlib...
mkdir -p /bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c/.
cd /bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c/.; PATH=/bitcoin/depends/x86_64-apple-darwin16/native/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin make libz.a
make[1]: Entering directory '/bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c'
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o adler32.o adler32.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o crc32.o crc32.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o deflate.o deflate.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o infback.o infback.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o inffast.o inffast.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o inflate.o inflate.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o inftrees.o inftrees.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o trees.o trees.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o zutil.o zutil.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o compress.o compress.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o uncompr.o uncompr.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o gzclose.o gzclose.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o gzlib.o gzlib.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o gzread.o gzread.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o gzwrite.o gzwrite.c
x86_64-apple-darwin16-libtool -o libz.a adler32.o crc32.o deflate.o infback.o inffast.o inflate.o inftrees.o trees.o zutil.o compress.o uncompr.o gzclose.o gzlib.o gzread.o gzwrite.o
error: x86_64-apple-darwin16-libtool: file: adler32.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: crc32.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: deflate.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: infback.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: inffast.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: inflate.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: inftrees.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: trees.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: zutil.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: compress.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: uncompr.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: gzclose.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: gzlib.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: gzread.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: gzwrite.o is not an object file (not allowed in a library)
make[1]: *** [Makefile:131: libz.a] Error 1
make[1]: Leaving directory '/bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c'
make: *** [funcs.mk:262: /bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c/./.stamp_built] Error 2
make: Leaving directory '/bitcoin/depends'I assume this is because we lose our macOS flags. i.e @dongcarl thoughts? If you want to address this in a follow up I think that's reasonable. |
|
I'll also just mention that you can't seem to cross-compile for Windows when using Clang. The build bombs out at Boost (haven't looked into this at all 😴): echo Building boost...
Building boost...
mkdir -p /bitcoin/depends/work/build/x86_64-w64-mingw32/boost/1_70_0-d650037a261/.
cd /bitcoin/depends/work/build/x86_64-w64-mingw32/boost/1_70_0-d650037a261/.; PATH=/bitcoin/depends/x86_64-w64-mingw32/native/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ./b2 -d2 -j2 -d1 --prefix=/bitcoin/depends/work/staging/x86_64-w64-mingw32/boost/1_70_0-d650037a261/bitcoin/depends/x86_64-w64-mingw32 --layout=tagged --build-type=complete --user-config=user-config.jam threading=multi link=static -sNO_BZIP2=1 -sNO_ZLIB=1 variant=release target-os=windows binary-format=pe threadapi=win32 runtime-link=static address-model=64 toolset=clang stage
Performing configuration checks
- default address-model : 64-bit (cached)
- default architecture : x86 (cached)
- BOOST_COMP_GNUC >= 4.3.0 : no (cached)
Component configuration:
- atomic : not building
- chrono : not building
- container : not building
- context : not building
- contract : not building
- coroutine : not building
- date_time : not building
- exception : not building
- fiber : not building
- filesystem : building
- graph : not building
- graph_parallel : not building
- headers : not building
- iostreams : not building
- locale : not building
- log : not building
- math : not building
- mpi : not building
- program_options : not building
- python : not building
- random : not building
- regex : not building
- serialization : not building
- stacktrace : not building
- system : building
- test : building
- thread : building
- timer : not building
- type_erasure : not building
- wave : not building
...patience...
...patience...
...found 1632 targets...
...updating 5 targets...
clang-linux.compile.c++.without-pch bin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden/win32/thread_primitives.obj
"clang++" -c -x c++ -fvisibility-inlines-hidden -std=c++11 -fvisibility=hidden -I/bitcoin/depends/x86_64-w64-mingw32/include -m64 -O3 -Wall -fvisibility=hidden -Wno-inline -pthread -Wextra -Wno-long-long -Wno-unused-parameter -Wno-variadic-macros -Wunused-function -DBOOST_ALL_NO_LIB=1 -DBOOST_CHRONO_STATIC_LINK=1 -DBOOST_THREAD_BUILD_LIB=1 -DBOOST_THREAD_USES_CHRONO -DBOOST_THREAD_WIN32 -DBOOST_USE_WINDOWS_H -DNDEBUG -DWIN32_LEAN_AND_MEAN -I"." -o "bin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden/win32/thread_primitives.obj" "libs/thread/src/win32/thread_primitives.cpp"
In file included from libs/thread/src/win32/thread_primitives.cpp:10:
In file included from ./boost/winapi/dll.hpp:14:
./boost/winapi/basic_types.hpp:19:11: fatal error: 'windows.h' file not found
# include <windows.h>
^~~~~~~~~~~
1 error generated.
...failed clang-linux.compile.c++.without-pch bin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden/win32/thread_primitives.obj...
clang-linux.compile.c++.without-pch bin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden/win32/thread.obj
"clang++" -c -x c++ -fvisibility-inlines-hidden -std=c++11 -fvisibility=hidden -I/bitcoin/depends/x86_64-w64-mingw32/include -m64 -O3 -Wall -fvisibility=hidden -Wno-inline -pthread -Wextra -Wno-long-long -Wno-unused-parameter -Wno-variadic-macros -Wunused-function -DBOOST_ALL_NO_LIB=1 -DBOOST_CHRONO_STATIC_LINK=1 -DBOOST_THREAD_BUILD_LIB=1 -DBOOST_THREAD_USES_CHRONO -DBOOST_THREAD_WIN32 -DBOOST_USE_WINDOWS_H -DNDEBUG -DWIN32_LEAN_AND_MEAN -I"." -o "bin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden/win32/thread.obj" "libs/thread/src/win32/thread.cpp"
In file included from libs/thread/src/win32/thread.cpp:11:
In file included from ./boost/thread/thread_only.hpp:15:
In file included from ./boost/thread/win32/thread_data.hpp:11:
In file included from ./boost/thread/win32/thread_primitives.hpp:18:
./boost/detail/interlocked.hpp:53:11: fatal error: 'windows.h' file not found
# include <windows.h>
^~~~~~~~~~~
1 error generated.
...failed clang-linux.compile.c++.without-pch bin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden/win32/thread.obj...
...skipped <pbin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden>libboost_thread-mt-s-x64.lib(clean) for lack of <pbin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden>win32/thread.obj...
...skipped <pbin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden>libboost_thread-mt-s-x64.lib for lack of <pbin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden>win32/thread.obj...
...skipped <p/bitcoin/depends/work/build/x86_64-w64-mingw32/boost/1_70_0-d650037a261/stage/lib>libboost_thread-mt-s-x64.lib for lack of <pbin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden>libboost_thread-mt-s-x64.lib...
...failed updating 2 targets...
...skipped 3 targets...
make: *** [funcs.mk:262: /bitcoin/depends/work/build/x86_64-w64-mingw32/boost/1_70_0-d650037a261/./.stamp_built] Error 1
make: Leaving directory '/bitcoin/depends' |
|
There is also a build failure on a fresh Ubuntu focal vm: Steps to reproduce: |
|
I think this PR by itself is already an improvement over the current situation (all cases that worked before still works, and a few cases that didn't work before now works) and I don't really want to overload this PR any longer. After doing some investigation, I think the following are worthwhile followups:
One thing to keep in mind for both of these followups is that we've been abusing @fanquake @MarcoFalke Let me know if that sounds okay. |
Is there documentation on how to pass CXXFLAGS into depends qt? I recently tried that and it appeared they were not picked up. |
|
Concept ACK on merging this |
We do some extremely nasty A cleanup of how we interact with QT's build system probably also deserves to be a followup |
|
@dongcarl followups sound ok. I think we've figured out a path to improving the macOS CFLAGS post this change as well. |
Gitian builds
|
f0d7ed1 depends: Propagate only specific CLI variables to sub-makes (Carl Dong) 0a33803 depends: boost: Use clang toolset if clang in CXX (Carl Dong) 1ce74bc depends: boost: Split target-os from toolset (Carl Dong) 2d4e480 depends: boost: Specify toolset to bootstrap.sh (Carl Dong) 3d6603e depends: Propagate well-known vars into depends (Carl Dong) Pull request description: From: bitcoin#18308 (comment) The following monstrosity is quite useful when invoked inside `depends`, and reviewers can use it to compare the behaviour of this change against master. ```bash make print-{{,{host,{,{i686,x86_64,riscv64}_}linux}_}{CC,CXX},boost_{cc,cxx}} ``` It would also be helpful to make sure that setting `HOST`, `CC`, and `CXX` does the right thing. The 3 hosts I found offered good coverage were: `{x86_64,i686,riscv64}-linux-gnu`. As we special-case the `x86_64` and `i686` hosts in `depends/hosts/linux.mk`, and `riscv64` is a sanity check for a non-special-cased host. ACKs for top commit: hebasto: ACK f0d7ed1, tested on Linux Mint 19.3 (x86_64): practicalswift: ACK f0d7ed1 -- patch looks correct laanwj: Code review and concept ACK f0d7ed1 ryanofsky: Code review ACK f0d7ed1. Changes since last review: adding comment explaining check for predefined make variables, dropping freetype commit, adding commit whitelisting overrides for recursive makes Tree-SHA512: b6b8e76f713c26a0add6cd685824e2f5639109236ee9f89338f7c79cb1b1f2c3897bfb62b80b023d6d1943b5a6eb282a2f827f1f499c5e556eca015d6635fa65
Summary: This will makes the boost depends build follow the CC/CXX environment variables (and allow the build with clang). Backport of core [[bitcoin/bitcoin#18820 | PR18820]]. Test Plan: Run the Gitian builds. Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D7672
Boost build backports Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#17087 - bitcoin/bitcoin#17231 - bitcoin/bitcoin#17928 - bitcoin/bitcoin#18820 - bitcoin/bitcoin#19764 Kudos to @dongcarl for all the excellent upstream depends system hackery!
de7766c Build: Update Boost download URL (Fuzzbawls) 7be66c9 Doc: document updated boost version in dependencies.md (Fuzzbawls) bcb77b6 depends: boost: Specify cflags+compileflags (Carl Dong) b8f8574 depends: boost: Remove unnecessary _archiver_ (Carl Dong) 29fdbd9 depends: boost: Cleanup toolset selection (Carl Dong) 28393b6 depends: boost: Cleanup architecture/address-model (Carl Dong) 6af3ffa depends: boost: Disable all compression (Carl Dong) 0f09788 depends: boost: Split into non-/native packages (Carl Dong) de97b06 depends: boost: Bump to 1.71.0 (Carl Dong) 19f474b depends: boost: Refer to version in URL (Carl Dong) 7d4257c depends: Propagate only specific CLI variables to sub-makes (Carl Dong) fcbf870 depends: boost: Use clang toolset if clang in CXX (Carl Dong) aad5009 depends: boost: Split target-os from toolset (Carl Dong) fae749b depends: boost: Specify toolset to bootstrap.sh (Carl Dong) c2bfedb depends: Propagate well-known vars into depends (Carl Dong) 091ae4a depends: Consistent use of package variable (Peter Bushnell) 635bdc1 depends: fix boost mac cross build with clang 9+ (Cory Fields) d796365 build: Add variable printing target to Makefiles (Carl Dong) Pull request description: Backports the following upstream PRs to clean up and update the Boost dependency - bitcoin#17087 - bitcoin#17231 - bitcoin#17928 - bitcoin#18820 - bitcoin#19764 ACKs for top commit: random-zebra: ACK de7766c furszy: no code changes after rebase, utACK de7766c and merging.. Tree-SHA512: 4abe88718892bce40a2df023e99a26a16ce3c5d470f55e70d6c6cca117ee8b8bb29968be6d40873bc9ece3f9df769bea248cbb38c1c5c2f318016702533f2736
merge bitcoin#18820, bitcoin#19764, bitcoin#13686, bitcoin#17538, bitcoin#18405: zmq backports, boost depends split
From: #18308 (comment)
The following monstrosity is quite useful when invoked inside
depends, and reviewers can use it to compare the behaviour of this change against master.make print-{{,{host,{,{i686,x86_64,riscv64}_}linux}_}{CC,CXX},boost_{cc,cxx}}It would also be helpful to make sure that setting
HOST,CC, andCXXdoes the right thing. The 3 hosts I found offered good coverage were:{x86_64,i686,riscv64}-linux-gnu. As we special-case thex86_64andi686hosts independs/hosts/linux.mk, andriscv64is a sanity check for a non-special-cased host.