Skip to content

leverage installed OpenSSL/BoringSSL when cURL is used as a dependency for TensorFlow#2197

Merged
boegel merged 1 commit intoeasybuilders:developfrom
Flamefire:tf_boringssl
Oct 8, 2020
Merged

leverage installed OpenSSL/BoringSSL when cURL is used as a dependency for TensorFlow#2197
boegel merged 1 commit intoeasybuilders:developfrom
Flamefire:tf_boringssl

Conversation

@Flamefire
Copy link
Copy Markdown
Contributor

@Flamefire Flamefire commented Oct 8, 2020 •
edited
Loading

When using EB provided cURL but NOT using EB/system provided boringssl then TensorFlow will build and use an own version of BoringSSL which causes symbol name conflicts with the OpenSSL used by cURL ultimately leading to crashes. See tensorflow/tensorflow#43878 for details.

As the "system boringssl" simply links -lssl -lcrypto we can simply tell TF to use the system "boringssl" which is in fact the system OpenSSL, so all checks out.

Validated via eb TensorFlow-2.3.1-fosscuda-2019b-Python-3.7.4.eb --rebuild --installpath /tmp/ebinstall --disable-cleanup-tmpdir --disable-cleanup-builddir -r and then grep -r X509_VERIFY_PARAM_set_flags $EASYBUILD_BUILDPATH which shows only a few hits but no files of BoringSSL. Before this PR there are sources from BoringSSL.

Example code to trigger this and stacktrace/core dump (needs an additional package pip install --user tensorflow_datasets):

import tensorflow_datasets as tfds

builder = tfds.builder('stanford_dogs')
builder.download_and_prepare()

Stack trace

*** Error in `python': free(): invalid pointer: 0x00000000046bfd08 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81299)[0x7f2b1dba9299]
<prefix>/lib/python3.7/site-packages/tensorflow/python/../libtensorflow_framework.so.2(ASN1_STRING_free+0x35)[0x7f2aac3d0205]
<prefix>/lib/python3.7/site-packages/tensorflow/python/../libtensorflow_framework.so.2(ASN1_primitive_free+0xbd)[0x7f2aac3d371d]
<prefix>/lib/python3.7/site-packages/tensorflow/python/../libtensorflow_framework.so.2(ASN1_template_free+0x8f)[0x7f2aac3d3b1f]
<prefix>/lib/python3.7/site-packages/tensorflow/python/../libtensorflow_framework.so.2(asn1_item_combine_free+0x2a0)[0x7f2aac3d39f0]
<prefix>/lib/python3.7/site-packages/tensorflow/python/../libtensorflow_framework.so.2(ASN1_item_free+0x17)[0x7f2aac3d3a77]
<prefix>/lib/libcurl.so.4(+0x57a20)[0x7f2aaae7ea20]
<prefix>/lib/libcurl.so.4(+0x593fb)[0x7f2aaae803fb]
<prefix>/lib/libcurl.so.4(+0x59fd7)[0x7f2aaae80fd7]
<prefix>/lib/libcurl.so.4(+0x110e2)[0x7f2aaae380e2]
<prefix>/lib/libcurl.so.4(+0x30215)[0x7f2aaae57215]
<prefix>/lib/libcurl.so.4(curl_multi_perform+0x83)[0x7f2aaae58473]
<prefix>/lib/libcurl.so.4(curl_easy_perform+0x11b)[0x7f2aaae50a6b]
<prefix>/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZN10tensorflow15CurlHttpRequest4SendEv+0x2b9)[0x7f2ac88c34f9]

@Flamefire Flamefire changed the title Enable EB installed boringssl/openssl when cURL is also used Enable installed boringssl/openssl when cURL is also used Oct 8, 2020
@boegel boegel changed the title Enable installed boringssl/openssl when cURL is also used leverage installed OpenSSL/BoringSSL when cURL is used as a dependency for TensorFlow Oct 8, 2020
@boegel boegel added the bug fix label Oct 8, 2020
@boegel boegel added this to the next release (4.3.1) milestone Oct 8, 2020
boegel
boegel approved these changes Oct 8, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, tested with various TensorFlow easyconfigs

@boegel boegel merged commit 7305ce1 into easybuilders:develop Oct 8, 2020
@Flamefire Flamefire deleted the tf_boringssl branch October 9, 2020 14:20
@sushreebarsa sushreebarsa mentioned this pull request Oct 22, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@boegel boegel boegel approved these changes

Assignees

No one assigned

Labels

bug fix

Projects

None yet

Milestone

4.3.1

Development

Successfully merging this pull request may close these issues.

2 participants

@Flamefire @boegel