Skip to content

[qa] blockstore: Switch to dumb dbm#8834

Merged
laanwj merged 1 commit intobitcoin:masterfrom
maflcko:Mf1610-qaBlockstoreDumb
Sep 29, 2016
Merged

[qa] blockstore: Switch to dumb dbm#8834
laanwj merged 1 commit intobitcoin:masterfrom
maflcko:Mf1610-qaBlockstoreDumb

Conversation

@maflcko
Copy link
Member

@maflcko maflcko commented Sep 29, 2016

Closes #8605

I could not see any performance degradation by this commit:

$ time qa/pull-tester/rpc-tests.py p2p-fullblocktest

Before:

TEST                 | PASSED | DURATION

p2p-fullblocktest.py | True   | 297 s

ALL                  | True   | 297 s (accumulated)

Runtime: 297 s

real    4m57.633s
user    4m1.588s
sys 0m11.419s

After:

p2p-fullblocktest.py:
Pass: True, Duration: 288 s

TEST                 | PASSED | DURATION

p2p-fullblocktest.py | True   | 288 s

ALL                  | True   | 288 s (accumulated)

Runtime: 288 s

real    4m48.573s
user    3m56.126s
sys 0m11.823s

@maflcko maflcko added this to the 0.13.1 milestone Sep 29, 2016
@laanwj
Copy link
Member

laanwj commented Sep 29, 2016

Concept ACK, this is by far the most straightforward solution, if there is no need for platform-specific exceptions then let's avoid them.

@laanwj
Copy link
Member

laanwj commented Sep 29, 2016 •
edited
Loading

Interesting, apparently this causes the NULLDUMMY test to fail during close, on some platforms. I don't understand why this file would suddenly be missing during close:

stderr:
Exception ignored in: <bound method _Database.close of <dbm.dumb._Database object at 0x7f8f097f1be0>>
Traceback (most recent call last):
  File "/usr/lib/python3.4/dbm/dumb.py", line 250, in close
    self._commit()
  File "/usr/lib/python3.4/dbm/dumb.py", line 112, in _commit
    with self._io.open(self._dirfile, 'w', encoding="Latin-1") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/testkz7n1z0q/1/blocks.dir'
Exception ignored in: <bound method _Database.close of <dbm.dumb._Database object at 0x7f8f097f1d68>>
Traceback (most recent call last):
  File "/usr/lib/python3.4/dbm/dumb.py", line 250, in close
    self._commit()
  File "/usr/lib/python3.4/dbm/dumb.py", line 112, in _commit
    with self._io.open(self._dirfile, 'w', encoding="Latin-1") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/testkz7n1z0q/1/transactions.dir'

I can reproduce this locally on FreeBSD.
Are perhaps the database files wiped before closing it?
This only affects nulldummy.py not the other tests involving block stores.

@jonasschnelli
Copy link
Contributor

jonasschnelli commented Sep 29, 2016

Failed on my OSX machine:

time qa/pull-tester/rpc-tests.py p2p-fullblocktest
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
p2p-fullblocktest.py:
Initializing test directory /var/folders/hp/kb9p9q8x4k3_z_ccy588hxrc0000gn/T/test7n1apn12/0
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11000
Test 1: PASS [1]
Test 2: PASS [100]
Test 3: PASS [101]
Test 4: PASS [102]
Test 5: PASS [102]
Test 6: PASS [103]
Test 7: PASS [103]
Test 8: PASS [104]
Test 9: PASS [104]
Test 10: PASS [104]
Test 11: PASS [104]
Test 12: PASS [104]
Test 13: PASS [104]
Test 14: PASS [104]
Test 15: PASS [104]
Test 16: PASS [105]
Test 17: PASS [106]
Test 18: PASS [106]
Test 19: PASS [106]
Test 20: PASS [106]
Test 21: PASS [106]
Test 22: PASS [106]
Test 23: PASS [106]
Test 24: PASS [106]
Test 25: PASS [107]
Test 26: PASS [107]
Test 27: PASS [107]
Test 28: PASS [107]
Test 29: PASS [107]
Test 30: PASS [107]
Test 31: PASS [107]
Test 32: PASS [108]
Test 33: PASS [109]
Test 34: PASS [109]
Test 35: PASS [110]
Test 36: PASS [110]
Test 37: PASS [111]
Test 38: PASS [111]
Test 39: PASS [111]
Test 40: PASS [111]
Test 41: PASS [112]
Test 42: PASS [112]
Test 43: PASS [113]
Test 44: PASS [113]
Test 45: PASS [114]
Test 46: PASS [115]
Test 47: PASS [115]
Test 48: PASS [115]
Test 49: PASS [115]
Test 50: PASS [115]
Test 51: PASS [115]
Test 52: PASS [115]
Test 53: PASS [115]
Test 54: PASS [115]
Test 55: PASS [115]
Test 56: PASS [115]
Test 57: PASS [116]
Test 58: PASS [116]
Test 59: PASS [116]
Test 60: PASS [117]
Test 61: PASS [117]
Test 62: PASS [117]
Test 63: PASS [117]
Test 64: PASS [118]
Test 65: PASS [118]
Test 66: PASS [118]
Test 67: PASS [118]
Test 68: PASS [118]
Test 69: PASS [119]
Test 70: PASS [120]
Test 71: PASS [120]
Test 72: PASS [120]
Test 73: PASS [120]
Test 74: PASS [121]
Test 75: PASS [121]
Test 76: PASS [121]
Test 77: PASS [122]
Test 78: PASS [122]
Test 79: PASS [122]
Test 80: PASS [123]
Test 81: PASS [124]
Test 82: PASS [125]
Test 83: PASS [126]
Test 84: PASS [127]
Test 85: PASS [127]
Test 86: PASS [127]
Test 87: PASS [128]
Test 88: PASS [129]
Test 89: PASS [130]
Test 90: PASS [130]
Test 91: PASS [131]
Test 92: PASS [131]
Test 93: PASS [132]
Test 94: PASS [132]
Test 95: PASS [1220]
Test 96: PASS [1220]
Test 97: PASS [1221]
Test 98: PASS [1221]
Unexpected exception caught during testing: ConnectionResetError(54, 'Connection reset by peer')
Stopping nodes
Not cleaning up dir /var/folders/hp/kb9p9q8x4k3_z_ccy588hxrc0000gn/T/test7n1apn12/0
Failed

stderr:
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/test_framework.py", line 151, in main
    self.run_test()
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/p2p-fullblocktest.py", line 73, in run_test
    self.test.run()
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/comptool.py", line 395, in run
    print("Test %d: PASS" % test_number, [ c.rpc.getblockcount() for c in self.connections ])
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/comptool.py", line 395, in <listcomp>
    print("Test %d: PASS" % test_number, [ c.rpc.getblockcount() for c in self.connections ])
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/coverage.py", line 49, in __call__
    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 148, in __call__
    response = self._request('POST', self.__url.path, postdata.encode('utf-8'))
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 125, in _request
    return self._get_response()
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 163, in _get_response
    http_response = self.__conn.getresponse()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1197, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)

Pass: False, Duration: 928 s

TEST                 | PASSED | DURATION

p2p-fullblocktest.py | False  | 928 s

ALL                  | False  | 928 s (accumulated)

Runtime: 928 s

real    15m28.427s
user    14m47.171s
sys 0m31.176s

Trying again....

@laanwj
Copy link
Member

laanwj commented Sep 29, 2016 •
edited
Loading

"Connection reset by peer" is a different issue, unrelated to the database. I have the same on FreeBSD. @jonasschnelli can you try:

diff --git a/qa/rpc-tests/test_framework/authproxy.py b/qa/rpc-tests/test_framework/authproxy.py
index f5e0be2..fd7f32b 100644
--- a/qa/rpc-tests/test_framework/authproxy.py
+++ b/qa/rpc-tests/test_framework/authproxy.py
@@ -130,8 +130,9 @@ class AuthServiceProxy(object):
                 return self._get_response()
             else:
                 raise
-        except BrokenPipeError:
-            # Python 3.5+ raises this instead of BadStatusLine when the connection was reset
+        except (BrokenPipeError,ConnectionResetError):
+            # Python 3.5+ raises BrokenPipeError instead of BadStatusLine when the connection was reset
+            # ConnectionResetError happens on FreeBSD with Python 3.4
             self.__conn.close()
             self.__conn.request(method, path, postdata, headers)
             return self._get_response()

@jonasschnelli
Copy link
Contributor

jonasschnelli commented Sep 29, 2016

Passed after adding ConnectionResetError.

time qa/pull-tester/rpc-tests.py p2p-fullblocktest
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
p2p-fullblocktest.py:
Pass: True, Duration: 969 s

TEST                 | PASSED | DURATION

p2p-fullblocktest.py | True   | 969 s

ALL                  | True   | 969 s (accumulated)

Runtime: 969 s

real    16m9.820s
user    15m22.178s
sys 0m34.593s

@maflcko
Copy link
Member Author

maflcko commented Sep 29, 2016

Interesting, apparently this causes the NULLDUMMY test to fail during close, on some platforms. I don't understand why this file would suddenly be missing during close:

Unrelated to this pull, but needs travis retriggered after #8835

@laanwj laanwj mentioned this pull request Sep 29, 2016
Merged
@laanwj
Copy link
Member

laanwj commented Sep 29, 2016

Unrelated to this pull, but needs travis retriggered after #8835

Done

laanwj added a commit to laanwj/bitcoin that referenced this pull request Sep 29, 2016
@laanwj
test: Avoid ConnectionResetErrors during RPC tests
1d28faf 
This is necessary on FreeBSD and MacOSX, at least.

See bitcoin#8834 (comment)
@laanwj laanwj mentioned this pull request Sep 29, 2016
Merged
@laanwj
Copy link
Member

laanwj commented Sep 29, 2016

Tested ACK fa9cd25

@laanwj laanwj merged commit fa9cd25 into bitcoin:master Sep 29, 2016
laanwj added a commit that referenced this pull request Sep 29, 2016
@laanwj
Merge #8834: [qa] blockstore: Switch to dumb dbm
c9d7b0d 
fa9cd25 [qa] blockstore: Switch to dumb dbm (MarcoFalke)
@maflcko maflcko deleted the Mf1610-qaBlockstoreDumb branch September 29, 2016 15:25
maflcko pushed a commit to maflcko/bitcoin-core that referenced this pull request Oct 3, 2016
[qa] blockstore: Switch to dumb dbm
1dd1783 
Github-Pull: bitcoin#8834
Rebased-From: fa9cd25
maflcko pushed a commit to maflcko/bitcoin-core that referenced this pull request Oct 3, 2016
@laanwj
test: Avoid ConnectionResetErrors during RPC tests
31ab2f8 
This is necessary on FreeBSD and MacOSX, at least.

See bitcoin#8834 (comment)

Github-Pull: bitcoin#8839
Rebased-From: 1d28faf
str4d pushed a commit to str4d/zcash that referenced this pull request Mar 24, 2017
@laanwj @str4d
test: Avoid ConnectionResetErrors during RPC tests
e93a3e1 
This is necessary on FreeBSD and MacOSX, at least.

See bitcoin/bitcoin#8834 (comment)
@ftrader ftrader mentioned this pull request Apr 5, 2017
Merged
dagurval pushed a commit to dagurval/bitcoinxt that referenced this pull request Aug 28, 2017
@laanwj @dagurval
test: Avoid ConnectionResetErrors during RPC tests
621894d 
This is necessary on FreeBSD and MacOSX, at least.

See bitcoin/bitcoin#8834 (comment)
dagurval pushed a commit to dagurval/bitcoinxt that referenced this pull request Aug 28, 2017
@laanwj @dagurval
test: Avoid ConnectionResetErrors during RPC tests
7e28438 
This is necessary on FreeBSD and MacOSX, at least.

See bitcoin/bitcoin#8834 (comment)
dagurval pushed a commit to dagurval/bitcoinxt that referenced this pull request Aug 29, 2017
@laanwj @dagurval
test: Avoid ConnectionResetErrors during RPC tests
4e3c29e 
This is necessary on FreeBSD and MacOSX, at least.

See bitcoin/bitcoin#8834 (comment)
dagurval pushed a commit to dagurval/bitcoinxt that referenced this pull request Sep 18, 2017
@laanwj @dagurval
test: Avoid ConnectionResetErrors during RPC tests
9b7555e 
This is necessary on FreeBSD and MacOSX, at least.

See bitcoin/bitcoin#8834 (comment)
OlegGirko pushed a commit to OlegGirko/dash that referenced this pull request Nov 7, 2017
@laanwj @OlegGirko
test: Avoid ConnectionResetErrors during RPC tests
c50ed3f 
This is necessary on FreeBSD and MacOSX, at least.

See bitcoin#8834 (comment)
codablock pushed a commit to codablock/dash that referenced this pull request Jan 12, 2018
@laanwj @codablock
Merge bitcoin#8834: [qa] blockstore: Switch to dumb dbm
e6af218 
fa9cd25 [qa] blockstore: Switch to dumb dbm (MarcoFalke)
andvgal pushed a commit to energicryptocurrency/gen2-energi that referenced this pull request Jan 6, 2019
@laanwj @andvgal
Merge bitcoin#8834: [qa] blockstore: Switch to dumb dbm
c91e199 
fa9cd25 [qa] blockstore: Switch to dumb dbm (MarcoFalke)
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

Tests

Projects

None yet

Milestone

0.13.1

Development

Successfully merging this pull request may close these issues.

p2p-fullblocktest fails on BSD derivates

3 participants

@maflcko @laanwj @jonasschnelli