use urlopen context manager to handle connection close in a timely manner#47
use urlopen context manager to handle connection close in a timely manner#47jubrad wants to merge 1 commit intokevinsteves:masterfrom
Conversation
jubrad
force-pushed
the
close-connection
branch
from
e1055bd to
bb3a0c7
Compare
|
Hello guys, we do have the same error and looking for a way to at least close the connection manually. |
|
This fix looks good, resulting in |
|
Seems good to me |
#47 from Justin Bradfield. 1. Use urlopen() as a context manager 2. Immediately read() response 3. Set object 'pan_body' attribute to read() result
|
Modified version of |
@kevinsteves I think I did this to log a few things during debug. I then removed the log messages and didn't put the code back to the simpler form. Thanks for merging. |
@devbollinger make sure that you're not exceeding the max threads of the PA server... There was a longer discussion in the pango repo about that. This is especially applicable in the terraform provider, if have parallelization > 4 then you will hit timeout issues to some extent. |
3 participants
What it does:
More explicitly closes the xapi response object to avoid downstream circular reference or other potential issues where garbage collector does not promptly clean the request and close the connection.
Background:
I'm noticing many "Session Timed Out" errors while using local-execs in terraform to configure some resources that the tf provider cannot. At first I tested to see what rate I would hit this bug, but was not able to with >6k requests happening serially (creating and deleting the same dns proxy static entries). Since terraform runs with 10 parallel threads walking its tree I decided to test some parallel calls and quickly was able to rerpro. ~20-40 threads making 4 entries each.
I believe this relates to terraform-provider-panos/issues/255, at least it does for me, but it's possible there's something wonky going on with the PA server here.
How it was tested:
Using a python script that adds and removes dns-proxy static entries I validated that with ~20 processes each making 8 entries as fast as they can the python client would regularly produce "Session Timed Out" errors. My goal of this test was to see if I could increase the rate of collision or push the server beyond its concurrent open request limits. These "errors" would produce a valid http response to the urlopen response object with the following error:
b'<response status="unauth" code="22"><msg><line>Session timed out</line></msg></response>'To me, this indicates that there's something weird going on on the server side, potentially to do with auth, but probably not due to any tcp / request issues.
When using the context manager I could not reproduce this error in python, even when increasing the concurrent processes to 40. Although... the terraform provider does give me a
session timed outso there's still more for me to figure out.Not the best solution, but it's what I have time for.