Describe the bug
When using the crawler, threads crash one after another with an uncaught exception from accessing the CookieJar while it is being changed by another thread. This seems to kill the thread permanently, making the crawling process much slower than necessary and fail crawling the current URL.
Cause seems to be this line in lib/request/connect.py:
cookies = conf.cj._cookies_for_request(req)
which accesses a private method of the cookie jar that apparently is not thread-safe.
Additionally, since the _threadFunction (crawlThread) of the crawler is a long running function, an uncaught exception seems to kill the thread completely until the current crawl level is done.
To Reproduce
- Run
sqlmap --crawl=[n] --threads=10 -u [url] (Where the URL has to set cookies for each request)
- Wait for exception to occur. This might take a while (since this is a race condition).
Running environment:
- sqlmap version: 1.6.9
- Installation method: git
- Operating system: Ubuntu 22.04
- Python version: 3.10.7
Target details:
[00:02:19] [ERROR] thread 0: 'RuntimeError: RuntimeError: dictionary changed size during iteration
Traceback (most recent call last):
File "sqlmap/lib/core/threads.py", line 97, in exceptionHandledFunction
threadFunction()
File "sqlmap/lib/core/threads.py", line 124, in _threadFunction
threadFunction()
File "sqlmap/lib/utils/crawler.py", line 75, in crawlThread
content = Request.getPage(url=current, post=post, cookie=None, crawling=True, raise404=False)[0]
File "sqlmap/lib/request/connect.py", line 590, in getPage
cookies = conf.cj._cookies_for_request(req)
File "/usr/lib/python3.10/http/cookiejar.py", line 1299, in _cookies_for_request
cookies.extend(self._cookies_for_domain(domain, request))
File "/usr/lib/python3.10/http/cookiejar.py", line 1283, in _cookies_for_domain
for path in cookies_by_path.keys():
RuntimeError: dictionary changed size during iteration
'
Describe the bug
When using the crawler, threads crash one after another with an uncaught exception from accessing the CookieJar while it is being changed by another thread. This seems to kill the thread permanently, making the crawling process much slower than necessary and fail crawling the current URL.
Cause seems to be this line in
lib/request/connect.py:which accesses a private method of the cookie jar that apparently is not thread-safe.
Additionally, since the
_threadFunction(crawlThread) of the crawler is a long running function, an uncaught exception seems to kill the thread completely until the current crawl level is done.To Reproduce
sqlmap --crawl=[n] --threads=10 -u [url](Where the URL has to set cookies for each request)Running environment:
Target details: