Skip to content

dictionary changed size during iteration Exception in Crawler CookieJar kills crawlerThread #5187

@Murgeye

Description

@Murgeye

Describe the bug
When using the crawler, threads crash one after another with an uncaught exception from accessing the CookieJar while it is being changed by another thread. This seems to kill the thread permanently, making the crawling process much slower than necessary and fail crawling the current URL.

Cause seems to be this line in lib/request/connect.py:

cookies = conf.cj._cookies_for_request(req)

which accesses a private method of the cookie jar that apparently is not thread-safe.

Additionally, since the _threadFunction (crawlThread) of the crawler is a long running function, an uncaught exception seems to kill the thread completely until the current crawl level is done.

To Reproduce

  1. Run sqlmap --crawl=[n] --threads=10 -u [url] (Where the URL has to set cookies for each request)
  2. Wait for exception to occur. This might take a while (since this is a race condition).

Running environment:

  • sqlmap version: 1.6.9
  • Installation method: git
  • Operating system: Ubuntu 22.04
  • Python version: 3.10.7

Target details:

  • Exception traceback
[00:02:19] [ERROR] thread 0: 'RuntimeError: RuntimeError: dictionary changed size during iteration
 Traceback (most recent call last):
  File "sqlmap/lib/core/threads.py", line 97, in exceptionHandledFunction
    threadFunction()
  File "sqlmap/lib/core/threads.py", line 124, in _threadFunction
    threadFunction()
  File "sqlmap/lib/utils/crawler.py", line 75, in crawlThread
    content = Request.getPage(url=current, post=post, cookie=None, crawling=True, raise404=False)[0]
  File "sqlmap/lib/request/connect.py", line 590, in getPage
    cookies = conf.cj._cookies_for_request(req)
  File "/usr/lib/python3.10/http/cookiejar.py", line 1299, in _cookies_for_request
    cookies.extend(self._cookies_for_domain(domain, request))
  File "/usr/lib/python3.10/http/cookiejar.py", line 1283, in _cookies_for_domain
    for path in cookies_by_path.keys():
RuntimeError: dictionary changed size during iteration
'

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions