Remove chardet/charset-normalizer. Add fallback_charset_resolver ClientSession parameter. by john-parton · Pull Request #7561 · aio-libs/aiohttp

john-parton · 2023-08-27T19:30:32Z

What do these changes do?

Continuing work from #7559

~~This replaces cchardet with chardetng and also paves a path towards deprecating character detection completely. It will issue a DeprecationWarning if the character detection method is invoked~~

Need updates.

Are there changes in behavior for the user?

Previously, encoding detection was done on every request without a content-type header. This was a bottleneck in some code I wrote. This changes it so that first utf-8 is tried, and then encoding detection happens aftewards.

Need updates.

Related issue number

Should close #7126

Checklist

I think the code is well written
Unit tests for the changes exist
Documentation reflects the changes
If you provide code modification, please add yourself to CONTRIBUTORS.txt
- The format is <Name> <Surname>.
- Please keep alphabetical order, the file is sorted by names.
Add a new news fragment into the CHANGES folder
- name it <issue_id>.<type> for example (588.bugfix)
- if you don't have an issue_id change it to the pr id after creating the pr
- ensure type is one of the following:
  - .feature: Signifying a new feature.
  - .bugfix: Signifying a bug fix.
  - .doc: Signifying a documentation improvement.
  - .removal: Signifying a deprecation or removal of public API.
  - .misc: A ticket has been closed, but it is not of interest to users.
- Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files."

john-parton · 2023-08-27T20:35:15Z

Note: When we decide on which approach to go with, I'll close the other two, and polish the remaining pull request. I haven't done docs or release notes quite as well as I would like.

Dreamsorcerer · 2023-08-28T13:22:35Z

I think the best approach would be to implement the new callable in master and backport to 3.9.
Then do a backport to 3.8, where we set the default callable to the existing chardet approaches, so behaviour should work the same in 3.8, while still allowing people to make the change that will be needed for 3.9.

john-parton · 2023-08-28T14:53:14Z

@Dreamsorcerer

I've updated this pull request with your suggestions, with a few tweaks

ClientSession now takes the argument default_encoding. I added this to the __init__ function and the corresponding private property to __slots__.
I chose default_encoding over fallback_encoding because the word default is used elsewhere and is a more familiar word
The valid types for default_encoding are None, str, Callable[[ClientRequest], str]. I chose only to pass the ClientRequest and not the body because in other places in the code where callables like this are passed around, that's usually how it's done. However, as I'm typing this I realize that a user will have to access _body directly, and won't be able to use read() because it's a coro. We can change the signature for this to include the body, or we can make it return Awaitable[str] if we want to enforce that the only parameter is ClientRequest, but also allow accessing of _body via a public method.

If this general approach looks good, I'll start updating the docs.

Dreamsorcerer · 2023-08-28T15:12:57Z

I chose default_encoding over fallback_encoding because the word default is used elsewhere and is a more familiar word

I feel like default makes it sound more like it should be a constant than a function, though I realise fallback isn't a lot better. Maybe something else like charset_func or something..?

The valid types for default_encoding are None, str, Callable[[ClientRequest], str].

I think this adds unnecessary complexity and performance overhead. If we just accept a callable only, it is trivial for someone to do lamba *args: "encoding", and likewise for the default value.

I chose only to pass the ClientRequest and not the body because in other places in the code where callables like this are passed around, that's usually how it's done.

The one thing that a charset guesser will need is the body (or part of it) and the text() method will already have read the body, so I think we should just pass it through and same any trouble.

john-parton · 2023-08-28T16:59:19Z

Charset guesser might also need the tld, as we already discussed. Removing str seems reasonable as well.

So I think Callable[[ClientResponse, bytes], str] and None would cover it?

john-parton · 2023-08-28T17:00:46Z

We could literally just make the function called encoding_guesser or charset_guesser.

Dreamsorcerer · 2023-08-28T18:45:30Z

Charset guesser might also need the tld, as we already discussed.

It's available in the Response, which I why I included it in the original examples.

So I think Callable[[ClientResponse, bytes], str] and None would cover it?

We don't need None, just default to lambda: r, b: "utf-8".

Dreamsorcerer · 2023-08-28T18:46:19Z

We could literally just make the function called encoding_guesser or charset_guesser.

Yeah, I guess that's fine, just might be a little confusing given that we default to a function which doesn't do any guessing.

john-parton · 2023-08-28T18:57:24Z

Okay, I changed the variable/callback to 'fallback_encoding', I change the type signature to just be Optional[Callable], and I updated code in appropriate places.

As far as code snippets are concerned for instructing a user on how to operate the character set detection once the feature is removed, here are two options

# chardetng_py: fast charset detection written in rust
from chardetng_py import detect

def fallback_encoding(response, body):
    tld = response.url.host.split(".")[-1]
    return detect(body, tld=tld)

# cChardet, chardet, charset_normalizer
from charset_normalizer import chardet

def fallback_encoding(response, body):
    return chardet(body)["encoding"] or "utf-8"

async with ClientSession(fallback_encoding=fallback_encoding) as client:
    ....

Dreamsorcerer · 2023-08-28T19:07:17Z

I think things are getting mixed up a little.

Can we start with a PR that removes chardet and supports the new parameter? This will go into master. Once that's in we can look at backports and including chardetng in the 3.8 backport.

john-parton · 2023-08-28T19:16:57Z

No problem.

john-parton · 2023-08-28T19:28:31Z

@Dreamsorcerer So this removes the character set detection now and introduces the fallback_encoding callback we talked about.

Do you want me to update the ClientSession documentation and the relevant text() method documentation to refer to that?

Dreamsorcerer · 2023-08-28T19:55:09Z

Do you want me to update the ClientSession documentation and the relevant text() method documentation to refer to that?

Yep, we'll want that included too with examples of correctly using it for chardetng and charset-normalizer.

john-parton · 2023-08-29T02:34:16Z

@Dreamsorcerer I did my best. I'm not great at writing documentation. I tried to make some of the finer points clear, and I documented that text() will raise UnicodeDecodeError if not configured correctly, and I added a similar note in the "advanced usage" section that I added, so hopefully that will help with google-abilty-of-error.

Completely remove chardet and character set detection when content encoding is missing. remove charset-normalizer Document new parameter in ClientSession. Remove tests which specifically test encoding detection and replace with test that tests fallback function. Document get_encoding behavior with updated logic In documentation for text(), direct reader to refer to 'get_encoding' documentation if encoding is 'None'

… that they're in the glossary because they were previously used.

for more information, see https://pre-commit.ci

patchback · 2023-08-29T23:43:27Z

Backport to 3.9: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 6755796 on top of patchback/backports/3.9/675579699422680607108a7dd68c85ec5284220c/pr-7561

Backporting merged PR #7561 into master

Ensure you have a local repo clone of your fork. Unless you cloned it
from the upstream, this would be your origin remote.
Make sure you have an upstream repo added as a remote too. In these
instructions you'll refer to it by the name upstream. If you don't
have it, here's how you can add it:
```
$ git remote add upstream https://github.com/aio-libs/aiohttp.git
```

Ensure you have the latest copy of upstream and prepare a branch
that will hold the backported code:

$ git fetch upstream
$ git checkout -b patchback/backports/3.9/675579699422680607108a7dd68c85ec5284220c/pr-7561 upstream/3.9

Now, cherry-pick PR Remove chardet/charset-normalizer. Add fallback_charset_resolver ClientSession parameter. #7561 contents into that branch:
```
$ git cherry-pick -x 675579699422680607108a7dd68c85ec5284220c
```
If it'll yell at you with something like fatal: Commit 675579699422680607108a7dd68c85ec5284220c is a merge but no -m option was given., add -m 1 as follows instead:
```
$ git cherry-pick -m1 -x 675579699422680607108a7dd68c85ec5284220c
```
At this point, you'll probably encounter some merge conflicts. You must
resolve them in to preserve the patch from PR Remove chardet/charset-normalizer. Add fallback_charset_resolver ClientSession parameter. #7561 as close to the
original as possible.

Push this branch to your fork on GitHub:

$ git push origin patchback/backports/3.9/675579699422680607108a7dd68c85ec5284220c/pr-7561

Create a PR, ensure that the CI is green. If it's not — update it so that
the tests and any other checks pass. This is it!
Now relax and wait for the maintainers to process your pull request
when they have some cycles to do reviews. Don't worry — they'll tell you if
any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback · 2023-08-29T23:43:48Z

Backport to 3.8: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 6755796 on top of patchback/backports/3.8/675579699422680607108a7dd68c85ec5284220c/pr-7561

Backporting merged PR #7561 into master

Ensure you have a local repo clone of your fork. Unless you cloned it
from the upstream, this would be your origin remote.
Make sure you have an upstream repo added as a remote too. In these
instructions you'll refer to it by the name upstream. If you don't
have it, here's how you can add it:
```
$ git remote add upstream https://github.com/aio-libs/aiohttp.git
```

Ensure you have the latest copy of upstream and prepare a branch
that will hold the backported code:

$ git fetch upstream
$ git checkout -b patchback/backports/3.8/675579699422680607108a7dd68c85ec5284220c/pr-7561 upstream/3.8

Now, cherry-pick PR Remove chardet/charset-normalizer. Add fallback_charset_resolver ClientSession parameter. #7561 contents into that branch:
```
$ git cherry-pick -x 675579699422680607108a7dd68c85ec5284220c
```
If it'll yell at you with something like fatal: Commit 675579699422680607108a7dd68c85ec5284220c is a merge but no -m option was given., add -m 1 as follows instead:
```
$ git cherry-pick -m1 -x 675579699422680607108a7dd68c85ec5284220c
```
At this point, you'll probably encounter some merge conflicts. You must
resolve them in to preserve the patch from PR Remove chardet/charset-normalizer. Add fallback_charset_resolver ClientSession parameter. #7561 as close to the
original as possible.

Push this branch to your fork on GitHub:

$ git push origin patchback/backports/3.8/675579699422680607108a7dd68c85ec5284220c/pr-7561

Create a PR, ensure that the CI is green. If it's not — update it so that
the tests and any other checks pass. This is it!
Now relax and wait for the maintainers to process your pull request
when they have some cycles to do reviews. Don't worry — they'll tell you if
any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

Dreamsorcerer · 2023-08-29T23:47:02Z

@john-parton If you could handle the backports, that'd be great. We'll need to:

Follow the instructions above for 3.9 backport. This should be a straight backport.
Follow the instructions above for 3.8 backport.
- Modify the 3.8 backport to change the default function to emulate the current chardet behaviour (preferably without chardetng).
- Add a DeprecationWarning in the default function when it is called (or maybe when it returns a non "utf-8" value?).
Once they are both merged, create a new PR on 3.8 branch that replaces cchardet with chardetng.

webknjaz · 2023-08-30T01:10:48Z

+Character Set Detection
+-----------------------
+
+If you encounter a :exc:`UnicodeDecodeError` when using :meth:`ClientResponse.text()`


Does this actually work? The correct syntax AFAIK doesn't include ().

And so it doesn't: https://aiohttp--7561.org.readthedocs.build/en/7561/client_advanced.html#character-set-detection.

cc @Dreamsorcerer

Ah, sorry, I actually copied from the section above, which makes the same mistake with .close(). I wonder why Sphinx doesn't produce a warning...

Dunno. Maybe because of this ignore https://github.com/aio-libs/aiohttp/blob/6755796/docs/conf.py#L384 or some Sphinx but, or a corner case. Somebody needs to go through all those ignores and fix them.

@Dreamsorcerer so it looks like we don't run the normal docs build @ GHA and spellcheck that we do run doesn't emit those warnings: https://github.com/aio-libs/aiohttp/actions/runs/6018473896/job/16326723444#step:13:18.

OTOH, RTD does have those warnings in the log but isn't set up to turn them into errors: https://readthedocs.org/projects/aiohttp/builds/21761448/.

I think the CI was set up to fail on warnings at some point. Maybe that got removed, or I'm just confusing the repos…

Well, other warnings seem to fail CI in the docs spelling step (I'm not aware of another part that produces warnings):
https://github.com/aio-libs/aiohttp/actions/runs/6017616744/job/16324167311#step:13:86

john-parton · 2023-08-30T14:49:36Z

@john-parton If you could handle the backports, that'd be great. We'll need to:

I don't have the time to dedicate this week.

…ntSession parameter. (#7561) Co-authored-by: Sam Bull <[email protected]> (cherry picked from commit 6755796)

Dreamsorcerer · 2023-09-07T16:05:05Z

3.9 backport done: #7586
If you have some time to handle the 3.8 backport and finish #7561 (comment) soon, that'd be great. Hoping to get a new release out shortly.

john-parton · 2023-09-07T16:07:34Z

I'll take a crack at it later today.

Add fallback_charset_resolver ClientSession parameter. (#7561) Co-authored-by: Sam Bull <[email protected]> (cherry picked from commit 6755796) Co-authored-by: John Parton <[email protected]>

…ntSession parameter. (aio-libs#7561) Co-authored-by: Sam Bull <[email protected]> (cherry picked from commit 6755796)

john-parton · 2023-09-07T20:53:06Z

@Dreamsorcerer PR opened here: #7589

Add fallback_charset_resolver ClientSession parameter. (#7561) Co-authored-by: Sam Bull <[email protected]> (cherry picked from commit 6755796) --------- Co-authored-by: Sam Bull <[email protected]>

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [aiohttp](https://togithub.com/aio-libs/aiohttp) | `==3.8.5` -> `==3.8.6` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/aiohttp/3.8.6?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/aiohttp/3.8.6?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/aiohttp/3.8.5/3.8.6?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/aiohttp/3.8.5/3.8.6?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>aio-libs/aiohttp (aiohttp)</summary> ### [`v3.8.6`](https://togithub.com/aio-libs/aiohttp/blob/HEAD/CHANGES.rst#386-2023-10-07) [Compare Source](https://togithub.com/aio-libs/aiohttp/compare/v3.8.5...v3.8.6) \================== ## Security bugfixes - Upgraded the vendored copy of llhttp\_ to v9.1.3 -- by :user:`Dreamsorcerer` Thanks to :user:`kenballus` for reporting this, see GHSA-pjjw-qhg8-p2p9. .. \_llhttp: https://llhttp.org `#7647 <https://github.com/aio-libs/aiohttp/issues/7647>`\_ - Updated Python parser to comply with RFCs 9110/9112 -- by :user:`Dreamorcerer` Thanks to :user:`kenballus` for reporting this, see GHSA-gfw2-4jvh-wgfg. `#7663 <https://github.com/aio-libs/aiohttp/issues/7663>`\_ ## Deprecation - Added `fallback_charset_resolver` parameter in `ClientSession` to allow a user-supplied character set detection function. Character set detection will no longer be included in 3.9 as a default. If this feature is needed, please use `fallback_charset_resolver <https://docs.aiohttp.org/en/stable/client_advanced.html#character-set-detection>`\_. `#7561 <https://github.com/aio-libs/aiohttp/issues/7561>`\_ ## Features - Enabled lenient response parsing for more flexible parsing in the client (this should resolve some regressions when dealing with badly formatted HTTP responses). -- by :user:`Dreamsorcerer` `#7490 <https://github.com/aio-libs/aiohttp/issues/7490>`\_ ## Bugfixes - Fixed `PermissionError` when `.netrc` is unreadable due to permissions. `#7237 <https://github.com/aio-libs/aiohttp/issues/7237>`\_ - Fixed output of parsing errors pointing to a `\n`. -- by :user:`Dreamsorcerer` `#7468 <https://github.com/aio-libs/aiohttp/issues/7468>`\_ - Fixed `GunicornWebWorker` max_requests_jitter not working. `#7518 <https://github.com/aio-libs/aiohttp/issues/7518>`\_ - Fixed sorting in `filter_cookies` to use cookie with longest path. -- by :user:`marq24`. `#7577 <https://github.com/aio-libs/aiohttp/issues/7577>`\_ - Fixed display of `BadStatusLine` messages from llhttp\_. -- by :user:`Dreamsorcerer` `#7651 <https://github.com/aio-libs/aiohttp/issues/7651>`\_ *** </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/allenporter/pyrainbird).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.8.5 to 3.8.6. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/releases">aiohttp's">https://github.com/aio-libs/aiohttp/releases">aiohttp's releases</a>.</em></p> <blockquote> <h2>3.8.6</h2> <h2>Security bugfixes</h2> <ul> <li> <p>Upgraded the vendored copy of llhttp_ to v9.1.3 -- by :user:<code>Dreamsorcerer</code></p> <p>Thanks to :user:<code>kenballus</code> for reporting this, see <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/security/advisories/GHSA-pjjw-qhg8-p2p9">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-pjjw-qhg8-p2p9</a>.</p">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-pjjw-qhg8-p2p9">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-pjjw-qhg8-p2p9</a>.</p> <p>.. _llhttp: <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://llhttp.org">https://llhttp.org</a></p" rel="nofollow">https://llhttp.org">https://llhttp.org</a></p> <p>(<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7647">#7647</a>)</p">https://redirect.github.com/aio-libs/aiohttp/issues/7647">#7647</a>)</p> </li> <li> <p>Updated Python parser to comply with RFCs 9110/9112 -- by :user:<code>Dreamorcerer</code></p> <p>Thanks to :user:<code>kenballus</code> for reporting this, see <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/security/advisories/GHSA-gfw2-4jvh-wgfg">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-gfw2-4jvh-wgfg</a>.</p">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-gfw2-4jvh-wgfg">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-gfw2-4jvh-wgfg</a>.</p> <p>(<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7663">#7663</a>)</p">https://redirect.github.com/aio-libs/aiohttp/issues/7663">#7663</a>)</p> </li> </ul> <h2>Deprecation</h2> <ul> <li> <p>Added <code>fallback_charset_resolver</code> parameter in <code>ClientSession</code> to allow a user-supplied character set detection function.</p> <p>Character set detection will no longer be included in 3.9 as a default. If this feature is needed, please use <code>fallback_charset_resolver <https://docs.aiohttp.org/en/stable/client_advanced.html#character-set-detection></code>_.</p> <p>(<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7561">#7561</a>)</p">https://redirect.github.com/aio-libs/aiohttp/issues/7561">#7561</a>)</p> </li> </ul> <h2>Features</h2> <ul> <li> <p>Enabled lenient response parsing for more flexible parsing in the client (this should resolve some regressions when dealing with badly formatted HTTP responses). -- by :user:<code>Dreamsorcerer</code></p> <p>(<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7490">#7490</a>)</p">https://redirect.github.com/aio-libs/aiohttp/issues/7490">#7490</a>)</p> </li> </ul> <h2>Bugfixes</h2> <ul> <li> <p>Fixed <code>PermissionError</code> when <code>.netrc</code> is unreadable due to permissions.</p> <p>(<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7237">#7237</a>)</p">https://redirect.github.com/aio-libs/aiohttp/issues/7237">#7237</a>)</p> </li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst">aiohttp's">https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst">aiohttp's changelog</a>.</em></p> <blockquote> <h1>3.8.6 (2023-10-07)</h1> <h2>Security bugfixes</h2> <ul> <li> <p>Upgraded the vendored copy of llhttp_ to v9.1.3 -- by :user:<code>Dreamsorcerer</code></p> <p>Thanks to :user:<code>kenballus</code> for reporting this, see <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/security/advisories/GHSA-pjjw-qhg8-p2p9">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-pjjw-qhg8-p2p9</a>.</p">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-pjjw-qhg8-p2p9">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-pjjw-qhg8-p2p9</a>.</p> <p>.. _llhttp: <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://llhttp.org">https://llhttp.org</a></p" rel="nofollow">https://llhttp.org">https://llhttp.org</a></p> <p><code>[#7647](aio-libs/aiohttp#7647) <https://github.com/aio-libs/aiohttp/issues/7647></code>_</p> </li> <li> <p>Updated Python parser to comply with RFCs 9110/9112 -- by :user:<code>Dreamorcerer</code></p> <p>Thanks to :user:<code>kenballus</code> for reporting this, see <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/security/advisories/GHSA-gfw2-4jvh-wgfg">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-gfw2-4jvh-wgfg</a>.</p">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-gfw2-4jvh-wgfg">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-gfw2-4jvh-wgfg</a>.</p> <p><code>[#7663](aio-libs/aiohttp#7663) <https://github.com/aio-libs/aiohttp/issues/7663></code>_</p> </li> </ul> <h2>Deprecation</h2> <ul> <li> <p>Added <code>fallback_charset_resolver</code> parameter in <code>ClientSession</code> to allow a user-supplied character set detection function.</p> <p>Character set detection will no longer be included in 3.9 as a default. If this feature is needed, please use <code>fallback_charset_resolver <https://docs.aiohttp.org/en/stable/client_advanced.html#character-set-detection></code>_.</p> <p><code>[#7561](aio-libs/aiohttp#7561) <https://github.com/aio-libs/aiohttp/issues/7561></code>_</p> </li> </ul> <h2>Features</h2> <ul> <li> <p>Enabled lenient response parsing for more flexible parsing in the client (this should resolve some regressions when dealing with badly formatted HTTP responses). -- by :user:<code>Dreamsorcerer</code></p> <p><code>[#7490](aio-libs/aiohttp#7490) <https://github.com/aio-libs/aiohttp/issues/7490></code>_</p> </li> </ul> <h2>Bugfixes</h2> <ul> <li>Fixed <code>PermissionError</code> when <code>.netrc</code> is unreadable due to permissions.</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/commit/996de2629ef6b4c2934a7c04dfd49d0950d4c43b"><code>996de26</code></a">https://github.com/aio-libs/aiohttp/commit/996de2629ef6b4c2934a7c04dfd49d0950d4c43b"><code>996de26</code></a> Release v3.8.6 (<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7668">#7668</a>)</li">https://redirect.github.com/aio-libs/aiohttp/issues/7668">#7668</a>)</li> <li><a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/commit/8c128d4f042ca36ebdc55ecdd76099b7722331ba"><code>8c128d4</code></a">https://github.com/aio-libs/aiohttp/commit/8c128d4f042ca36ebdc55ecdd76099b7722331ba"><code>8c128d4</code></a> [PR <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7651">#7651</a>/45f98b7d">https://redirect.github.com/aio-libs/aiohttp/issues/7651">#7651</a>/45f98b7d backport][3.8] Fix BadStatusLine message (<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7666">#7666</a>)</li">https://redirect.github.com/aio-libs/aiohttp/issues/7666">#7666</a>)</li> <li><a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/commit/89b7df157886ff390cdcdc44ecf3c277045838b1"><code>89b7df1</code></a">https://github.com/aio-libs/aiohttp/commit/89b7df157886ff390cdcdc44ecf3c277045838b1"><code>89b7df1</code></a> Allow lax response parsing on Py parser (<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7663">#7663</a">https://redirect.github.com/aio-libs/aiohttp/issues/7663">#7663</a>) (<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7664">#7664</a>)</li">https://redirect.github.com/aio-libs/aiohttp/issues/7664">#7664</a>)</li> <li><a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/commit/d5c12ba890557a575c313bb3017910d7616fce3d"><code>d5c12ba</code></a">https://github.com/aio-libs/aiohttp/commit/d5c12ba890557a575c313bb3017910d7616fce3d"><code>d5c12ba</code></a> [PR <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7661">#7661</a>/85713a48">https://redirect.github.com/aio-libs/aiohttp/issues/7661">#7661</a>/85713a48 backport][3.8] Update Python parser for RFCs 9110/9112 (<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7">#7</a>...</li">https://redirect.github.com/aio-libs/aiohttp/issues/7">#7</a>...</li> <li><a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/commit/8a3977acac632d1f02aa7e047da51e27a717d724"><code>8a3977a</code></a">https://github.com/aio-libs/aiohttp/commit/8a3977acac632d1f02aa7e047da51e27a717d724"><code>8a3977a</code></a> [PR <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7272">#7272</a>/b2a7983a">https://redirect.github.com/aio-libs/aiohttp/issues/7272">#7272</a>/b2a7983a backport][3.8] Fix Read The Docs config (<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7650">#7650</a>)</li">https://redirect.github.com/aio-libs/aiohttp/issues/7650">#7650</a>)</li> <li><a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/commit/bcc416e533796d04fb8124ef1e7686b1f338767a"><code>bcc416e</code></a">https://github.com/aio-libs/aiohttp/commit/bcc416e533796d04fb8124ef1e7686b1f338767a"><code>bcc416e</code></a> [PR <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7647">#7647</a>/1303350e">https://redirect.github.com/aio-libs/aiohttp/issues/7647">#7647</a>/1303350e backport][3.8] Upgrade to llhttp 9.1.3 (<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7648">#7648</a>)</li">https://redirect.github.com/aio-libs/aiohttp/issues/7648">#7648</a>)</li> <li><a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/commit/b30c0cd2c96e57cc273ffe29c0313487b364f15a"><code>b30c0cd</code></a">https://github.com/aio-libs/aiohttp/commit/b30c0cd2c96e57cc273ffe29c0313487b364f15a"><code>b30c0cd</code></a> Remove chardet/charset-normalizer. (<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7589">#7589</a>)</li">https://redirect.github.com/aio-libs/aiohttp/issues/7589">#7589</a>)</li> <li><a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/commit/5946c7436044bae14617ef06ee7c530ed72622da"><code>5946c74</code></a">https://github.com/aio-libs/aiohttp/commit/5946c7436044bae14617ef06ee7c530ed72622da"><code>5946c74</code></a> CookieJar - return 'best-match' and not LIFO (<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7577">#7577</a">https://redirect.github.com/aio-libs/aiohttp/issues/7577">#7577</a>) (<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7588">#7588</a>)</li">https://redirect.github.com/aio-libs/aiohttp/issues/7588">#7588</a>)</li> <li><a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/commit/8c4ec62f5ba514479ef1c2e74741bc7fa33be3f4"><code>8c4ec62</code></a">https://github.com/aio-libs/aiohttp/commit/8c4ec62f5ba514479ef1c2e74741bc7fa33be3f4"><code>8c4ec62</code></a> [PR <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7518">#7518</a>/8bd42e74">https://redirect.github.com/aio-libs/aiohttp/issues/7518">#7518</a>/8bd42e74 backport][3.8] Fix GunicornWebWorker max_requests_jitter n...</li> <li><a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/commit/a0d234df392bd5cd67d378d31c9531c5ac87c07f"><code>a0d234d</code></a">https://github.com/aio-libs/aiohttp/commit/a0d234df392bd5cd67d378d31c9531c5ac87c07f"><code>a0d234d</code></a> Use lenient headers for response parser (<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7490">#7490</a">https://redirect.github.com/aio-libs/aiohttp/issues/7490">#7490</a>) (<a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://redirect.github.com/aio-libs/aiohttp/issues/7492">#7492</a>)</li">https://redirect.github.com/aio-libs/aiohttp/issues/7492">#7492</a>)</li> <li>Additional commits viewable in <a href="proxy.php?url=https%3A%2F%2Fredirect.github.com%2F%3Ca+href%3D"https://github.com/aio-libs/aiohttp/compare/v3.8.5...v3.8.6">compare">https://github.com/aio-libs/aiohttp/compare/v3.8.5...v3.8.6">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=aiohttp&package-manager=pip&previous-version=3.8.5&new-version=3.8.6)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>

john-parton requested review from asvetlov and webknjaz as code owners August 27, 2023 19:30

psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Aug 27, 2023

john-parton mentioned this pull request Aug 27, 2023

Replace cChardet with chardetng_py. #7559

Closed

5 tasks

john-parton force-pushed the feature/deprecate-charset-detection branch from df4fe75 to e1c7efe Compare August 28, 2023 14:49

john-parton mentioned this pull request Aug 28, 2023

Remove charset-normalizer and cchardet as dependencies. Update docs #7560

Closed

5 tasks

Dreamsorcerer reviewed Aug 28, 2023

View reviewed changes

Comment thread aiohttp/client.py Outdated

Comment thread aiohttp/client_reqrep.py

Comment thread aiohttp/client_reqrep.py Outdated

Comment thread setup.cfg

john-parton force-pushed the feature/deprecate-charset-detection branch from 4b6a570 to 9e303d1 Compare August 28, 2023 20:15

Dreamsorcerer reviewed Aug 28, 2023

View reviewed changes

Comment thread aiohttp/client_reqrep.py Outdated

Comment thread tests/test_client_response.py

Comment thread docs/client_reference.rst

Dreamsorcerer reviewed Aug 28, 2023

View reviewed changes

Comment thread tests/test_client_response.py

john-parton force-pushed the feature/deprecate-charset-detection branch from 31a778b to d8888ea Compare August 29, 2023 02:14

john-parton added 5 commits August 28, 2023 22:20

Remove unmaintained admonition from glossary.

3a85c65

Update glossary entries for charset-normalizer and cchardet with note…

3335841

… that they're in the glossary because they were previously used.

Document raising UnicodeDecodeError and link get_encoding.

6a173d7

Add advanced usage for character set detection.

a39683a

Dreamsorcerer and others added 4 commits August 29, 2023 23:47

Update client_advanced.rst

f1f58d1

Update client_reference.rst

b459bbf

Update test_client_response.py

e1e8346

[pre-commit.ci] auto fixes from pre-commit.com hooks

fe16619

for more information, see https://pre-commit.ci

Dreamsorcerer changed the title ~~Feature/deprecate charset detection~~ Remove chardet/charset-normalizer. Add fallback_charset_resolver ClientSession parameter. Aug 29, 2023

Update client_reference.rst

ef83b00

Dreamsorcerer reviewed Aug 29, 2023

View reviewed changes

Comment thread docs/client_reference.rst

Dreamsorcerer merged commit 6755796 into aio-libs:master Aug 29, 2023

Dreamsorcerer added the backport-3.8 label Aug 29, 2023

webknjaz reviewed Aug 30, 2023

View reviewed changes

Dreamsorcerer added a commit that referenced this pull request Sep 7, 2023

Remove chardet/charset-normalizer. Add fallback_charset_resolver Clie…

eacb758

…ntSession parameter. (#7561) Co-authored-by: Sam Bull <[email protected]> (cherry picked from commit 6755796)

Dreamsorcerer mentioned this pull request Sep 7, 2023

Remove chardet/charset-normalizer. #7586

Merged

john-parton mentioned this pull request Sep 7, 2023

Remove chardet/charset-normalizer. #7589

Merged

pseudoyim mentioned this pull request Sep 8, 2023

PKG-2774: Updates for v3.8.5 AnacondaRecipes/aiohttp-feedstock#7

Merged

bbhtt mentioned this pull request Oct 14, 2025

test_checker: Don't try to fetch 10 random bytes as JSON flathub-infra/flatpak-external-data-checker#402

Closed

Uh oh!

Conversation

john-parton commented Aug 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What do these changes do?

Are there changes in behavior for the user?

Related issue number

Checklist

Uh oh!

john-parton commented Aug 27, 2023

Uh oh!

Dreamsorcerer commented Aug 28, 2023

Uh oh!

john-parton commented Aug 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dreamsorcerer commented Aug 28, 2023

Uh oh!

john-parton commented Aug 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

john-parton commented Aug 28, 2023

Uh oh!

Dreamsorcerer commented Aug 28, 2023

Uh oh!

Dreamsorcerer commented Aug 28, 2023

Uh oh!

john-parton commented Aug 28, 2023

Uh oh!

Dreamsorcerer commented Aug 28, 2023

Uh oh!

john-parton commented Aug 28, 2023

Uh oh!

john-parton commented Aug 28, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Dreamsorcerer commented Aug 28, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

john-parton commented Aug 29, 2023

Uh oh!

Uh oh!

patchback bot commented Aug 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Backport to 3.9: 💔 cherry-picking failed — conflicts found

Backporting merged PR #7561 into master

Uh oh!

patchback bot commented Aug 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Backport to 3.8: 💔 cherry-picking failed — conflicts found

Backporting merged PR #7561 into master

Uh oh!

Dreamsorcerer commented Aug 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webknjaz Aug 30, 2023

Choose a reason for hiding this comment

Uh oh!

webknjaz Aug 30, 2023

Choose a reason for hiding this comment

Uh oh!

Dreamsorcerer Aug 30, 2023

Choose a reason for hiding this comment

Uh oh!

webknjaz Aug 30, 2023

Choose a reason for hiding this comment

Uh oh!

webknjaz Aug 30, 2023

Choose a reason for hiding this comment

Uh oh!

Dreamsorcerer Aug 30, 2023

Choose a reason for hiding this comment

Uh oh!

john-parton commented Aug 30, 2023

john-parton commented Aug 27, 2023 •

edited

Loading

john-parton commented Aug 28, 2023 •

edited

Loading

john-parton commented Aug 28, 2023 •

edited

Loading

patchback bot commented Aug 29, 2023 •

edited

Loading

patchback bot commented Aug 29, 2023 •

edited

Loading

Dreamsorcerer commented Aug 29, 2023 •

edited

Loading

Dreamsorcerer commented Sep 7, 2023 •

edited

Loading