Do not consider bare TypeVar not overlapping with None for reachability analysis by sterliakov · Pull Request #18138 · python/mypy

sterliakov · 2024-11-10T17:55:54Z

Simply allowing such intersection was insufficient: existing binder logic widened the type to T | None after the is None check.

This PR extends the binder logic to prevent constructing a union type when all conditional branches are reachable and contain no assignments: checking if isinstance(something, Something) does not change the type of something after the end of the if block.

sterliakov · 2024-11-10T20:48:55Z

CI failed with the Windows problem I faced multiple times before, the rest is green.

Primer diff is amazing this time! Every diff line seems to be an improvement.

jinja2: unrelated meet-vs-join problem. Minimal repro (playground):

class A: pass
class B: pass

foos: tuple[A, ...] | tuple[B, ...]

for _, foo in enumerate(foos):
    reveal_type(foo)  # N: Revealed type is "builtins.object"

prefect: the error is correct. I don't know why it wasn't emitted previously, though: there's assignment of different types to an unannotated variable in if/else branches.
sphinx/search/__init__.py: bug in docutils-stubs (children is declared in Node, but not in stubs), and mypy is correct now.
sphinx/ext/graphviz.py: correct! This attribute isn't declared on any of the classes. Still don't know how it relates to this change, though...
schema_salad: correct! self.name is Any | None and is passed to a function expecting str.

kopf: another unrelated bug:( Minimal repro (playground with sync/async comparison):

from typing import Any, Callable, TypeVar

T = TypeVar("T")

async def call(fn: Callable[[], T]) -> T:
    return fn()

make_x: Any

async def fn() -> None:
    x: Any | None = None
    x = await call(make_x)
    reveal_type(x)  # N: Revealed type is "Union[Any, None]"

pytest: correct. There's really no error.
paroxython: correct, there's a narrowing not supported by mypy
pandera: fine, undeclared attributes

cki-lib: correct! It boils down to the following:

x: Any
if x is None:
    ...
do_something_with_not_none(x)

pygithub: correct! That is really a type error.
discord: all correct! Especially nice to get rid of Item "object" of the upper bound "Cog | None" - that was strange.
steam: both correct! Second one suffered from needless type widening after isinstance
ibis: all correct!

JukkaL

Nice, based on mypy_primer output this fixes multiple bugs, and makes the inferred types more consistent. Just left some minor comments about style (optional), looks good otherwise.

mypy/binder.py

mypy/checker.py

…pevar-overlap

github-actions · 2024-11-15T18:20:39Z

Diff from mypy_primer, showing the effect of this PR on open source code:

jinja (https://github.com/pallets/jinja)
+ src/jinja2/lexer.py:761: error: "object" not callable  [operator]
+ src/jinja2/lexer.py:781: error: Incompatible types in "yield" (actual type "Tuple[int, object, str]", expected type "Tuple[int, str, str]")  [misc]

prefect (https://github.com/PrefectHQ/prefect)
+ src/prefect/states.py:229: error: Incompatible types in assignment (expression has type "BaseException", variable has type "ResultRecord[Any]")  [assignment]

sphinx (https://github.com/sphinx-doc/sphinx)
+ sphinx/search/__init__.py:628:18: error: "Node" has no attribute "children"  [attr-defined]
+ sphinx/ext/graphviz.py: note: In function "render_dot":
+ sphinx/ext/graphviz.py:307:9: error: Item "StandaloneHTMLBuilder" of "StandaloneHTMLBuilder | LaTeXBuilder | TexinfoBuilder" has no attribute "_graphviz_warned_dot"  [union-attr]
+ sphinx/ext/graphviz.py:307:9: error: Item "LaTeXBuilder" of "StandaloneHTMLBuilder | LaTeXBuilder | TexinfoBuilder" has no attribute "_graphviz_warned_dot"  [union-attr]
+ sphinx/ext/graphviz.py:307:9: error: Item "TexinfoBuilder" of "StandaloneHTMLBuilder | LaTeXBuilder | TexinfoBuilder" has no attribute "_graphviz_warned_dot"  [union-attr]

schema_salad (https://github.com/common-workflow-language/schema_salad)
+ schema_salad/metaschema.py: note: In member "save" of class "EnumSchema":
+ schema_salad/metaschema.py:1882:49: error: Argument 2 to "save_relative_uri" has incompatible type "Any | None"; expected "str"  [arg-type]
+ schema_salad/metaschema.py:1886:49: error: Argument "base_url" to "save" has incompatible type "Any | None"; expected "str"  [arg-type]
+ schema_salad/metaschema.py: note: In member "save" of class "SaladEnumSchema":
+ schema_salad/metaschema.py:5378:51: error: Argument "base_url" to "save" has incompatible type "Any | None"; expected "str"  [arg-type]
+ schema_salad/metaschema.py:5381:49: error: Argument 2 to "save_relative_uri" has incompatible type "Any | None"; expected "str"  [arg-type]
+ schema_salad/metaschema.py:5385:49: error: Argument "base_url" to "save" has incompatible type "Any | None"; expected "str"  [arg-type]
+ schema_salad/metaschema.py:5389:47: error: Argument "base_url" to "save" has incompatible type "Any | None"; expected "str"  [arg-type]
+ schema_salad/metaschema.py:5392:51: error: Argument 2 to "save_relative_uri" has incompatible type "Any | None"; expected "str"  [arg-type]
+ schema_salad/metaschema.py:5395:50: error: Argument 2 to "save_relative_uri" has incompatible type "Any | None"; expected "str"  [arg-type]
+ schema_salad/metaschema.py:5398:50: error: Argument 2 to "save_relative_uri" has incompatible type "Any | None"; expected "str"  [arg-type]
+ schema_salad/metaschema.py:5404:26: error: Argument "base_url" to "save" has incompatible type "Any | None"; expected "str"  [arg-type]
+ schema_salad/metaschema.py:5411:26: error: Argument "base_url" to "save" has incompatible type "Any | None"; expected "str"  [arg-type]
+ schema_salad/metaschema.py:5415:49: error: Argument 2 to "save_relative_uri" has incompatible type "Any | None"; expected "str"  [arg-type]

kopf (https://github.com/nolar/kopf)
+ kopf/_kits/webhooks.py:539: error: Item "None" of "Any | None" has no attribute "public_url"  [union-attr]

pytest (https://github.com/pytest-dev/pytest)
- src/_pytest/_code/code.py:1071: error: Argument 1 to "from_exception" of "ExceptionInfo" has incompatible type "Optional[BaseException]"; expected "BaseException"  [arg-type]
- src/_pytest/_code/code.py:1071: error: Item "None" of "Optional[BaseException]" has no attribute "__traceback__"  [union-attr]
- src/_pytest/_code/code.py:1077: error: Argument 1 to "from_exception" of "ExceptionInfo" has incompatible type "Optional[BaseException]"; expected "BaseException"  [arg-type]
- src/_pytest/_code/code.py:1077: error: Item "None" of "Optional[BaseException]" has no attribute "__traceback__"  [union-attr]

paroxython (https://github.com/laowantong/paroxython)
+ paroxython/flatten_ast.py:401: error: "AST" has no attribute "lineno"  [attr-defined]

pandera (https://github.com/pandera-dev/pandera)
- pandera/api/dataframe/model.py:72: error: Unused "type: ignore" comment  [unused-ignore]
- pandera/api/dataframe/model.py:77: error: Unused "type: ignore" comment  [unused-ignore]

cki-lib (https://gitlab.com/cki-project/cki-lib)
- cki_lib/s3bucket.py:57: error: Item "None" of "Any | None" has no attribute "get_credentials"  [union-attr]

PyGithub (https://github.com/PyGithub/PyGithub)
- github/GithubObject.py:315: error: Unused "type: ignore" comment  [unused-ignore]

discord.py (https://github.com/Rapptz/discord.py)
- discord/ext/commands/core.py:590: error: Unused "type: ignore" comment  [unused-ignore]
- discord/ext/commands/core.py:645: error: Item "object" of the upper bound "Cog | None" of type variable "CogT" has no attribute "cog_command_error"  [union-attr]
- discord/ext/commands/core.py:884: error: Item "object" of the upper bound "Cog | None" of type variable "CogT" has no attribute "cog_before_invoke"  [union-attr]
- discord/ext/commands/core.py:904: error: Item "object" of the upper bound "Cog | None" of type variable "CogT" has no attribute "cog_after_invoke"  [union-attr]
- discord/ext/commands/core.py:1148: error: "type[CogT]" has no attribute "__cog_name__"  [attr-defined]
- discord/ext/commands/core.py:1277: error: Item "object" of the upper bound "Cog | None" of type variable "CogT" has no attribute "cog_check"  [union-attr]

steam.py (https://github.com/Gobot1234/steam.py)
- steam/app.py:130: error: Argument "game_extra_info" to "CMsgClientGamesPlayedGamePlayed" has incompatible type "NameT"; expected "str"  [arg-type]
- steam/ext/commands/commands.py:923: error: Incompatible return value type (got "Command[Any, Any, Any] | MaybeCommandT", expected "MaybeBool | MaybeCommandT")  [return-value]

ibis (https://github.com/ibis-project/ibis)
+ ibis/util.py:137: error: Incompatible return value type (got "tuple[()]", expected "tuple[V]")  [return-value]
- ibis/selectors.py:708: error: Argument 1 to "ColumnIndex" has incompatible type "int | slice[Any, Any, Any] | Iterable[int | str]"; expected "str | int | Slice | tuple[int | str, ...]"  [arg-type]
+ ibis/selectors.py:708: error: Argument 1 to "ColumnIndex" has incompatible type "str | int | slice[Any, Any, Any] | Iterable[int | str]"; expected "str | int | Slice | tuple[int | str, ...]"  [arg-type]
- ibis/expr/types/relations.py:2629: error: Argument 3 to "DropNull" has incompatible type "Sequence[str] | None"; expected "tuple[Value[Any, Columnar], ...] | None"  [arg-type]
+ ibis/expr/types/relations.py:2629: error: Argument 3 to "DropNull" has incompatible type "Sequence[str] | str | None"; expected "tuple[Value[Any, Columnar], ...] | None"  [arg-type]
+ ibis/backends/sqlite/__init__.py:497: error: Item "None" of "Any | Any | Any | Any | Any | None" has no attribute "schema"  [union-attr]
- ibis/backends/duckdb/__init__.py:532: error: Argument 1 to "read_parquet" of "Backend" has incompatible type "str | Any | Path | list[Any] | tuple[Any, ...]"; expected "str | Iterable[str]"  [arg-type]
+ ibis/backends/duckdb/__init__.py:532: error: Argument 1 to "read_parquet" of "Backend" has incompatible type "str | Path | Any"; expected "str | Iterable[str]"  [arg-type]
- ibis/backends/duckdb/__init__.py:536: error: Argument 1 to "read_csv" of "Backend" has incompatible type "str | Any | Path | list[Any] | tuple[Any, ...]"; expected "str | list[str] | tuple[str]"  [arg-type]
+ ibis/backends/duckdb/__init__.py:536: error: Argument 1 to "read_csv" of "Backend" has incompatible type "str | Path | Any"; expected "str | list[str] | tuple[str]"  [arg-type]
- ibis/backends/duckdb/__init__.py:538: error: Argument 1 to "read_postgres" of "Backend" has incompatible type "str | Any | Path | list[Any] | tuple[Any, ...]"; expected "str"  [arg-type]
+ ibis/backends/duckdb/__init__.py:538: error: Argument 1 to "read_postgres" of "Backend" has incompatible type "str | Path | Any"; expected "str"  [arg-type]
- ibis/backends/pyspark/__init__.py:962: error: Argument 1 to "read_parquet" of "Backend" has incompatible type "str | Any | Path | list[Any] | tuple[Any, ...]"; expected "str | Path"  [arg-type]
- ibis/backends/pyspark/__init__.py:966: error: Argument 1 to "read_csv" of "Backend" has incompatible type "str | Any | Path | list[Any] | tuple[Any, ...]"; expected "str | list[str] | tuple[str]"  [arg-type]
+ ibis/backends/pyspark/__init__.py:966: error: Argument 1 to "read_csv" of "Backend" has incompatible type "str | Path | Any"; expected "str | list[str] | tuple[str]"  [arg-type]

JukkaL

Thanks for the updates, LGTM!

Regression test for #11839 resolved with #18138.

PetterS · 2025-02-23T10:01:39Z

Very nice!

Allow TypeVar & None intersection when checking reachability.

42620b0

sterliakov marked this pull request as draft November 10, 2024 18:04

This comment has been minimized.

Sign in to view

sterliakov added 2 commits November 10, 2024 19:18

Remove cast that caused a problem

7160e4e

Move cases with 3.10+ syntax to python310 file

207c75a

This comment has been minimized.

Sign in to view

sterliakov marked this pull request as ready for review November 10, 2024 20:49

Merge branch 'master' into bugfix/st-none-typevar-overlap

374d4dd

This comment has been minimized.

Sign in to view

JukkaL approved these changes Nov 15, 2024

View reviewed changes

mypy/binder.py Outdated Show resolved Hide resolved

mypy/checker.py Outdated Show resolved Hide resolved

sterliakov added 2 commits November 15, 2024 18:45

Merge remote-tracking branch 'upstream/master' into bugfix/st-none-ty…

c2af54f

…pevar-overlap

Avoid passing boolean flags positionally

70aa977

JukkaL approved these changes Nov 21, 2024

View reviewed changes

JukkaL merged commit 08340c2 into python:master Nov 21, 2024

This was referenced Dec 8, 2024

Attribute of type A | B | None becomes object after narrowing checks #12009

Closed

Regression with type narrowing of bool #11839

Closed

cdce8p mentioned this pull request Dec 9, 2024

Add regression test for isinstance narrowing #18272

Merged

JelleZijlstra pushed a commit that referenced this pull request Dec 9, 2024

Add regression test for isinstance narrowing (#18272)

ac89577

Regression test for #11839 resolved with #18138.

hauntsaninja mentioned this pull request Dec 28, 2024

Mypy forgets type information after a nested if-statement. #5378

Closed

sterliakov mentioned this pull request Jan 16, 2025

attribute is incorrectly Any when ParamSpec decorated function is referenced in itself followed by an if statement with a class that has a method in it #16496

Closed

hauntsaninja mentioned this pull request Jun 27, 2025

Reachable code marked as unreachable in generic class with None type #8871

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Do not consider bare TypeVar not overlapping with None for reachability analysis#18138

Do not consider bare TypeVar not overlapping with None for reachability analysis#18138
JukkaL merged 6 commits intopython:masterfrom
sterliakov:bugfix/st-none-typevar-overlap

sterliakov commented Nov 10, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

sterliakov commented Nov 10, 2024 •

edited

Loading

Uh oh!

This comment has been minimized.

JukkaL left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Nov 15, 2024

Uh oh!

JukkaL left a comment

Uh oh!

PetterS commented Feb 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

sterliakov commented Nov 10, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

sterliakov commented Nov 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

JukkaL left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Nov 15, 2024

Uh oh!

JukkaL left a comment

Choose a reason for hiding this comment

Uh oh!

PetterS commented Feb 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sterliakov commented Nov 10, 2024 •

edited

Loading