[ty] Smarter semantic token classification for attribute access on union type#23841
[ty] Smarter semantic token classification for attribute access on union type#23841MichaReiser merged 7 commits intoastral-sh:mainfrom
Conversation
access on objects with a union type. Mirrors the semantic_tokens::tests::attribute_classification test. Also fixes that test in regards to `property` classification. Previously the comment did not match the result.
on union types. Update snapshot test from previous commit to the desired results.
crates/ty_ide/src/semantic_tokens.rs
Outdated
| y = obj.method # method should be method (bound method) | ||
| z = obj.CONSTANT # CONSTANT should be variable with readonly modifier | ||
| w = obj.prop # prop should be property | ||
| w = obj.prop # prop should be variable on an instance |
There was a problem hiding this comment.
Let's leave this comment as is. I do think we want to classify this as a property and I find variable on an instance more difficult to understand. We can rephrase it to instance property if you prefer that.
crates/ty_ide/src/semantic_tokens.rs
Outdated
| if Self::is_constant_name(attr_name_str) { | ||
| modifiers |= SemanticTokenModifier::READONLY; | ||
| } | ||
| // If we got a union type, check if all elements of the union belong to the same |
There was a problem hiding this comment.
The issue with foobar.prop is that ty isn't the type of the value (that is, the type of foobar) but the type of foobar.prop and foobar.pop inferes to Unknown (because the properties lack a return type annotation).
If we change the code snippet like so
from random import random
class Foo:
@property
def prop(self) -> int:
return 4
class Bar:
@property
def prop(self) -> int:
return 3
foobar: Foo | Bar = Foo() if random() else Bar()
w = foobar.prop # prop should be variable on an instancethen ty is int as expected.
Which makes me question if the fix here is correct. I think what we want is to infer the type of foobar and if that's a union, classify variant.attr for each union variant.
There was a problem hiding this comment.
Mhm yes, i agree with your analysis, but i don't think the semantic token type changes between Unknown and int, because the same match arm will be taken?
So i think the match arm to check for a property here does not do the correct thing for foobar.prop as you said, but i think that preexists this change and is also happening in the preexisting, other test where i changed the comment.
Regarding your second sentence; i think this is already done somewhat by the type inference anyway? Since at least for foobar.method the inferred type is a union of bound methods, which we can than classify.
Would you be fine with deferring the bugfix for property classification or would you rather have it fixed in this PR? (I feel like that fix could be more involved than the simple loop i added here until now 😅 )
There was a problem hiding this comment.
I don't think the fix requires that many changes. It's probably something like:
let value_ty = value.inferred_type(model);
if let Some(union) = value_ty.as_union() {
// classify each variant
// if all classify the same, return early
}
// Old, unchanged logic:
let ty = expr.inferred_type(model);
...
There was a problem hiding this comment.
I thought a bit more about this and i think this change alone would still not be perfect, consider this case:
from random import random
class Baz:
if random():
@property
def prop(self) -> int:
return 42
else:
@property
def prop(self) -> str:
return "hello"
baz = Baz()
x = baz.propwhere the value_ty would not be a union.
(There may actually be a real bug in type inference for the above example: astral-sh/ty#3032)
I think to fix the property classification you would need to look up the attribute name on the class of the object and check if it is defined with @property. Is that something that is already supported by some function you are aware of in the codebase?
There was a problem hiding this comment.
I'll create a follow up issue for this
crates/ty_ide/src/semantic_tokens.rs
Outdated
| } else { | ||
| None | ||
| } | ||
| })() |
There was a problem hiding this comment.
What's the reason that this is an immediately invoked lambda?
There was a problem hiding this comment.
I thought i could combine it with the outer match somehow, but iiuc r-a transformed it to a "if let guard" which seems to only be stabilized very recently. I'll change it.
There was a problem hiding this comment.
lmk if you find the functional style even less appealing 😅
- Move union check above match
|
I tried implementing the suggestion from #23841 (comment) let value_ty = attr
.value
.inferred_type(self.model)
.unwrap_or(Type::unknown());
if let Some(class_ty) = value_ty.as_class_literal() {
let place = class_ty.class_member(
self.model.db(),
&attr.attr,
MemberLookupPolicy::MRO_NO_OBJECT_FALLBACK,
);
// Check if the member is a property
if let Ok(typ) = place.into_lookup_result(self.model.db()) {
if typ.inner_type().is_property_instance() {
self.add_token(
&attr.attr,
SemanticTokenType::Property,
SemanticTokenModifier::empty(),
);
}
}
}which did not work and required making many more things public. So all in all i don't feel comfortable making that change as well and opening back up for review. |
Typing conformance resultsNo changes detected ✅Current numbersThe percentage of diagnostics emitted that were expected errors held steady at 85.29%. The percentage of expected errors that received a diagnostic held steady at 78.13%. The number of fully passing files held steady at 64/132. |
|
Thank you. Your approach makes sense. I simplified it a bit to avoid using |
Memory usage reportMemory usage unchanged ✅ |
|
| Lint rule | Added | Removed | Changed |
|---|---|---|---|
invalid-await |
40 | 0 | 0 |
invalid-return-type |
1 | 0 | 0 |
| Total | 41 | 0 | 0 |
* main: [ty] Filter out unsatisfiable inference attempts during generic call narrowing (#24025) [ty] Avoid inferring intersection types for call arguments (#23933) [ty] Pin mypy_primer in `setup_primer_project.py` (#24020) [`pycodestyle`] Recognize `pyrefly:` as a pragma comment (`E501`) (#24019) Add company AI policy to contributing guide (#24021) [ty] Remove the mypy_primer CI workflow (#24016) Update prek dependencies (#23980) [ty] Smarter semantic token classification for attribute access on union type (#23841) [ty] ecosystem-analyzer: Inline diffs and panic messages (#24015) [ty] Improve `.toml` support in the ty playground (#23476) PEP 639 license information (#19661)
|
no worries at all and your implementation is way cleaner! 😄 |
Summary
ref ty/#791
Updates the semantic token classification for attribute access on union types. It now checks if all elements of the union belong to the same type variant and if so, recursively tries to classify that common variant. Imo this produces nicer results especially when calling a method on an object of some union type.
I decided not to specifically check if all union elements are of some fixed set of variants (e.g. only
Type::BoundMethodetc), because i felt like duplicating the logic may be brittle. That means, that some hypothetical type ofUnion[Union[BoundMethod, BoundMethod], Union[something_else, something_else]]may be misclassified asMethodin this case, by i am unsure if it is possible to create such a type.
I also did not change behavior for free function classification, because i could not think of a way where this matters, i.e. creating an alias to a function, already changes the classification to `Variable` without involving unions at all:https://play.ty.dev/d8acae76-5642-4841-b4cf-8fd6066cea87
I also added a line in the existing
semantic_token::test::attribute_classificationtest, where there was no test forMyClass.prop, which actually classifies asProperty.Test Plan
Added these two unit test to
semantic_tokens.rs:attribute_on_union_1
Playground 1
main.pyattribute_on_union_2
Playground 2
main.pyattributes_on_union_3
Playground 3
This one is more of a negative test, where the union does not contain elements of the same type variant and the classification should therefore fallback to just
Variable.main.py