Interpretation of type Any & T

@CarliJoy

Following on from the lengthy discussion here #1, it was concluded that Any & T is the best resultant type for intersecting Any. This is symmetric with the way union works Any | T -> Any | T. Conversation on this new thread will investigate the types of the attributes and methods of this intersection type. Just a note, inspired by @CarliJoy we can number and summarise different interpretations. We should also seek to again create a real world example that explores the new types, and compares different interpretations.

Also if a summary of each new interpretation could be succinctly provided alongside a usage example that would be grand :)

Assume this simple but verbose example.
I will handle only accessing properties at the moment.

Module a_untyped

class A:
    common: str
    a: int
    a_str: str

    def fn_a(self, a: int) -> int:
        return a

    def fn_mix(self, mix: int) -> int:

    ab_mix: "A"

Module b_typed

class B:
    common: str
    b: str

    def fn_b(self, b: str) -> str:
        return b

    def fn_mix(self, mix: str) -> str:
        return mix

    ab_mix: "B"

`main.py`

import typing as t
from a_untyped import A
from b_typed import B

def needs_int(x: int) -> None:
    """Example function that needs to be called with integer to work"""

def needs_str(x: str) -> None:
    """Example function that needs to be called with string to work"""

def can_handle_any(x: t.Any) -> None:
    """Example function that is able to handle any type"""

def use_ab_intersected(x: A & B) -> None:
    # All this examples should work, as they work in runtime
    needs_str(x.common)

    needs_int(x.a)
    needs_int(x.fn_a(1))
    needs_int(x.fn_mix(1))  # should be detected as the correct call
    needs_int(len(x.a_str.split()[0]))
    can_handle_any(x.a)

    needs_str(x.b)
    needs_str(x.fn_b("b"))
    needs_str(x.fn_mix("b"))
    can_handle_any(x.b)

    use_ab_intersected(x.ab_mix)  # Intersect the property is valid
    needs_int(x.ab_mix.b)
    can_handle_any(x.ab_mix.a)

    # This should raise an error detected by the type checker (if both types known)
    invalid1 = x.not_existing
    invalid2 = x.fn_mix(object)

In general even so a variable is Any we can assume that runtime implementation of the object hold by this variable has a concrete implementation.

But if package is not marked as typed, everything is imported as Any, so let's assume A is typed as Any.

As an user I would want the following behaviour. This is probably not possible, but we should check rules against this example.

def use_ab_intersected(x: A & B) -> None:
    # All this examples should work, as they work in runtime
    needs_str(x.common)

    needs_int(x.a) # type: ignore - required
    needs_int(x.fn_a(1))  # type: ignore - required
    needs_int(x.fn_mix(1)) # type: ignore - required
    needs_int(len(t.cast(str, x.a_str).split()[0]))  # this needs a cast
    can_handle_any(x.a) # should be fine, because it is `Any` already

    needs_str(x.b)  # Hard requirement, this *must* work
    needs_str(x.fn_b("b"))  # Hard requirement, this *must* work
    needs_str(x.fn_mix("b"))  # Hard requirement, this *must* work
    can_handle_any(x.b) # Hard requirement, this *must* work

    use_ab_intersected(x.ab_mix)  # Hard requirement, this *must* work
    needs_int(x.ab_mix.b) # Hard requirement, this *must* work
    can_handle_any(x.ab_mix.a) # Hard requirement, this *must* work

    # This should raise an error detected by the type checker (if both types known)
    invalid1 = x.not_existing  # type checker should complain
    invalid2 = x.fn_mix(object) # type checker should complain

@ippeiukai

Hmm looking at the example it seems that we have to decided between allowing can_handle_any(x.a) and raising type errors on invalid accesses.

As for the most A related call we have to # type: ignore or cast things anyway, for accessing properties I would tend for the T & Any -> T.

This is in contract to the example given by @ippeiukai here.

Copied and modified below for easier reference:

By the way, in terms of gradual typing, I feel Intersection[T, Any] = T is awful.

Module 1

some_fn(some_obj)

Module 2

def wrap_obj(val: T) -> T & SomeAddedTrait:
  …

a = wrap_obj(some_obj)
some_fn(a)

Here, let’s assume some_fn and some_obj are provided by two different libraries both currently untyped. What if some_fn got typed before some_obj gets typed? I expect I can do anything to a that I can do to some_obj. I don’t expect a to be reduced to mere SomeAddedTrait.

TODO: formulate example more concrete, I don't get it atm,@ippeiukai could you formulate it with proper files and imports like I did above

@DiscordLiz

By the way, in terms of gradual typing, I feel Intersection[T, Any] = T is awful.

I'll type up some examples when I have more time, hopefully later today, but both of the following are true here:

In terms of gradual typing, T & Any = T is awful
In terms of gradual typing T & Any = Any or otherwise unchecked (one interpretation of Any & T) is awful

I think @DiscordLiz had a good point in the prior thread about Any & T never being an intentional construct. When is this going to come up? Do people intending on having Any have a use case for intersections, or is this sufficiently advanced that people should be expected to have typed their bases better and we should err on the side of safety for users on this? I think we need to start from where we expect this to occur before we can evaluate the impact of any choice. It's clear it can only happen at the boundary between typed and either untyped or gradually typed things, but can we get more specific than that and determine if there's ever an intentional use for T & Any which isn't better served by other existing things?

@CarliJoy

@CarliJoy Thanks for this example! It's very thorough - I've reduced it down a bit to consider a particular subcase, not because the others aren't relevant, but just because I think it might be easier to discuss in pieces:

import typing
from intersection_examples import Intersection
class A:
    common: str

# A effectively gets cast to Any, so nobody knows the type common even exists

class B:
    common: int

def needs_int(x: int) -> None:
    """Example function that needs to be called with integer to work"""

def needs_str(x: str) -> None:
    """Example function that needs to be called with string to work"""

def use_ab_intersected(x: Intersection[A, B]) -> None:
    # Note: A&B is an LSP violation, but we don't know that it is, because
    # we don't know the type of A
    needs_int(x.common) # should be fine - this is a property of B
    needs_str(x.common) # This should also be fine - if A is used at runtime
    # It could be string? Or could it?
    
    # Follow up idea - do we have to assume the intersection created with Any
    # has not created any LSP violations?

My conclusion from this is that similarly to other types we intersect, perhaps we have to assume LSP has not been violated by the intersection with the Any type - although we have no way to be sure

I am still groping my way around this subject. I am unsure of the technical meaning of "gradual typing" that seems to be used by some participants.

In the meantime, I have an example, which has actually convinced me that this proposal is fine:

def f(a: Any & int):
    x = a + 1
    reveal_type(x)

Given that a: Any & T means thata has all the attributes of T and they have the types specified by T, in this example, x has __add__(self, arg: int) -> int so the type of x should be revealed as int. That matches my intuition.

@DiscordLiz

My conclusion from this is that similarly to other types we intersect, perhaps we have to assume LSP has not been violated by the intersection with the Any type - although we have no way to be sure

Which makes it intersections all the way down and no different than being Any, cause we have to assume LSP wasn't violated.

If that were intentional, someone could skip the intersection and just write Any, and that works today.

I think @DiscordLiz had a good point in the prior thread about Any & T never being an intentional construct. When is this going to come up? Do people intending on having Any have a use case for intersections, or is this sufficiently advanced that people should be expected to have typed their bases better and we should err on the side of safety for users on this?

I don't think it's intentional, but I do think it can happen. The middle ground (that was option 5? and we're only considering options 4 and 5 now? Sorry, I seem to have missed the reason for splitting this to a new thread.) seems to avoid the accidental issues possible in 4, but I'd like more examples of it from @mikeshardmind

Given that a: Any & T means thata has all the attributes of T and they have the types specified by T, in this example, x has add(self, arg: int) -> int so the type of x should be revealed as int. That matches my intuition.

That would be the behavior in option 5. Option 4, the return type can only be int & Any, it's recursive with that option, and more pervasive than I think is beneficial.

Given that a: Any & T means thata has all the attributes of T and they have the types specified by T, in this example, x has __add__(self, arg: int) -> int so the type of x should be revealed as int. That matches my intuition.

Just something for you to consider. Let's just look at an example that doesn't involve Any:

class C:
  def f(self) -> None: pass
class D:
  def g(self) -> None: pass
class E(C, D): pass

def f(x: C):
  if isinstance(x, D):
    reveal_type(x)  # C & D
    x.f()  # should be okay since x inherits from C
    x.g()  # should be okay since x inherits from D

f(E())

This is why the interface of C&D is the union of the interfaces of C and D.

Now, consider:

what is the interface of Any?
what is the interface of Any & C?
So if x has type Any & C, what is the type of x.f(), x.g(), and x.h()?

It's really worth reading Eric's elucidating comment.

For reference, there are two proposals being considered here. I'll call them 4 and 5 (relating to names from #1).
Proposal 4 says: x.f(), x.g(), x.h() are all Any.
Proposal 5 says: x.g(), x.h() are both Any, but x.f() is None.

Even though I'm not yet in favor of proposal 5, I have a way to illustrate the problem it may be trying to solve:

Consider what isinstance does:

x: Any
assert isinstance(x, U)
y: T
assert isinstance(y, U)

The general case illustrated by y gives y type T & U. But there's a special case: if we start with something of type Any, then we don't want to end up with Any & U, and we instead set x's type to U. The reason we don't want Any & U is because that would cripple type checking: You would not be able to check most things related to the interface of x.

Essentially, there are two isinstance operators. One acts is an intersection that acts on ordinary types and type variables, and the other is a type replacement that acts replaces an Any type with whatever type is provided. Currently, type checkers do a lot of type replacement, and only a rudimentary intersection in some special cases.

I think the people in camp 5 are worried that this pernicious behavior that we avoid using the type replacement will creep in when we enable intersections generally.

I considered a 6th proposal: encourage to choose replacement if they want to protect against the spread of Any. This should only be relevant in some special cases involving things like unbounded generics. This can be done like this:

def f[T](x: T):
     assert isinstance(x, U)
     # Normally, this would have type T & U, but you can force replacement by casting.
     x_as_u = cast(x, U)
     # The over-broad interface of T & U is avoided.

This would allow us to leave the ordinary meaning of T & Any alone without any special cases, but potentially eliminate the problems associated with it.

I think we need to see some concrete examples to test if this could work.

If I need a cast everywhere I use an intersection to avoid issues with Any, then I don't need an intersection, I can cast to a protocol that matches my use.

You're just replacing one wart with another worse one that disables other checks here, and requires runtime checkable protocols and a runtime cast use.

The reason we don't want Any & U is because that would cripple type checking: You would not be able to check most things related to the interface of x.

That's exactly the issue with option 4. It cripples type checking for the person with fully typed info on their side of the intersection and is no better for the person with Any on their yet untyped object than not using an intersection and using Any.

That's exactly the issue with option 4. It cripples type checking for the person with fully typed info

So we're on the same page, and I think I understand your point of view.

I see how the deference proposal prevents false negatives, but it does so at the cost of false positives. Historically, the Python typing community has preferred solutions that avoid false positives even if they cause false negatives. For example, the adoption of the sequence protocol into the typeshed causes false negatives in order to avoid false positives.

I think we just need to see at least a few common examples to get a better handle of what the tradeoffs are.

If I need a cast everywhere I use an intersection

I don't think it would be "everywhere you use an intersection", but as I stated in the previous comment, it would only be in the cases of intersections with unbound type variables, and similar. Like I said, this was just something I thought of. We need to see some examples to know what it would look like.

@gvanrossum

I am still groping my way around this subject. I am unsure of the technical meaning of "gradual typing" that seems to be used by some participants.

In the meantime, I have an example, which has actually convinced me that this proposal is fine:
def f(a: Any & int):
    x = a + 1
    reveal_type(x)
Given that a: Any & T means thata has all the attributes of T and they have the types specified by T, in this example, x has __add__(self, arg: int) -> int so the type of x should be revealed as int. That matches my intuition.

@gvanrossum I think your example makes sense, but the controversial part comes from the other side of the intersection - as you describe the result must have all the properties of int so using it with __add__ makes sense here. But on the other side we have Any, and therefore x must also have all the properties of Any. What does it mean to have all the properties of Any? If it encompasses everything, can we say it says any property we like? What if Any also has a method __add__ that receives str?

My conclusion from this is that perhaps it has to be some subset of every available option, e.g. Any type that fulfils this LSP condition. Or perhaps no methods that "clash" with those from the other half of the intersection.

@DiscordLiz

Which makes it intersections all the way down and no different than being Any, cause we have to assume LSP wasn't violated.

Perhaps we should restrict our examples to those that fulfil this condition? Even in the case we're untyped, it seems we have to assume that this method clash hasn't occurred.

@NeilGirdhar @mikeshardmind Specifics aside, I think whatever solution we come up with, it shouldn't require cast or type:ignore to pass through the type checker successfully. I mentioned the below relating to an earlier debate, but I think the same philosophy applies here - given the choice between:

Being given a type that's too broad for the attributes available - there won't be any errors raised when we do some things that are invalid.

Being given a type that's too narrow for the attributes available - there will be errors raised when we do things that are valid

The second one just seems much worse to me than the first, but I guess that is personal preference somewhat.

I guess this is maybe another way of phrasing @NeilGirdhar 's previous point about false positives vs false negatives.

The False positives case is preferable because we still give the user a way to handle it without a type ignore and without additional runtime costs. The False negative case cannot be remedied except by removing Any, and the whole reason we can't ban Any from intersections is that type checkers have stated this could happen in a way users don't have a way to remedy or even a clear way to inform them.

Consider this:

from untyped_library import SomeParserAugmentor  # user may not realize this is untyped at first
from typed_library import SomeParser  # generic over T, provides .parse(self, arg: str) -> T

def my_func(input_: str, parser: SomeParser[int] & SomeParserAugmenter ) -> None:
    """ Typed permissively so that other people can use the function """

   # This errors, we restrict the interface of Any when it clashes with what is typed
   # This also means the person writing the intersection is the one who gets the error
    x = parser.parse(input_, fix_mojibake=True) 


class _Parser(SomeParser, SomeParserAugmentor):
    """ Internal fallback where we create only the baseline functionality """
    pass
...


# this is fine, subclassing from Any means the attributes needed are assumed there. 
# This is the case currently with subclassing Any, not even just with this proposal
my_func(some_user_content, _Parser(int))

In this case, the user actually has the tools to fix this without a type ignore in multiple ways.

Because they are informed that something isn't meeting their expectations, they may contribute upstream types
They can also add a protocol stating that the attribute exists:

from untyped_library import SomeParserAugmentor  # user may not realize this is untyped at first
from typed_library import SomeParser  # generic over T, provides .parse(self, arg: str) -> T
from typing import Protocol

class FixesMojiBake[T](Protocol[T]):
    def parse(self, input_: str, fix_mojibake: bool=False) -> T: 
        ...
    

def my_func(input_: str, parser: SomeParser[int] & SomeParserAugmenter & FixesMojiBake) -> None:
    x = parser.parse(input_, fix_mojibake=True)  # This no longer errors

class _Parser(SomeParser, SomeParserAugmentor):
    """ Internal fallback where we create only the baseline functionality """
    pass

# this is still fine
my_func(some_user_content, _Parser(int))

With this, they've now explicitly stated they expect this compatible method to exist. This gives users the information they need to type things without compromising on safety for the portions of code that are already typed, and while being easy to accommodate situations outside of their control with untyped libraries. Note that this is safer thana type ignore, it would rely on the nature of Any for the thing Assigned to it, but it requires explicitly escaping out for things that could otherwise be accidentally hidden in terms of misuse by restricting the interface on usage to what is explicitly stated. This also automatically continues working should the untyped library become typed in the future without something lurking around for compatibility that could be silencing errors.

I think if we pair this approach with a note that type checkers should inform users when Any is present in an intersection that is causing an error and point them to a reference of how to handle this, we can make this easy to learn and ergonomic as well.

A few observations after having read Eric's comment and trying to reason various things out in my head.

It is very easy to confuse yourself by mixing up the meaning of & with that of | (or vice versa). This happened to myself a few times. I think it's due to the fact that after x: T1 | T2, we consider valid properties of x those that are valid for T1 and for T2, while after x: T1 & T2, intuitively, the set of valid properties is the union of the valid properties of T1 and T2. (It's a similar confusion as often happens around contravariance.)
If is easy to reason things out for proper types, including edge cases like object and Never.(top and bottom, respectively), by thinking of types as sets of objects. The intersection and union operators work sensibly in this case, and all the rules you can derive this way (e.g. distributivity, subtyping, and attribute type evaluation, as shown in Eric's comment). However, Any is *not* just another edge case. It is the defining building block of gradual typing, and behaves like top or bottom depending on which side of the subtyping operator (<:`) it appears.
Using assert isinstance(x, T) in examples is not all that helpful, because at runtime T is constrained to concrete types. We need to go back to presenting examples in terms of whether x = y is allowable or not according to the type system given specific types for x and y (and possibly what the type of x is afterwards, in situations where narrowing happens).
Generics present a whole nother can of worms. I haven't spent enough time thinking about these.
There are probably also traps due to the difference between structural subtyping (Protocol) and nominal subtyping.
And perhaps others due to the pragmatic treatment of TypedDict.
Folks, please proofread your messages.

I'l write more later.

@NeilGirdhar

[me]

Given that a: Any & T means thata has all the attributes of T and they have the types specified by T, in this example, x has __add__(self, arg: int) -> int so the type of x should be revealed as int. That matches my intuition.

@NeilGirdhar

Just something for you to consider. Let's just look at an example that doesn't involve Any:

Great, it should be possible to reason this out by considering types as sets of objects.

class C:
  def f(self) -> None: pass
class D:
  def g(self) -> None: pass
class E(C, D): pass

def f(x: C):
  if isinstance(x, D):
    reveal_type(x)  # C & D
    x.f()  # should be okay since x inherits from C
    x.g()  # should be okay since x inherits from D

f(E())

This is why the interface of C&D is the union of the interfaces of C and D.

But why introduce a new term, "interface"? Python's type system never uses this word -- we talk about types. I feel it's misleading to introduce this term -- probably because of my point (2). We should only consider questions of the form "if x has type T1 and y has type T2, is x = y acceptable?" And in addition, questions of the form "given x has type T and OP(a) is some operation (such as getting an attribute of a specific name, or calling it with specific arguments), is OP(x) valid?"

Now, consider:

* what is the interface of `Any`?

I feel this is not helping clear reasoning, given that in x = y, the rules when x has type Any are totally different from the rules when y has type Any. (And different again for OP().)

* what is the interface of `Any & C`?

Should be rephrase in terms of what happens to x = y or perhaps OP(x), separately for the case where x: Any & C and y: Any & C.

* So if `x` has type `Any & C`, what is the type of `x.f()`, `x.g()`, and `x.h()`?

That's a valid question. I think it should be answered by considering separately what the type of an expression is when x: Any and when x: C, and then doing an & over the resulting types.

Interestingly, since the answer might well be "an error occurs" for one of the branches of the & operator, maybe we should consider what the result should be of Error & T. Thinking about some examples, it seems that the Error type should disappear and the answer should be T -- IOW, it is not an error unless all branches of the & operator have type Error. This is nicely the dual of the error behavior for |, where the rule is that it's an error if it is an error for any branch of the | operator.

But now we get to think about when an error should be reported to the user -- surely not as soon as it occurs, in the case of &, or else x.attr where x: C & D would report an error if C supports attr but D doesn't -- and the intersection is meant to not report an error in that case.

Also, it seems that whatever rules we come up with for Error should also apply to Any? Except they don't -- It would seem that Error | T reduces to Error, but Any | T is irreducible. (I now understand that word. :-) It seems that we perhaps might regret that choice, but even those who regret it seem to agree that it's too late to change -- and I'm with Eric in that I think we should make sure that & is the dual of | in this respect, like it is everywhere else.

It's really worth reading Eric's elucidating comment.

For reference, there are two proposals being considered here. I'll call them 4 and 5 (relating to names from #1). Proposal 4 says: x.f(), x.g(), x.h() are all Any. Proposal 5 says: x.g(), x.h() are both Any, but x.f() is None.

I don't know what the type of x is here, but it can't be from the code snippet you gave at the top of your comment (which has neither Any nor h()). Could you provide the whole context?

@NeilGirdhar

Specifics aside, I think whatever solution we come up with, it shouldn't require cast or type:ignore to pass through the type checker successfully....I guess this is maybe another way of phrasing @NeilGirdhar 's previous point about false positives vs false negatives.

Exactly what I was getting at :)

I think if we pair this approach with a note that type checkers should inform users when Any is present in an intersection that is causing an error and point them to a reference of how to handle this, we can make this easy to learn and ergonomic as well.

This is a great idea that actually works with both proposals. We could leave the "correct" semantics of proposal 4 coupled, and encourage type checkers to implement an optional warning when someone accesses an attribute on something with type Any & T. This warning could then be resolved by:

using type-ignore,
adding a cast, or
disabling the warning.

3. Using assert isinstance(x, T) in examples is not all that helpful, because at runtime T is constrained to concrete types.
4. Generics present a whole nother can of worms.

I agree that we should explore other ways in which Any & T pops up naturally. However, I think the is-instance example works when the type of x before the instance check is a type variable bound to Any.

3. We need to go back to presenting examples in terms of whether x = y is allowable or not according to the type system...

But why introduce a new term, "interface"? Python's type system never uses this word -- we talk about types. I feel it's misleading to introduce this term -- probably because of my point (2). We should only consider questions of the form "if x has type T1 and y has type T2, is x = y acceptable?"

Right, I see where you're coming from. If that's the language you like, then I think you'd find Jelle's comment the most clarifying. I was using language closer to Eric's comment.

As per Jelle's comment, two things constitute the "type" (as returned by reveal_type) of x:

For which y is x = y is allowable? And,
What operations can you do with x?

These correspond to what Eric called subtyping and attribute type evaluation. And the latter is what you're calling OP(x).

That's a valid question. I think it should be answered by considering separately what the type of an expression is when x: Any and when x: C, and then doing an & over the resulting types.

Yes, you're right of course. I made a mistake in my evaluation.

Could you provide the whole context?

I think I just made an error when evaluating the values 😄

Regarding the ideas surrounding Error, I need to think about it more.

To delve into the more practical, I'll put this thought into an example:

class T:
    x : int

inst: T & Any = T()

z = inst.x # So this is now type int & Any
z += 1 # Definately valid
z = z + 'test' # Now here we get the controversial bit.

My hunch is that (by some means) the final line should be invalid, but here's the problem. If we say it's valid, how is this any different from z just being Any? Similarly if we say it's invalid, how is this any different from just saying it's type int? For int & Any to be a type in it's own right, surely it must behave differently to both Any and int?

Yes, that's the comment I had in mind.

z = inst.x # So this is now type int & Any

Yes, I think so.

z = z + 'test' # Now here we get the controversial bit.
Right, that's technically okay since you could do

class T:
    x : int
class C(int):
    def __add__(self, other: Any) -> C: ...
class U(T):
    x: C
inst: T & Any = U()
z = inst.x # So this is now type int & Any
z += 1 # Definately valid
z = z + 'test' # No error.

how is this any different from z just being Any?

If I understand correctly, you can't do anything with int & Any. For example, z + 'test' must return an int or subtype thereof.

For int & Any to be a type in it's own right, surely it must behave differently to both Any and int?

I think it follows the rules that Eric gave recursively.

If I understand correctly, you can't do anything with int & Any. For example, z + 'test' must return an int or subtype thereof.

If it must return an int, how is it any different from an int?

I think it follows the rules that Eric gave recursively.

I've had a follow up thought - Any | T currently I believe behaves exactly the same as Any, so maybe if Any & T behaves exactly the same as Any in practice that's not the end of the world either.

EDIT: Tested and Any | T does not behave the same - attributes must be present on all objects in the union.

(A & B).x:
Invalid if x is missing from both A and B
A.x if x is present on A but not B
B.x if x is present on B but not A
(A.x & B.x) if x is present on A and B

Eric's rules here are very compelling. For cases outside of Any, I think these aren't even really controversial. Replacing A with Any, is not much of a jump, and that leads us to Any & B.x. On the flipside though I do see how this is frustrating, because it's kind of catering for a case that rarely comes up.

I'm also drawn back to my previous post: #31 (comment)

In the case where T has another type for foo, this becomes a problem. But it has been noted this would create an LSP violation. If we assume that A and B must have the same type for property x these rules can become a little bit different.

(A & B).x:
Invalid if x is missing from both A and B
A.x/B.x if x is present on A or on B (inclusive or)
Invalid if x is present on A and on B and the types are different.

Are all the cases where option 5 fails the any-substitution principle LSP violations?

Ok slightly radical idea: Intersections between two classes where the types are different on two of their attributes & methods are banned on access (as in option 6, but expanded to methods). I appreciate this might go further than LSP but it has some advantages:

mixing two classes together always produces the intersection (may not be a requirement, but it's handy!)
(Any & T).x must condense to T.x because for whatever Any is for it to be valid in an intersection it must subscribe to this rule. If x is not present on T, the type must be Any.
Callable intersections go away (no longer valid)
By banning on access we allow classes to be merged that have attributes that clash, so long as they aren't used
It works with the any-substitution principle
It has a symmetry with union, based on the following set of rules:

(A & B).x:
Invalid if x is missing from both A and B
A.x/B.x if x is present on A or on B (inclusive or)
Invalid if x is present on A and on B and the types are different.

The only real disadvantage I see is we lose the utility of combining methods of the same name into being overloads, but in comparison to all of the advantages this feels like a small loss. Thoughts?

PS: I may have mentioned this before, but if so I wasn't aware of all it's advantages.

EDIT: I also vaguely remember when we discussed something similar before, it was concluded that would take too much runtime cost to evaluate both classes and determine if they're compatible. Now it's on access however, that means we just have to compare the relevant attribute.

@DiscordLiz

Intersections between two classes where the types are different on two of their attributes & methods are banned on access

That's an interesting idea. That restriction makes many of the problems fall away.

Question to others on the thread: Does that restriction eliminate any important use cases that you've identified for intersections? It would preclude my BookMovie example above, but that was a toy example intended to demonstrate an inconsistency, not a real-world example that I think is important to support.

@DiscordLiz said "I would not want to use an intersection with non-disjoint types. It generally doesn't make sense to.". This proposal effectively takes advantage of this observation and codifies it into the rules.

The only real disadvantage I see is we lose the utility of combining methods of the same name into being overloads

I see this as more of an advantage than a disadvantage. Combining methods of the same name into overloads is problematic because overloads are ordered and intersections are not. Overloads also have a bunch of other complicated rules associated with them that don't apply to intersections. This proposal eliminates this problem entirely.

One issue that would need to be tightened up is the specification of what it means for two types to be "different". We've talked in the past about "bidirectionally compatible types" (type A is compatible with B and vice versa), but I'm not sure what it means for two types to be "different". Any is bidirectionally compatible with int; are these two types "different"? Is Literal[True, False] "different from" bool? What about two structural types with different names but the same definition? Would it make sense to replace "types are different" with "types are not bidirectionally compatible"?

Another issue to consider is related to method access and binding. If classes A and B both define def method(self) -> Self, how would (A & B).method work? Instance and class methods are bound and partially specialized at access time, so the type of A().method is () -> A and the type of B().method is () -> B. Does that mean the two are "different types" according to the proposed rule above? Or perhaps binding & partial specialization should be done with the intersection type, which would produce () -> A & B in both cases. (Note: Method binding and partial specialization is currently under-specified in the typing spec. It's one of the areas I hope to firm up in the upcoming months.)

@NeilGirdhar

@NeilGirdhar

While I would love to find consensus, it just seems unlikely at this point.

First of all, in the last few comments Mike is saying that " If [proposal 4] were accepted, I'd probably give up on trying to improve typing in python". I think it's fair to say he's not willing to compromise. Compromises I'd like to consider include:

adding an optional warning when T & Any has properties accessed in a way that violates proposal 5—this would essentially mean that users would optionally have proposal 5 at the discretion of type checker authors,
adding an additional operator SafeIntersection[T, U], and
adding additional constructs to declare that certain interfaces can't be broadened.

Some of those compromises are things I've already said I'm fine with (Well, not entirely as stated, but close), but re-raised a question about something where someone else said it would be a problem or other issues with those compromises. See here

I'm very much still wanting to work towards something that can work for everyone, but in unequivocal terms, if option 4 is accepted as-is, I'm personally done with python typing. There are enough issues with it for my real world use that this would kill typing usability for me long term as this started to be used in that fashion, and I've already explained how those would manifest.

The fact that I am opposed to one option, but still working towards finding others that meet multiple groups needs is not an unwillingness to find compromise at all, but an unwillingness to accept a solution which has major flaws for my own use when there is more to explore.

@mark-todd

@mark-todd

Ok slightly radical idea: Intersections between two classes where the types are different on two of their attributes & methods are banned on access (as in option 6, but expanded to methods). I appreciate this might go further than LSP but it has some advantages:

I was viewing option 6 as also banning methods, properties, and functions with diferring types on access as well when I wrote it. The primary reason for allowing overlap at all is the unavoidable overlaps of things inherited from object, but the rule "just working" without special casing those things was good.

@erictraut

One issue that would need to be tightened up is the specification of what it means for two types to be "different". We've talked in the past about "bidirectionally compatible types" (type A is compatible with B and vice versa), but I'm not sure what it means for two types to be "different". Any is bidirectionally compatible with int; are these two types "different"? Is Literal[True, False] "different from" bool? What about two structural types with different names but the same definition? Would it make sense to replace "types are different" with "types are not bidirectionally compatible"?

bidierectionally compatible is almost fine for what I was considering when proposing that (See below, because I think we do need to be careful in wording this), I can come up with more formal definitions as needed if this approach is viable for others here.

Another issue to consider is related to method access and binding. If classes A and B both define def method(self) -> Self, how would (A & B).method work? Instance and class methods are bound and partially specialized at access time, so the type of A().method is () -> A and the type of B().method is () -> B. Does that mean the two are "different types" according to the proposed rule above? Or perhaps binding & partial specialization should be done with the intersection type, which would produce () -> A & B in both cases. (Note: Method binding and partial specialization is currently under-specified in the typing spec. It's one of the areas I hope to firm up in the upcoming months.)

methods should be fine here with how I was considering the option when I proposed it, the type of (A & B).method would bind to (A & B), We can use type[T] / Self here to improve reasoning about this when considering unbound methods prior to binding. This may sneak a small bit of special casing into implementations that don't understand method binding or descriptors though, and is definitely worth making sure we spell out to avoid diverging implementations.

@mark-todd

@mark-todd

Callable intersections go away (no longer valid)

They only go away partially. Please do not ban callables from intersections entirely, we have a stated use case in this repo that still works with this option.

Callable[[int], T] & type[T] where type[T] is callable with a single int plugs a type hole that exists to make subclassing easier, but LSP unsafe with __init__

@NeilGirdhar

@NeilGirdhar

Finally, there have been some absolutely disrespectful comments, and I don't think that it makes sense to work with people that insist on that kind of behavior.

You just accused another collaborator of not being willing to compromise in the same post when they are here still working on a compromise. You are continuing to not read or otherwise ignore other people's contributions to this thread, and even so, I'd rather you keep providing your input here if you have something valuable to say. I don't think competing PEPs are the answer. We can make the PEP stronger by finding what issues people have with it, and that includes any issues you have with it.

@erictraut

@erictraut

Question to others on the thread: Does that restriction eliminate any important use cases that you've identified for intersections? It would preclude my BookMovie example #31 (comment), but that was a toy example intended to demonstrate an inconsistency, not a real-world example that I think is important to support.

Sort of? I said an intersection would only make sense to me when I care about disjoint types, and that's true, but there are cases for extending base functionality beyond what this would support. However, the thing that @mark-todd and @mikeshardmind went back and forth on in discord and here with type and object or Inherit allowing multiple type parameters in an ordered fashion can address this case without intersections needing to be the tool for this, so I think it's okay to directly state this in a "why this behavior" wasn't included section for the pep since we have a potential path for that behavior in a different mechanism that can be worked on in the future without intersections blocking it.

@erictraut

@erictraut

I see this as more of an advantage than a disadvantage. Combining methods of the same name into overloads is problematic because overloads are ordered and intersections are not. Overloads also have a bunch of other complicated rules associated with them that don't apply to intersections. This proposal eliminates this problem entirely.

That's a good point - I hadn't considered it, but you're right it saves us from that issue.

One issue that would need to be tightened up is the specification of what it means for two types to be "different". We've talked in the past about "bidirectionally compatible types" (type A is compatible with B and vice versa), but I'm not sure what it means for two types to be "different". Any is bidirectionally compatible with int; are these two types "different"? Is Literal[True, False] "different from" bool? What about two structural types with different names but the same definition? Would it make sense to replace "types are different" with "types are not bidirectionally compatible"?

One property of the above description is I was trying to make A.x and B.x effectively interchangeable. If we consider Literal[True, False] to be identical to bool, it might matter which from A and B gets chosen.

Another issue to consider is related to method access and binding. If classes A and B both define def method(self) -> Self, how would (A & B).method work? Instance and class methods are bound and partially specialized at access time, so the type of A().method is () -> A and the type of B().method is () -> B. Does that mean the two are "different types" according to the proposed rule above? Or perhaps binding & partial specialization should be done with the intersection type, which would produce () -> A & B in both cases. (Note: Method binding and partial specialization is currently under-specified in the typing spec. It's one of the areas I hope to firm up in the upcoming months.)

Interesting edge case! I think these would be compatible, returning the new type (A & B) as Self, like you describe. If we found down the road though that this has to be banned I don't think that's the end of the world.

@mikeshardmind

I was viewing option 6 as also banning methods, properties, and functions with deferring types on access as well when I wrote it. The primary reason for allowing overlap at all is the unavoidable overlaps of things inherited from object, but the rule "just working" without special casing those things was good.

Oh ok, great! Apologies, I didn't realise this

@DiscordLiz

They only go away partially. Please do not ban callables from intersections entirely, we have a stated use case in this repo that still works with this option.

Yeah I was thinking about the case of combining two Callables - I see that so long as the other item in the intersection doesn't have a call method that should be find.

Sort of? I said an intersection would only make sense to me when I care about disjoint types, and that's true, but there are cases for extending base functionality beyond what this would support. However, the thing that @mark-todd and @mikeshardmind went back and forth on in discord and here with type and object or Inherit allowing multiple type parameters in an ordered fashion can address this case without intersections needing to be the tool for this, so I think it's okay to directly state this in a "why this behavior" wasn't included section for the pep since we have a potential path for that behavior in a different mechanism that can be worked on in the future without intersections blocking it.

Yeah I think separating that behaviour is a good plan - there's certainly a use case for it but I think it can go into a different PEP.

The Issue got too long again. Please continue discussing in #37.

Please open new Issues if you start discussing on new sub aspects.

Navigation Menu

Repository navigation

Interpretation of type `Any & T` #31

Module a_untyped

Module b_typed

`main.py`

Module 1

Module 2

101 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Interpretation of type Any & T #31

Description

Activity

CarliJoy commented on Dec 15, 2023

Module a_untyped

Module b_typed

main.py

CarliJoy commented on Dec 15, 2023

Module 1

Module 2

mikeshardmind commented on Dec 15, 2023

mark-todd commented on Dec 15, 2023

gvanrossum commented on Dec 15, 2023

DiscordLiz commented on Dec 15, 2023

NeilGirdhar commented on Dec 15, 2023

NeilGirdhar commented on Dec 15, 2023

mikeshardmind commented on Dec 15, 2023

NeilGirdhar commented on Dec 16, 2023

mark-todd commented on Dec 16, 2023

mikeshardmind commented on Dec 16, 2023

gvanrossum commented on Dec 16, 2023

gvanrossum commented on Dec 16, 2023

NeilGirdhar commented on Dec 16, 2023

101 remaining items

mark-todd commented on Jan 10, 2024

NeilGirdhar commented on Jan 10, 2024

mark-todd commented on Jan 10, 2024

mark-todd commented on Jan 10, 2024

mark-todd commented on Jan 10, 2024

erictraut commented on Jan 10, 2024

mikeshardmind commented on Jan 10, 2024

mikeshardmind commented on Jan 10, 2024

DiscordLiz commented on Jan 10, 2024

DiscordLiz commented on Jan 10, 2024

DiscordLiz commented on Jan 10, 2024

mark-todd commented on Jan 10, 2024

CarliJoy commented on Jan 11, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Interpretation of type `Any & T` #31

`main.py`