Typing changes in 8.1.4 cause failure of mypy check

Click's simple example https://github.com/pallets/click#a-simple-example doesn't pass mypy 1.4.1 since 8.1.4. With click 8.1.3 the example did pass. This affects not only the simple example but also existing projects that use click. I ran mypy on one of my projects and everything passed, ran it 30 minutes later and saw failures out of the blue. Click 8.1.4 was published to PyPI between my checks.

To reproduce:

Save the simple example as click.py
pip install mypy==1.4.1 click==8.1.3
mypy click.py

This is the output:

click.py:3: error: Argument 1 has incompatible type "Callable[[Any, Any], Any]"; expected <nothing>  [arg-type]
click.py:12: error: <nothing> not callable  [misc]

Environment:

Python version: 3.11
Click version: 8.1.4

To anyone seeing or voting on this issue: Happy to review a PR.

We ran into this as well. Unfortunately I won't be able to work on a fix for several hours at least, but I'm happy to try things out and submit a PR if I can. If someone else gets to it first, I'd also be happy to help review and check the fix against our codebase if that would be useful.

I think the issue stems from the definition of the _Decorator type alias in the decorators. I'm not quite sure how the types should be shaped, but I tried a change which swaps an underlying TypeVar for a TypeAlias.

I put together a reproducer for the expected <nothing> behavior which tries to ignore most of the details, and a possible fix (though I'm not 100% sure that the types I played with are correct):

import typing as t 

from click import Command


T = t.TypeVar("T")
_AnyCallable: t.TypeAlias = t.Callable[..., t.Any]
_Decorator: t.TypeAlias = t.Callable[[T], T]
FC = t.TypeVar("FC", bound=t.Union[_AnyCallable, Command])


def option() -> _Decorator[FC]:
    def decorator(f: FC) -> FC:
        return f 

    return decorator


FC2: t.TypeAlias = t.Union[_AnyCallable, Command]
FC2_Decorator: t.TypeAlias = t.Callable[[FC2], FC2]


def option2() -> FC2_Decorator:
    def decorator(f: FC) -> FC:
        return f 

    return decorator


def foo() -> None:
    pass


reveal_type(option)
reveal_type(option2)
foo = option()(foo)
foo2 = option2()(foo)

Mark Click version 8.1.4 as incompatible (mypy typing errors)

Temporarily pin to click < 8.1.4

Update on "Temporarily pin to click < 8.1.4"

Temporarily pin to click < 8.1.4

A minor note: I'm now fairly certain that my "solution" above is incorrect.

If FC2 is defined as an alias for _AnyCallable | Command, then it won't allow subtypes of _AnyCallable, which is not acceptable. (It would mean that a command whose callback is defined with a return type other than Any gets rejected.)

It might be okay to use Callable (with no parameters) in lieu of _AnyCallable, but I haven't yet learned why _AnyCallable is being used rather than Callable.

I'm guessing the right way to type these decorators is to use the facilities described here, which are intended to deal with exactly this problem of decorating functions that may have arbitrary signatures:

https://mypy.readthedocs.io/en/stable/generics.html#declaring-decorators

That was what I used originally. It was apparently lacking, because we continue to get reports from users saying that their type checker doesn't like something, then we add more complexity to the type, then something else isn't correct, etc. Python's and mypy's implementation of typing can be a huge mess sometimes.

Whoever is going to change this, be sure to follow git blame (or GitHub's view of it) back in time until you get to the original addition of types. Make sure to take into account why the various commits were made along the way.

Taking a quick look at this.

Would you consider changing the signature of option (and maybe others) to:

def option(
    *param_decls: str, cls: t.Optional[t.Type[Option]] = None, **attrs: t.Any
) -> t.Callable[[FC], FC]:

? This syntax is equivalent to what we have today.

The current implementation with a generic type alias might be a little too much for Mypy at this time (I'm getting note: Revealed type is "<nothing>"). This by itself won't solve this issue but it might be a necessary step.

After that, I think this overload is the issue:

# variant: name omitted, cls _must_ be a keyword argument, @command(cmd=CommandCls, ...)
# The correct way to spell this overload is to use keyword-only argument syntax:
# def command(*, cls: t.Type[CmdType], **attrs: t.Any) -> ...
# However, mypy thinks this doesn't fit the overloaded function. Pyright does
# accept that spelling, and the following work-around makes pyright issue a
# warning that CmdType could be left unsolved, but mypy sees it as fine. *shrug*
@t.overload
def command(
    name: None = None,
    cls: t.Type[CmdType] = ...,
    **attrs: t.Any,
) -> t.Callable[[_AnyCallable], CmdType]:
    ...

If I comment it out, the example type-checks. So we need to figure out how to fix it. Given the comment, it might not be super easy.

Like the command comment mentions, the proper way to type that overload is:

@t.overload
def command(
    name: None = None,
    *,
    cls: t.Type[CmdType],
    **attrs: t.Any,
) -> t.Callable[[_AnyCallable], CmdType]:
    ...

The comment also mentions this doesn't actually pass Mypy. Now I haven't really used custom command in click, so I googled and found this example from the official docs: https://click.palletsprojects.com/en/8.1.x/advanced/#command-aliases. It looks like this example might be exercising the problematic overload?

If I try pasting the first part of the example into a file, Mypy seems to work fine with it:

import click


class AliasedGroup(click.Group):
    def get_command(self, ctx, cmd_name):
        rv = click.Group.get_command(self, ctx, cmd_name)
        if rv is not None:
            return rv
        matches = [x for x in self.list_commands(ctx) if x.startswith(cmd_name)]
        if not matches:
            return None
        elif len(matches) == 1:
            return click.Group.get_command(self, ctx, matches[0])
        ctx.fail(f"Too many matches: {', '.join(sorted(matches))}")

    def resolve_command(self, ctx, args):
        # always return the full command name
        _, cmd, args = super().resolve_command(ctx, args)
        return cmd.name if cmd is not None else None, cmd, args


@click.command(cls=AliasedGroup)
def cli():
    pass


reveal_type(cli)  # note: Revealed type is "a02.AliasedGroup"

So maybe Mypy fixed this in the meantime and we can use the proper overload variant?

Downgrade click due to typing issue with Mypy

I'm fine with changing signatures. I'm also fine with writing the "right" type for people using Click, even if mypy doesn't like it when checking Click itself; we can just add ignores and comments.

Another wrinkle in this, as shown by that comment, is that people want both mypy and pyright to work, and they have different behaviors regarding decorators.

@hynek

I can put together a PR over the weekend. How about I set up some typing tests too (by stealing them from @hynek in attrs)?

limit click versions to please mypy

REF: use torchscript models from huggingface hub (#144)

I'm still experiencing this issue, the examples in https://click.palletsprojects.com/en/8.1.x/quickstart/#nesting-commands fail to typecheck under MyPy 1.4.1. I imagine whatever type adjustments were done to click.command for 8.1.5 also needs to be done to click.group.

import click

@click.group()
def cli():
    pass

@cli.command()
def initdb():
    click.echo('Initialized the database')

@cli.command()
def dropdb():
    click.echo('Dropped the database')

❯ python3 -m pip list
Package           Version
----------------- -------
click             8.1.5
mypy              1.4.1
mypy-extensions   1.0.0
pip               23.1.2
setuptools        68.0.0
typing_extensions 4.7.1

❯ mypy --pretty example.py
example.py:7: error: <nothing> has no attribute "command"  [attr-defined]
    @cli.command()
     ^~~~~~~~~~~
example.py:11: error: <nothing> has no attribute "command"  [attr-defined]
    @cli.command()
     ^~~~~~~~~~~
Found 2 errors in 1 file (checked 1 source file)

Can you try #2565?

Can confirm #2565 resolves this issue for me. Thanks! Now to wait for 8.1.6...

Locking to avoid "me too" comments.

test: silence mypy failured due to bad click

Click 8.1.6 is available on PyPI.

For anyone that runs mypy with pre-commit, make sure to add click>=8.1.6 to additional_dependencies

Remove unnecessary type-ignores

Navigation Menu

Repository navigation

Typing changes in 8.1.4 cause failure of mypy check #2558

52 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Typing changes in 8.1.4 cause failure of mypy check #2558

Description

Activity

davidism commented on Jul 6, 2023

sirosen commented on Jul 6, 2023

sirosen commented on Jul 6, 2023

rra commented on Jul 6, 2023

davidism commented on Jul 6, 2023

Tinche commented on Jul 6, 2023

Tinche commented on Jul 7, 2023

davidism commented on Jul 7, 2023

Tinche commented on Jul 7, 2023

52 remaining items

blue42u commented on Jul 16, 2023

layday commented on Jul 16, 2023

blue42u commented on Jul 16, 2023

pierrejeambrun commented on Jul 16, 2023

davidism commented on Jul 16, 2023

davidism commented on Jul 18, 2023

aarnphm commented on Jul 29, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions