How to annotate code that uses None as an undefined value

I've seen a fair amount of code that uses None as un undefined value for an attribute. It's typically something like this:

class Response:
    def __init__(self):
        self.status = None
    def really_initialize(self, status):
        self.status = status  # status should be an int

In this case, the object is not fully initialized until somebody calls really_initialize. There may be a factory function that only returns fully initialized Response objects. Sometimes initialization may have multiple steps, but the common theme is that __init__ doesn't do full initialization and it uses None as the placeholder value.

Now the most natural type to give to status is Optional[int]. However, this would imply that all code that accesses status must add a guard agains None, or the type checker may complain. However, the idiomatic way to do this in dynamically typed Python does not involve checking for None, since it's not an expected value. Migrating code like this would be fairly painful, involving asserts or casts. Documentation for Response (that hasn't been updated to use PEP 484) probably would not mention that status can be None (as it would be considered an implementation detail), so the programmer may be confused about why he's expected to check for a None value.

Most recently I stumbled into this idiom in requests, which is considered by many to be a well-designed package, so this doesn't only apply to 'bad' code.

I have a few questions about code like this:

Should we support this idiom in statically typed code? Should we require Undefined to be used instead of None if one wants to type check code that uses this idiom? In the latter case, we have the issue that there's no supported way to check if a variable has an Undefined value other than try/except, whereas it's easy to check if a value is None.
What about library stubs for modules that use this idiom? Should we use Optional[int] or just plain int for the status attribute in a stub? I'd probably use just int.

Note that mypy doesn't currently properly type check None values (they are treated like null in Java), which sidesteps the issue. However, the PEP mentions that type checking None values should be supported.

After thinking about this for a while, here's a simple proposal:

For statically typed code, people should use Undefined instead of None for an 'uninitialized' attribute. Alternatively, they can just not assign to the attribute (resulting in NameError). They must not check whether the attribute is undefined by using something like is Undefined; code should just rely on Undefined raising an exception if used.
In stubs for a module that uses None for an uninitialized attribute, just use regular types instead of Optional[...] in stubs. There is no hard rule for when to use Optional[...] or not -- the goal should be to support type checking typical code using the module without needing to add extra checks/casts, even if things could plausibly go wrong if somebody forgets to initialize things correctly.

This is still probably not great for some code, such as if we want to print the state of an object when it's not fully initialized. Some (ugly) workarounds:

We could have an initialization flag attribute that records whether an attribute has an Undefined value and use that to check whether it's safe to read an attribute. This is actually a pretty clean approach.
We could avoid assigning to the attribute until it's initialized and use hasattr() to see whether it exists.
Initialize the attribute to Undefined in the class body and use 'attr' in obj.__dict__ to check whether it has been assigned to.

I think we should recommend some workaround, as otherwise people may just figure out a way to use is Undefined (we can't really prevent people from doing that if they want it...).

We used to check None-related problems in cases like these in PyCharm, but we stopped doing that since it used to produce a considerable amount of false positive warnings. Currently we infer types like Optional[...] for attributes initialized with None, but we don't warn about accessing such values outside of if value: ....

I like Jukka's proposal. Of the three workarounds, I prefer leaving it unset and then using hasattr(). The initialization flag seems cumbersome, and using obj.__dict__ has too many caveats (e.g. doesn't work with __slots__).

It's also quite likely that it's more pragmatic not to check for such issues, but I'd like to make that a choice for the type checker implementation or the user running it rather than pick a choice in the PEP or offer a way to control it in the source code.

Here are some more potential ways of dealing with it (I'm not saying that these are good ideas, though...):

Use self.status = cast(int, None) so that the static type of the attribute will be int, but the runtime value happens to be incompatible with the static type.
Use # type: ignore, for example something like this:

class Response:
    status = Undefined(int)
    def __init__(self) -> None:
        self.status = None # type: ignore
    def really_initialize(self, status: int) -> None:
        self.status = status  # status should be an int

We could define a variant of Optional[t] that doesn't use strict None type checking. Let's call it Nullable[t]. Nullable[t] would be compatible with t, unlike Optional[t]. Example:

class Response:
    def __init__(self) -> None:
        self.status = None # type: Nullable[int]
    def really_initialize(self, status: int) -> None:
        self.status = status  # status should be an int

r = Response()
r.status + 1  # Statically okay, since status is nullable

I think Swift has something resembling this (too lazy to check right now). Adding a new type system feature for this seems like feature bloat, though, since there are several other ways of dealing with this already. However, this has the benefit that the implementation of the class doesn't need to be touched (we only need to add a comment, and I don't count adding type annotations as modifications).

We could have a special way for suppressing optional type checking for variables initialized with None:

class C:
    def __init__(self):
        # Non-optional type. By convention, it doesn't result in a type error
        self.foo = None  # type: int

        # Regular optional type
        self.bar = None  # type: Optional[int]

Sounds good. Also for locals, globals and class variables, right? Perhaps for locals the checker could warn if the variable isn't set to a valid value before it's used? (For the others that seems hard to enforce.)

@gvanrossum

@gvanrossum Yes, sounds like a nice idea.

Agreed.

Need to add this to the PEP, specifically, that

x = None  # type: <non_optional_type>

is valid, doesn't cause a type error on this line, and doesn't alter the type of x to be Optional[...].

Navigation Menu

Repository navigation

How to annotate code that uses None as an undefined value #61

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

How to annotate code that uses None as an undefined value #61

Description

Activity

JukkaL commented on Mar 22, 2015

vlasovskikh commented on Mar 22, 2015

gvanrossum commented on Mar 23, 2015

JukkaL commented on Mar 25, 2015

vlasovskikh commented on Apr 14, 2015

gvanrossum commented on Apr 14, 2015

vlasovskikh commented on Apr 14, 2015

JukkaL commented on Apr 14, 2015

gvanrossum commented on May 18, 2015

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions