Proposal: Convert the typing-module PEPs into a series of HOWTOs

@srittau

Problem description

The typing module docs have a problem.

typing.rst is littered with references to the many typing-related PEPs that have been accepted by the Steering Council. Often these references to PEPs are used in lieu of detailed examples or explanation as to how a feature is supposed to work.

However, PEPs are not supposed to be living pieces of documentation. As such, proposals to update old typing PEPs to reflect changes in implementation details have been rejected. But when these PEPs are the closest thing we have to a full Python-typing specification, this means that our specifications are slowly growing out of date.

This unfortunate situation has also been recognised by the Steering Council in a recent email to the typing-sig mailing list.

Proposed solution

I propose that we slowly start to convert various typing PEPs into a series of Python-typing HOWTOs. In this way, we'd follow the example of other modules such as enum or socket. These modules have terse API references in socket.rst and enum.rst, but these API references are supplemented with expansive HOWTOs that serve as a guide for users who are new to the respective modules. Given the number of typing PEPs, the scale of the task for typing will be larger; but I still believe that this is a good example to follow.

Specifically, I think we should have something like the following structure:

"Introduction to type hints in Python". This HOWTO would combine material from the following sources (updating examples where relevant to use modern syntax such as PEP-604 union types).
1. PEPs 483 & 484 (introducing type hints to Python)
2. PEP 526 (syntax for variable annotations, ClassVar)
3. PEP 613 (explicit type aliases)
4. PEP 673 (typing.Self)
5. The sections in typing.rst on Any, Callable and type aliases.
6. Some of the material in typing.rst on TypeVar and generics (but only the most basic material).
"Generics HOWTO". While the intro HOWTO would introduce the basic concept of TypeVars, this HOWTO would tackle the nitty-gritty:
1. Using typing.Generic to make generic classes.
2. The differences between bound TypeVars, constrained TypeVars, and TypeVars which are neither.
3. Covariance versus contravariance versus invariance.
4. Maybe a brief mention of PEP 585 (generic stdlib collections).
5. ParamSpec and Concatenate (material from PEP 612).
6. Variadic generics (material from PEP 646).
"Structural subtyping HOWTO", using material from PEP 544 and the section in typing.rst on structural subtyping and typing.Protocol.
"TypedDict HOWTO", using material from PEPs 589 & 655, as well as the section on TypedDict in typing.rst.
"Literal types HOWTO", describing how to use PEPs 586 (typing.Literal) & 675 (typing.LiteralString).

Due to the brevity of their PEPs, and the fact that they don't really "fit in" neatly with any of the other proposed HOWTOs, I propose that the following typing features have their full specifications integrated into the typing.rst API reference, rather than having dedicated HOWTOs:

"Final/@final (PEP 581)
Annotated (PEP 593)
TypeGuard (PEP 647)

Why not have this information at typing.readthedocs.io?

There's an ongoing effort, spearheaded by @srittau among others, to include a body of typing documentation at https://typing.readthedocs.io/en/latest/. I am fully in support of this effort, and see the documentation there as complementary to the documentation that currently lives in the typing PEPs. However, I do not believe that the documentation currently in the typing PEPs should belong on that site, for several reasons:

https://typing.readthedocs.io/en/latest/ is a third-party site, rather than being an "official" corpus of Python-language documentation. This is a good thing -- it allows contributors to that site to iterate and improve on the documentation there at a more rapid pace. However, the information in the typing PEPs are officially recognised changes to the language and the language's typing framework. As such, -- in my opinion -- it should live in the central docs.python.org documentation along with the documentation for the rest of the language.
https://typing.readthedocs.io/en/latest/ is currently not particularly discoverable via search engines. For example, it does not appear in the first page of Google results for "Python typing".

Questions to be discussed

What do we think of the general idea of moving material out of PEPs, providing specifications through a series of HOWTOs, and slimming down typing.rst into a terser API reference?
What do we think of my specific proposal for how to structure this series of HOWTOs?

Copied to:

typing module maintainers: @gvanrossum, @Fidget-Spinner, @JelleZijlstra
@srittau, who does a lot of valuable work on typing documentation over at the https://github.com/python/typing repo.
@hauntsaninja, who recently expressed interest in helping with typing docs.
Several miscellaneous typing-PEP authors who may be interested in this proposal: @ambv, @JukkaL, @ilevkivskyi, @mrkmndz, @pradeep90, @Gobot1234, @mrahtz, @davidfstr, @erictraut
@encukou, who recently sent the email to typing-sig on this topic on behalf of the Steering Council, and is also a member of the docs working group.

Linked PRs

gh-91533: Move much of typing.rst to a new 'Guide to typing features' document #105827

Don’t worry about SEO. If what we place there is good, readers will come.

@AlexWaygood

@AlexWaygood I think once we have a fairly stable and good set of documentation on typing.readhedocs.io (and we are a long way off), we should look into moving some or all of it over to docs.python.org, but until then we are more free to experiment. I'd also like to merge your proposal here, the proposal in python/typing#845, and the "proposal" in https://github.com/python/typing/blob/master/docs/index.rst and https://github.com/python/typing/tree/master/docs/source into a coherent whole.

@JelleZijlstra

I'd also like to merge your proposal here, the proposal in python/typing#845, and the "proposal" in https://github.com/python/typing/blob/master/docs/index.rst and https://github.com/python/typing/tree/master/docs/source into a coherent whole.

Sure, but I also think we're in rough agreement about the place we want to end up at now, so it's probably time to stop talking about writing and start writing 🙂 we can debate the finer points of "what goes in which document" at a later date.

With that in mind:

I'd like to start work on some "Introduction to python typing" user guides today, with a view to sending a PR to the https://github.com/python/typing repo
@JelleZijlstra, would you like to start work on some formal specifications?
@davidfstr, would you perhaps be interested in possibly helping out with some TypedDict user guides? The first draft could simply lift large chunks from typing.rst, the mypy docs, and/or the two TypedDict PEPs (giving credit where appropriate)
Any volunteers for any other parts of this?

I'm thinking we can incrementally delete chunks from typing.rst and add links pointing to https://typing.readthedocs.io/en/latest/ as different pieces get merged into the https://github.com/python/typing repo. We don't need the docs at https://typing.readthedocs.io/en/latest/ to be "finished" before we start linking to them from typing.rst; we can just say "here is a nice tutorial on how to use Literal", etc.

@encukou

@encukou, would the docs working group/community be interested in helping review these pieces of documentation as and when they're ready?

@davidfstr

@davidfstr, would you perhaps be interested in possibly helping out with some TypedDict user guides?

I'd be happy to.

For my own reference: This initiative appears relevant to Python 3.11 documentation and therefore is more urgent than Python 3.12 items I'm already scheduled for, such as TypeForm/AnnotationType. So I'll probably pick it up next (in my queue of Python-related tasks).

@encukou

@encukou, would the docs working group/community be interested in helping review these pieces of documentation as and when they're ready?

Most likely. Please open an issue at https://github.com/python/docs-community/issues when you're ready for the review!

Please please please please please!! I am SO in need of this! (I can't possibly be the only newcomer who feels the same way)

@AlexWaygood

@AlexWaygood I would love to add this to use cases:

What is the Use Case for Using *args as a Type Variable Tuple?

I would also love if another link to the typeshed readthedocs could be added in an appropriate place:

I propose that we slowly start to convert various typing PEPs into a series of Python-typing HOWTOs

If for nothing else, I think you would help lift the burden on yourselves of having to handle multiple incorrect/false/erroneous issues popping up on the mypy issue tracker (or in typing).
The longer the issues/discussions lists get, the harder it becomes for new users to search through them to find a related issue/discussion, which only serves to add fuel to the fire.

From a 30,000-foot view, this certainly feels like it would be a net win for both sides.

I would also love if another link to the typeshed readthedocs could be added in an appropriate place:

After re-reading Why not have this information at typing.readthedocs.io?, this last one might not be relevant...

https://typing.readthedocs.io/en/latest/ is currently not particularly discoverable via search engines. For example, it does not appear in the first page of Google results for "Python typing".

Don’t worry about SEO. If what we place there is good, readers will come.

Actually, us new users are saying we don't know where to look:

If we don't know where to go, we can't come...

I guess I'm not 100% sold on the distinction between specifications and end-user documentation.

Providing a new user perspective, what I'm lacking is HOW-TO design principles. These arguably come with experience, but this knowledge is beginning to be needed more and more by new users:

A common task to throw at newcomers is to add documentation: It's an excellent way for them study the code base and familiarize themselves with how it works. But nowadays there's a catch: most documentation has been written already. The greatest thing missing in code bases is types. We've now begun to ask newcomers to understand a relatively complex thing, especially under the umbrella of a traditionally dynamically typed language.

If a "business need" is what's required, I feel fairly safe in claiming you could probably grep the issue tracker and discussions for "that's working as intended" compared with actual bugs and find the percentage of the former greatly outweighs the latter.

Regardless of all of that, I think you'll find new users do want to learn the right way of doing things. I'm certainly in that camp. I want to know how to type hint properly and I want to be able to point others to a proper reference so I don't spread misinformation.

However a HOW-TO can be achieved, I am extremely in favor of it.

pythongh-96917: link to typing.readthedocs.io from typing.rst

pythongh-96917: link to typing.readthedocs.io from typing.rst (python…

gh-96917: link to typing.readthedocs.io from typing.rst (#96921)

pythongh-96917: link to typing.readthedocs.io from typing.rst (python…

[3.10] pythongh-96917: link to typing.readthedocs.io from typing.rst (p…

gh-96917: link to typing.readthedocs.io from typing.rst (GH-96921)

[3.10] gh-96917: link to typing.readthedocs.io from typing.rst (GH-96921

gh-96917: link to typing.readthedocs.io from typing.rst (GH-96921)

pythongh-91533: Move much of `typing.rst` to a new 'Guide to typing f…

Navigation Menu

Repository navigation

Proposal: Convert the `typing`-module PEPs into a series of HOWTOs #91533

Problem description

Proposed solution

Why not have this information at typing.readthedocs.io?

Questions to be discussed

Copied to:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Proposal: Convert the typing-module PEPs into a series of HOWTOs #91533

Description

Problem description

Proposed solution

Why not have this information at typing.readthedocs.io?

Questions to be discussed

Copied to:

Linked PRs

Activity

gvanrossum commented on Apr 14, 2022

srittau commented on Apr 15, 2022

AlexWaygood commented on Apr 15, 2022

AlexWaygood commented on Apr 15, 2022

davidfstr commented on Apr 15, 2022

encukou commented on Apr 19, 2022

adam-grant-hendry commented on Sep 17, 2022

adam-grant-hendry commented on Sep 18, 2022

adam-grant-hendry commented on Sep 18, 2022

adam-grant-hendry commented on Sep 18, 2022

adam-grant-hendry commented on Sep 18, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Proposal: Convert the `typing`-module PEPs into a series of HOWTOs #91533