The future of Python-Markdown #1689

This project first started nearly two decades ago, when there were only a few clones of the original Perl implementation of Markdown. It was created as a direct clone, which copied the basic parsing structure of the Perl code. In essence, the code consisted of a series of regular expressions that were run in a specific order. Since then, this library has been expanded and adapted in various ways, most notably with an extension API, which allows third-party extensions to insert their own regular expressions into the series. Since then, the Markdown community has expanded and grown, CommonMark was defined and implemented, and the way that modern Markdown parsers work is very different under the hood. From time to time, we have considered updating our core by abandoning the current structure and using one of these more modern methods. However, we have not done so as that would require us to abandon support for all/most of the existing third-party extensions. Additionally, because of its core structure, it is not practical to adapt this library to become a modern (CommonMark?) parser.

After CommonMark began to gain traction, I assumed that users would abandon this project in favor of CommonMark. However, at least in Python, the various CommonMark libraries seem to have not gained much traction either due to lack of an extension API, or a lack of available extensions where an API exists. For that reason, Python Markdown continues to be the most popular Markdown implementation in Python by a large margin. Therefore, Python users can't use extensions with a modern parser, and when they want to use extensions, they don't get a modern parser.

Most recently, this library has received an explosion of bug reports for obscure inputs, which causes the running time of the parser to grow exponentially. When these issues were raised in the past, they were dismissed as so obscure that no reasonable human would ever create that input. However, with today’s readily available AI tools, anyone can prompt a tool to create input that could easily be used in a denial-of-service attack. These same tools can even find new attack vectors when the existing ones have been patched. Therefore, these reports can no longer be ignored. Most of these bug reports are due to basic usage of Python's regular expressions, and can be tied to the underlying core structure of our code. It is not sustainable to continue to patch each and every one of these piecemeal as we have been doing in recent months. The only way to properly mitigate these sorts of issues would be to use a modern parser, which brings us back to the lack of extension support.

On the one hand, it seems like this library is not long for this world. On the other hand, there continues to remain a need for the extension support that this library provides. Given the above, there are a few different ways that things could move forward.

1. We could continue to maintain the existing library, but include clear warnings that it should only be used for documents from a trusted source. In other words, we would warn users that they should not use this library to parse Markdown content provided by untrusted sources, such as comments, forum posts, and the like.
2. An existing modern implementation could be found and work could be done to implement many of the popular extensions to work with that library.
3. A new implementation could be built from the ground up. Then, various popular extensions could be developed on top of it.

To be clear, personally, I do not have the bandwidth to pursue any of those options except the first one. That said, I think that one of options 2 or 3 would be better for users overall. I just don't know when or how the work will get done.

Regarding option 1, it does not seem too far-fetched when you consider that the primary use case for many of our users is as a static site generator (i.e., MkDocs and its clones). In that use case, the entire Markdown content is being parsed either locally or on a CI server prior to it being posted live as rendered content. The only attack vector is if the source code for a given project was available on a public repository and someone submitted a pull request which would trigger one of these lengthy parsing sessions. But even then, the worst-case scenario is that the CI server would time out, causing the maintainer of the project to look more closely at the submitted code and reject it. No threat of a denial-of-service attach there.

The primary concern with option 1 is that the library is used in other contexts. For example, last time I checked, Django was still using Python Markdown to parse user-supplied Markdown comments, among many other use cases that a full-fledged web framework can address. That said, I can't imagine that these use cases make extensive use of third-party extensions. Therefore, it would not be terribly painful for users to have to switch to a different implementation in those instances. However, users would understandably be frustrated if they needed to continue using our library to support third-party extensions for their blog post, but a different implementation to parse user-submitted comments on that same blog post.

What is clear to me is that there is no great option going forward. However, the current pace of obscure bug reports is unsustainable, and for that reason, the future of this project is not clear to me. Something needs to give.