Repository navigation
rewrite the parser #2205
Description
Click currently uses a modified copy of optparse to parse command line arguments. In order to use it, Click creates an entire optparse class structure mirroring the structure that users write, which seems like unnecessary overhead. Click also goes beyond what optparse was intended for, which means there's a lot of working around how optparse works. There are old issues about the processing being an issue, and newer features are also affected because it's hard to manage whether the parser or the processing will get values or raise errors.
Most of the code in our parser copy isn't even the actual parsing code, it's all the optparse equivalent objects to Click's objects. I've started working on a new parser that works directly from the Click objects, instead of constructing a separate parser structure.
The following open issues are related, although I'm not guaranteeing that they'll all be immediately fixed with the rewrite. Some of them are also affected by the processing pipeline, which happens after parsing.
- The leftover arguments are empty in Click 6.0 Context #473
- Click Processing Pipeline Rewrite #590
- Wrong argument handling when specifying 'ignore_unknown_options' in combination with options #602
- Wrong parsing of argument when passing an option #611
- Double-dash separator can be used between group name and commands, has odd behavior #619
- Subcommand path from a top/root context #680
- Allow for prohibiting the no-blank way of specifying short option with option argument #742
- Chained command with argument and option results in "No such command" #1269
- provide access to a normalized list of args that will result in the same params #1279
- Subcommands prefixed with a forward slash are seemingly ignored #1322
- Ignore/pass over the double dash (--) marker #1340
- Command w/o args displaying usage should return non-zero #1394
- Help shown via no_args_is_help results in exit code 2, was 0 #1489
- Context.params isn't setup in the MultiCommand.list_commands method #1574
- Binary flags vs
default=None: semantics and issue with prompt #1992
Activity
davidism commentedon Mar 5, 2022
I've got the parser rewritten, and am going through making the tests pass. Things I've noticed so far:
OptionParseris exported at the top level, I'll use module-level__getattr__to show a deprecation warning.BaseCommandandCommand. It seems it was done to allow commands that used other parsers, but nothing aboutCommandprevents that, the developer could already overrideparse_argswithout needingBaseCommand. It also makes type annotations more complicated. I'll deprecateBaseCommandin the same way I didBaseRequestin Werkzeug.MultiCommandfromGroup, but I haven't looked at that as much yet.Contexthas three settings,allow_extra_args,allow_interspersed_args, andignore_unknown_options, that all take defaults based on class attributes onCommandandGroup, but can also be overridden bycontext_settings. These settings affect how the parser works, and should probably not be exposed to users. Mainly,allow_extra_argsshould not be a documented way to capture arguments in user code, as it mixes the state the processing pipeline works with for sub/chained commands with information the user might want. Instead, if users explicitly want extra args, they should useargument(nargs=-1). (On a side note, it's weird that one of them isignoreand the other two areallow.)ctx.argsandctx.protected_args. They are split after parameter parsing and processing, then immediately re-joined in the invoke step. The split was introduced as part of chaining at some point, but it's not clear why it was need for it, I removed it and everything still works.ofek commentedon Sep 16, 2024
Has this been merged?
davidism commentedon Sep 16, 2024
no, I haven't had time to finish it yet
ofek commentedon Sep 16, 2024
Ah okay, just wanted to check! Do you happen to have an estimate of the potential decrease in import/load time?
ofek commentedon Mar 28, 2025
I use
ignore_unknown_optionsvery frequently to capture arbitrary arguments (without users having to always supply--) in order to pass everything to some other CLI. Please don't remove that one 😅Fix behaviour change introduced by pallets/click#2205