Howto handle hook changes

I have a hook which needs to be changed. In essence it needs an additional argument.

The problem is, that plugins should be able to use the new argument, but still work on older versions where the argument didn't exist yet.

If I add the argument and use it in the hook implementation in the plugin, then the plugin doesn't work with older versions anymore: Argument(s) ... are declared in the hookimpl but can not be found in the hookspec

Using kwargs with defaults also doesn't work, the argument isn't passed then.

If I add a new hook, then both my implementations in the plugin are called, because the old hook still needs to be called for older plugins.

One idea would be the possibility to mark a hook specification deprecated with the name of the replacement and then when loading a plugin and it implements both hooks forget about the old hook. If it only supports the old hook, then add a warning about the deprecation.

this is a unsolved problem as of now - we didn't get the attempted solution to a thing that works

pytest also has this issue as the pytest_deselected hook is supposed to be invoked by externals but lacks a critical needed argument

@fschulze

@fschulze why can't you add the new argument to the hookspec and then then call the hook using both the new and old argument (hookimpls don't have to declare all args in the spec or in the call - though the latter is not documented well / it's implied). Then when you are ready to deprecate the old argument use the new deprecation feature from #138 to slowly fade out the old argument?

Using kwargs with defaults also doesn't work, the argument isn't passed then.

I actually did implement this and it works (#15 leading to #43) but we never got to consensus on whether it was actually the right way to solve this issue.

@RonnyPfannschmidt

Shoot just realized #138 is only for deprecating full hooks and not specific args.

@RonnyPfannschmidt what do you think about something similar like:

@hookspec(
    warn_on_args={'arg1': DeprecationWarning("arg1 is deprecated and will be removed soon")
)
def oldhook(arg1, new_arg1):
    pass

@tgoodlet doesn't work even remotely since it ignores stuff like caller handling

@tgoodlet that still wouldn't solve the issue, that the plugin should still work with older "hosts" where the argument wasn't there yet. And I don't want to deprecate the old argument, just introduce an additional one.

For me an entirely new hook would be fine, but then I need a way to prevent the old hook to be called in plugins that support the new hook.

Add a test demonstrating the opt in nature of args

@RonnyPfannschmidt

doesn't work even remotely since it ignores stuff like caller handling

@RonnyPfannschmidt explain? Also what are you referring to with this comment?

that still wouldn't solve the issue, that the plugin should still work with older "hosts" where the argument wasn't there yet.

@fschulze sorry, why wouldn't this work?

To make this work:

add the new arg to the hook spec and call
some new hookimpls receive the new arg
some old hookimpls receive the old arg

I'm not sure I'm following what the problem is and I've opened #172 to demonstrate how you can accomplish this.

@tgoodlet hook calls are in plugins, now plugins are broken in fun ways as all of them would have a breaking change

so the proposed mechanism you showed is one that require breaking api changes

hook calls are in plugins, now plugins are broken in fun ways as all of them would have a breaking change

I'm not following - where's the breakage? This new warn_on_args wouldn't conflict with anything (since it's an addition not a change to anything existing) and hook calls can continue to exist as they have inside plugins?

@tgoodlet plug-ins can no longer call hooks as the args changed

@RonnyPfannschmidt

@RonnyPfannschmidt right which is why you first deprecate and then later remove - just like any API progression? If you're going to deprecate and change an API at some point you have to stop supporting it. As per #172 you can definitely keep deprecated and new args working in tandem.

To me this is simple:

warn_on_args causes deprecation warnings to emit at the hook call line as well as the hookimpl def
old hook calls continue to work just fine but now with the warnings for any call or hookimpl that uses the deprecated args
eventually hook call sites change when the arg is fully removed or else you'll get the classic pluggy error
we'll need a slight bit of extra logic to warn differently on deprecated args (i.e. those listed in warn_on_args) which are in the spec but not in the impl

Implementing this will give good reason to finally bring in #57 as well since we'll want to keep track of which args per hookspec are deprecated.

@tgoodlet all callers must include all arguments - so i have no idea what you are talking about

only receivers can leave arguments out

all callers must include all arguments - so i have no idea what you are talking about

Yes - no confusion there. So you change the spec to include warn_on_args and keep all your callers as is.
Then after enough time of these args being marked deprecated you change the call lines?

At some point the calls have to change, I don't see how you can ever get around that.