Release process - tagged commits on branch

[21sw-clayton]

Downstream background

Yocto Project:

The Yocto Project (YP) is an open source collaboration project that helps developers create custom Linux-based systems regardless of the hardware architecture.

Yocto is used to build operating systems such as Automotive Grade Linux, Poky (reference distribution), TSEL, and Wind River Linux.

Yocto is industry-standard and has a reference in this project as "openembedded".

Yocto functionality is split into various "layers".

Understanding and Creating Layers:

Layers allow you to isolate different types of customizations from each other.

meta-security (a layer) includes this project ("downstream").

Project commits and branches

The meta-security master branch currently points to v0.1.76 using:
source revision (SRVREV): 616d436
source URI (SRC_URI) using this project's "stable" branch

At the time of the downstream commit, 616d436 was on the stable branch.
Something was changed in this project's branch history, as the commit is no longer on any branch:

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

When this happens, Yocto users are unable to use the recipe:

ERROR: scap-security-guide-0.1.76-r0 do_fetch: Fetcher failure: Unable to find revision 616d436 in branch stable even from upstream

I've also observed this behavior (inconsistently) in previous releases.

Request

Would it be possible to adjust the release process to ensure tagged commits remain on a branch?

[comps]

Would it be possible to adjust the release process to ensure tagged commits remain on a branch?

I've discussed this with other Red Hat maintainers and we might have a proposal (to be put into a PR for discussion at a later point in time).

I don't think it's feasible to have the tagged commits be part of the same linear history of master due to how the "stabilization" process works - this process can take weeks of testing & bug fixing, typically happening in parallel to the continuous master branch development.
Theoretically, as a project, we could decide to pause the merging of PRs during the stabilization phase, effectively switching on some "preparing for release, no wild changes" mode on master, but that might be more widely undesirable.

However, there is a way to keep the stable branch always pointing to the latest tagged release while having previous tagged releases in its history, which is what I think you're asking for.

The solution (to be proposed in a PR at a later point) is to have the stable branch consist of merge commits where

• The first parent is always a previous (merge) commit on the stable branch
• The second parent is the released tag

Effectively, this would create a tree like

A
|`-v0.1.77
B
|`-v0.1.76
C
|`-v0.1.75
D
|`-v0.1.74
...

where A,B,C,D,.. are merge commits (with two parents) titled ie. Merge in v0.1.77, and the tag names have full history under them (they are literally the same tagged commit hashes).

The A,B,C,D,.. merge commits would have exactly the same code (repository contents) as the latest merged-in tag. This can easily be double-checked by anyone with git-diff, or simply comparing the git tree object hashes.

Overall, this would effectively fulfill both use cases of

• I want latest released content without having to hardcode a tag name.
• I want a fast-forward-able branch that is never suddenly re-created from scratch, follows a contiguous (secured by SHA hashes) history, with all released tags reachable from it (ie. via git-describe).

Does that make sense?

[21sw-clayton]

Thank you for looking into this @comps!

This proposal appears to address my concerns from a downstream perspective.

I've submitted this: [meta-security][RFC] scap-security-guide do_fetch resolution - upstream release process

This is also a nice bonus (though the Yocto community explicitly prefers fixed revisions):

I want latest released content without having to hardcode a tag name.

Looking forward to the PR

[comps]

Also note that while the approach will likely solve the problem you're seeing, the current approach is valid too - 616d436 is reachable via the v0.1.76 tag.

In git, as long as an object is reachable through another object (blob via tree, tree/commit via another commit) or via what git calls a "ref" (branch, tag, anything under refs/), all is good. If an object (ie. a commit) is not reachable from any of these, it is considered "dangling" and subject to garbage collection.

In this case, 616d436 is not dangling, and having it reachable only from a tag is a perfectly valid scenario. So this looks like a Github UI bug (feature?) because it displays the same message for real dangling commits.

[comps]

@Mab879 @jan-cerny @vojtapolasek Is this on one of your radars? IIRC we wanted (and still want) to do it (meaning change the release process documentation) before the next release.

[vojtapolasek]

@21sw-clayton do you still consider this something wanted even after reading this comment? #13543 (comment)

[21sw-clayton]

Hi, @vojtapolasek

This was adjusted downstream with the comment:

Until their release model changes, just use the release commit with nobranch.

I would still like to see this change, as it matches typical expectations (e.g. GitHub's message) and eases traceability.