Agent builder report: end-to-end lessons from a generated skill release

[sergeliatko]

Agent Builder Report: End-to-End Skill Creation and Publication

Author: the Codex agent that built and released Coordinate GitHub Repositories
under user direction on 2026-07-17.

This is an agent-experience report. It records observed workflow behavior and
agent judgments from one complete template-to-public-release session. It does
not expose private intake, local paths, repository maps, credentials, or user
profile data.

Problem Statement

The template supplied a strong intake, packaging, safety, and cleanup
foundation, but several gaps appeared only when the generated repository was
taken through GitHub-native discovery, protected pull requests, two public
releases, exact release-asset audits, and installation design for nontechnical
users.

The highest-impact gap was a mismatch between the template's generated
src/SKILL.md convention and the installed GitHub CLI's publication discovery
contract. gh skill publish discovers skills/*/SKILL.md and requires the
skill name to match its containing directory. Resolving this after the initial
release required a second pull request that moved the runtime to
skills/coordinate-github-repositories/ and fixtures to tests/fixtures/.
One validator path remained pointed at src/, so the later v1.0.1 patch also
had to restore effective runtime privacy scanning.

A second gap was overlapping release authority. The generated draft-release
workflow and gh skill publish can both create or update tags and releases.
During this session, concurrent publication activity moved the v1.0.0 tag to
the discovery-layout commit and regenerated release assets after an earlier
audit. The final result was correct, but it required another download and
archive audit because previously recorded hashes were no longer authoritative.

A third gap was installation usability. The generated install guide was
accurate for maintainers but assumed users understood hidden folders, global
versus project scope, ZIP structure, plugin terminology, and GitHub's automatic
Source code archives. A nontechnical user did not have one obvious action.

Public Evidence

• Initial skill release pull request:
  Release Coordinate GitHub Repositories skill v1.0.0 skill-github-repositories-coordination#2
• GitHub discovery-layout correction:
  Fix GitHub skill discovery layout skill-github-repositories-coordination#4
• Nontechnical installation patch:
  Release easier global installation in v1.0.1 skill-github-repositories-coordination#5
• Final public patch release:
  https://github.com/TechSpokes/skill-github-repositories-coordination/releases/tag/v1.0.1
• Generated repository:
  https://github.com/TechSpokes/skill-github-repositories-coordination

What Worked Well

1. The intake-as-evidence rule protected authority boundaries. Private local
   repositories could inform the skill without becoming instructions or public
   package content.
2. The bootstrap and maintenance modes gave the agent a clear conversion
   boundary. Removing .template/ and planning files produced a maintainable
   public repository rather than a permanently half-generated project.
3. The portable core plus focused references worked well. The final runtime is
   seven files and remains byte-identical across standalone, Codex plugin, and
   Claude plugin packages.
4. Activation and behavior fixtures made it possible to review writers,
   researchers, beginners, mature organizations, conversation-only agents, and
   coding agents without treating exact model wording as the contract.
5. The three-package release model was understandable after its purpose was
   explained, and the package boundary successfully excluded intake, private
   research, maintenance docs, tests, local paths, and temporary output.
6. The .skill-template-feedback/ convention provided the right place to stage
   this report without shipping it in the skill.
7. The generated CI check name was stable enough to use in a required-status
   ruleset: Validate skill package.

Agent Experience by Phase

Intake and Research

The scope evolved through several user additions: code and non-code work,
general agents and coding agents, user-context calibration, tool-fit advice,
global installation, publication, repository hardening, and nontechnical
installation. The template's intake guidance handled evidence quality, but the
agent needed an additional persistent decision ledger to distinguish:

• user requirements;
• private local observations;
• official product facts;
• public comparison evidence;
• durable design judgments;
• deferred ideas.

Without that ledger, long research sessions risk losing why a conclusion was
adopted or which claim needs a dated source.

Generation and Cleanup

The template generated a large, coherent conversion diff, but path knowledge
was duplicated across AGENTS.md, README, architecture docs, validation,
packaging, tests, and release instructions. A later layout change therefore
required coordinated edits in many places and still left one stale src
reference inside the validator.

The generated source root should be a single configuration value or discovered
from the skill entry point. Validators and packagers should not independently
hardcode the same path.

Validation and Packaging

The built-in validator and packager were useful, but final confidence required
additional checks that are not yet first-class template tasks:

• Agent Skills quick validation;
• Codex plugin validation;
• Claude plugin validation;
• parsing every GitHub YAML file;
• gh skill publish --dry-run;
• exact ZIP entry inspection;
• cross-package runtime byte comparison;
• forbidden private/local/bootstrap pattern scanning;
• manifest version verification;
• downloading the server-generated release assets and repeating the audit.

The audit also had to normalize Windows backslashes and Linux forward slashes
inside ZIP entry names. A reusable cross-platform audit script would remove
this session-specific work.

The Python validators required PyYAML, which was not available in the base
environment and needed an ephemeral virtual environment plus elevated access
to a cached wheel. The template should either pin validation dependencies in
the repository or provide an equivalent dependency-free or Node-based check.

GitHub Publication and Repository Hardening

The repository started public but without a description, topics, Discussions,
security scanning, Dependabot security updates, merge restrictions, or
rulesets. The agent compared established TechSpokes skill repositories and
applied this baseline:

• squash-only merges and automatic branch deletion;
• Discussions enabled; Projects and Wiki disabled;
• secret scanning and push protection;
• Dependabot alerts and security updates;
• private vulnerability reporting;
• a main ruleset requiring pull requests, the package check, resolved review
  threads, current branches, linear history, and no deletion or force push;
• a v* tag ruleset preventing deletion and non-fast-forward updates;
• agent-skill and agent-skills discovery topics.

This was valuable but almost entirely manual. A generated repository hardening
checklist or parameterized setup script would make the intended public baseline
repeatable and auditable.

Two agents or publishing flows also operated on the same repository during the
initial release. That exposed a missing release coordination mechanism. The
template should assume that shared workspaces and concurrent agents are
possible: re-fetch remote state before tagging, compare exact tree hashes,
check open pull requests and active workflows, and establish one publication
owner.

Installation Experience

The latest prior TechSpokes skill release put installation near the top of its
README, explained all three assets, and warned against Source code downloads.
Those were good patterns. The generated repository still needed a clearer path
for a person who does not use a terminal.

Version v1.0.1 added:

• a copyable prompt asking a capable agent to install at user scope;
• one recommended standalone ZIP with a direct download link;
• browser-only File Explorer, Finder, and Linux file-manager steps;
• a plain-language package chooser;
• an explicit warning against GitHub's Source code archives;
• GitHub CLI examples with --scope user for Codex, GitHub Copilot, and Claude
  Code;
• a final folder tree and a copyable verification prompt;
• validation that the versioned download URL and installation commands match
  package metadata.

This should be the generated default, not an improvement every downstream skill
has to rediscover.

Proposed Solution

P0: Align the Generated Layout With Publication

Generate maintained repositories with:

skills/
`-- <skill-name>/
    |-- SKILL.md
    `-- references/

tests/
`-- fixtures/

Derive the canonical skill root once and reuse it in validation, packaging,
documentation, and tests. Require the directory name to equal frontmatter
name. Add gh skill publish --dry-run to the release gate.

Acceptance criteria:

• A freshly generated repository passes gh skill publish --dry-run without a
  layout-only follow-up change.
• No validator or packager references an obsolete source root.
• A test intentionally places a forbidden value in the canonical skill root
  and proves that the privacy validator fails.

P0: Establish One Release Authority

Choose and document one of these models:

1. The generated workflow owns tagging, packaging, draft creation, asset
   upload, audit, and publication; gh skill publish is validation/discovery
   only.
2. gh skill publish owns the release, and a compatible follow-up workflow
   attaches audited package assets without racing release creation.

Do not leave both flows able to create or repoint the same release implicitly.

Acceptance criteria:

• A release tag is created once from a verified merge commit.
• Workflow concurrency prevents two release jobs for the same tag.
• Published assets are downloaded and audited before completion is reported.
• A post-publication hash change causes a failed audit or explicit re-audit.
• The release checklist includes branch cleanup after merge and publication.

P1: Generate a Nontechnical Installation Experience

Put an install section above architecture or development content. Generate the
current version into:

• one recommended standalone download link;
• a copyable agent-assisted user-scope prompt;
• browser-only extraction instructions;
• a three-asset chooser;
• a Source code archive warning;
• host-specific user-scope CLI examples;
• final folder and activation verification.

Keep technical plugin and project-scope details below the beginner path. Add a
validator that ensures versioned asset links and commands match package.json.

P1: Add a First-Class Release Audit

Provide a script such as npm run audit:release -- vX.Y.Z that:

• requires exactly the expected ZIP names;
• normalizes ZIP entry separators across operating systems;
• checks the standalone and plugin runtime cores are byte-identical;
• verifies plugin manifest versions;
• rejects intake, bootstrap, local paths, private source names, placeholders,
  credentials, tests, and maintenance files;
• can audit local dist/assets/ or a downloaded release directory.

Run it in CI and again against GitHub-hosted assets before publication.

P1: Generate Repository Hardening Guidance

Add an optional, reviewed hardening step for public generated repositories:

• description and discovery topics;
• Discussions/Projects/Wiki defaults;
• squash-only merge configuration;
• secret scanning, push protection, Dependabot, and private vulnerability
  reporting;
• a parameterized main ruleset using the actual CI check name;
• protected version tags;
• verification output and a recovery path.

The step must remain opt-in because organization policy and permissions vary.

P2: Add a Build Decision Ledger

During bootstrap, create a git-ignored or explicitly private working record
with sections for requirements, evidence, dated external facts, decisions,
unknowns, rejected alternatives, validation evidence, and publication state.
Distill only durable public conclusions into docs before release.

P2: Make External Validation Reproducible

Pin or script the external validators and YAML parser. Report unavailable tools
honestly, but make the normal path one command instead of an ad hoc virtual
environment. Keep host-specific validation optional when the host is absent.

Impact on Generated Repositories

Existing generated repositories would not need an immediate migration if their
release and installation paths already work. The template should document the
layout change as a migration and provide a validator that can detect legacy
src/SKILL.md repositories.

New repositories would gain:

• GitHub-native discovery without a layout correction PR;
• fewer duplicated path edits;
• deterministic release ownership;
• repeatable privacy and archive verification;
• a safer public repository baseline;
• installation instructions usable without terminal knowledge;
• clearer evidence for future agents maintaining the skill.

Alternatives Considered

• Keep src/SKILL.md and package only into skills/<name>/: portable release
  ZIPs work, but GitHub repository discovery and publication still need special
  handling and the directory-name contract remains easy to miss.
• Use only gh skill publish: simple discovery and release creation, but it
  does not replace the current three-package asset build and privacy audit.
• Use only the custom workflow: good asset control, but native discovery must
  still be validated and the agent-skills topic managed.
• Add one-click executable installers: easier for some users, but expands trust,
  signing, platform, and maintenance scope. A copyable agent prompt plus browser
  steps is safer and more portable.
• Make hardening automatic: repeatable, but too invasive for repositories under
  different organization policies. Generate an opt-in, reviewable procedure
  instead.

Suggested Decomposition

1. Layout and single-source-root change.
2. Release ownership and concurrency change.
3. Cross-platform release audit command.
4. Generated installation UX and version-link validation.
5. Optional repository hardening procedure.
6. Bootstrap decision ledger and reproducible external validators.

Each change should update both template-mode validation and the generated files
under .template/generated/, then be exercised by creating a disposable skill
repository and taking it through gh skill publish --dry-run and a draft
release.

[sergeliatko]

Focused follow-up from this report is now tracked and implemented through reviewable pull requests:

• Fix generated workflow correctness and Windows portability #14 → Harden generated release and GitHub CLI delivery #19: generated workflow correctness and Windows portability
• Add deterministic, immutable release mechanics #17 → Harden generated release and GitHub CLI delivery #19: deterministic, immutable release mechanics
• Make GitHub CLI skill installation a release-gated contract #18 → Harden generated release and GitHub CLI delivery #19: release-gated GitHub CLI installation and update verification
• Generate a minimal evaluation and safety baseline #15 → Generate a reusable evaluation and safety baseline #20: generated evaluation and conditional safety baseline
• Use GitHub Issues as the maintenance and feedback surface #16 → Use GitHub Issues for maintenance and feedback #21: GitHub Issues as the maintenance and feedback surface

The canonical skills/<name>/SKILL.md layout and initial GitHub CLI delivery described here were already addressed on main. The focused issues preserve the remaining work and avoid turning this parent report into one implementation backlog.

Keep this issue as the coordination parent until the linked changes land.

[sergeliatko]

Coordination complete. #11 established the canonical GitHub CLI skill layout; #19 delivered deterministic release mechanics, workflow portability, and release-gated installation; #20 delivered the evaluation and safety baseline; #21 moved maintenance and feedback to GitHub Issues; and #22 prepared v1.4.0. The remaining P2 work is now tracked independently in #23 for the private bootstrap decision ledger and #24 for reproducible external validation. Closing this parent report because all landed scope is complete and every residual item has a focused owner issue.