Repository navigation
task(atlas): extract the GitHub corpus — all issue/PR comments from all our repos; external set gated on licensing #6023
Description
Parent: #6018 (Atlas ingestion program). Statements S11, S12, S14 of the 2026-11-04 brief.
Task (Drew, in substance)
Extract the GitHub corpus into Atlas: the comments are some of the highest-density design record we have, and today Atlas holds code/docs/sessions but not the issue/PR conversation layer.
- All of our repos: ALL comments from ALL pull requests and ALL issues — issue comments, PR review comments, and reviews — across the whole estate (developing-today/code, developing-today-forks/*, dezren39/darwin-nix, dezren39/hatch-personal, dezren39/hatch, dezren39/zettel, dezren39/cairn, and any other repo under our accounts/orgs).
- External repos: also extract from external repos we have interacted with in the past or rely on — a defined, enumerated set (repos where we filed issues/PRs, commented substantively, or that are load-bearing dependencies), NOT the whole of GitHub.
Gate
The external half is build-gated on the licensing research issue filed alongside this one under #6018 (may we copy external comments/discussions into Atlas at all?). Drew's sequencing: licensing is figured out before the external build, after the design is somewhat done. The our-repos half is NOT gated and may proceed independently.
Design/extraction must cover
- Enumeration method for the external set: derive candidates from gh activity (issues/PRs authored or commented by our accounts), dependency manifests across our repos, and fork upstreams; the enumerated list is an artifact Drew can strike items from before extraction.
- Extraction mechanics: gh API only; incremental by
updated_atwith a watermark (this corpus keeps growing — extraction is a recurring feed into the ingestion-operations design, not a one-time dump); rate-limit discipline; Discussions included where the repo has them (licensing answer may treat Discussions differently). - Corpus shape in Atlas: comment as the chunk unit with its thread context (issue/PR title, state, labels, author, timestamps, repo@hash-style provenance mark per the program's provenance standard —
github.com/<owner>/<repo>#<n>+ comment id + observed date). - Author attribution preserved (our comments vs others' comments are distinguishable — matters for both search quality and the licensing posture).
- Dedup against content already in Atlas (issue text quoted into docs/sessions) via the #5733 dedup edge + #3125 occurrence recording: a comment found in two places is one chunk with two occurrences.
Done looks like
Our-repos extraction running as a repeatable, watermark-incremental job with counts reported (repos × issues/PRs × comments landed, per tier); the external set enumerated and awaiting the licensing gate; follow-on automation folded into the ingestion-operations design.
Activity
developing-today commentedon Nov 5, 2026
CODE-1689
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
1 remaining item
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 5, 2026
🤖 Agent note · state: picked up
repo-worker-code), lanec8b9355e-319a-4640-8096-7ad19480d151dezren39 commentedon Nov 5, 2026
🤖 Agent note · triage complete (task 10390)
Verdict: #6023 remains OPEN; no matching extraction PR or existing Atlas extraction implementation found. This is a valid distinct task, scoped to issue comments, PR reviews, and review comments; no code change is warranted in triage. Revalidated against
mainat00a294ab6a38b23b3a3f855ff5e6102d5aed78c5.Current implementation evidence:
pm-stack/projectoralready reconciles and projects GitHub issues/PRs/comments/reviews for the configuredPM_REPOSlist, but its README defines it as a disposable PM read model, not Atlas ingestion. It covers only a configured repo set and does not create Atlas chunks/occurrences.pkgs/atlas-searchis search/runtime code with no GitHub ingestion writer. Reuse the projector/API patterns if useful, but this task still needs the Atlas source/ingestion path, provenance and idempotent occurrence linking (per #3125 / #5733), and explicit watermark behavior; the existing comment collections are re-fetched whole because GitHub does not supportsinceon those endpoints.Scope and order: proceed with our-repo full-text extraction independently (the parent #6018 says it is ungated); enumerate external repositories/metadata separately, and do not generalize the provisional #6024 result into permission for shared external-text ingestion. #6024 is still OPEN and #6055 retains lawyer questions; keep external full text personal-search-only pending an accepted posture. The already-triaged sibling #6025 consumes #6023’s output “as it lands”; keep this task as the comment/review source and avoid duplicating its first-pass code/docs/cairn ingestion. Coordinate recurring schedule/cost and source-set behavior with #6019; provenance follows #3125.
Blockers: none for our-repo extraction. External full-text ingestion remains restricted by the provisional licensing gate; its metadata enumeration can proceed.
repo-worker-code), lanec8b9355e-319a-4640-8096-7ad19480d151