You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
lab-ops-hub: container validator returns empty path, empty port and empty userinfo un-normalized — recorded URL differs from the Worker #639
The container validator returns three spellings without the WHATWG normalizations the Worker validator applies, so for the same accepted target the two crawlers record different URLs:
Empty path gets no /.http://example.com and http://example.com:8080 are returned with an empty path; the Worker returns http://example.com/ / http://example.com:8080/.
An empty port keeps its colon.http://example.com:/path is returned verbatim; the Worker drops the empty port and returns http://example.com/path.
An empty userinfo keeps its @.http://@example.com/ is returned verbatim; the Worker serialises the empty userinfo away and returns http://example.com/. (The fetcher already treats a bare @ as carrying no credentials — _split_userinfo, app.py — so this arm is purely about the returned/recorded form.)
validate_crawl_url (pkgs/lab-ops-hub/container/app.py) returns _url_with_encoded_components(raw …) — the caller's spelling with only the character-encoding fixes. No step supplies the / for an empty path, drops an empty port, or drops an empty userinfo. The Worker validator (pkgs/lab-ops-hub/src/lib.ts:130) returns u.toString(), the fully WHATWG-normalised form. urlparse(...).geturl() also drops empty ?/# markers the Worker preserves (http://example.com/path? → container …/path, Worker …/path?) — same root cause, lower stakes; fixing it is optional but should be decided explicitly.
This is distinct from #633 (dot segments, host case, default ports): an empty port is not a default port, and none of #633's three arms touches the empty path or empty userinfo.
Evidence
Verified this run on current main, driving both real validators (Python validate_crawl_url; Worker's validateCrawlUrl under Bun):
The validated URL is the crawl record: the two crawlers keying the same target under different spellings (http://example.com vs http://example.com/) defeats cross-crawler dedup/comparison, and an empty-port colon is a spelling no other URL parser in the pipeline produces.
Acceptance criteria
validate_crawl_url returns the WHATWG form for the cases above: / supplied for an empty path, empty port colon dropped, empty userinfo @ dropped.
A differential test against the Worker validator covers these spellings (plus a decision, recorded in the test or PR, on the empty ?/# markers).
Existing container validator tests stay green; no change to which targets are accepted or refused.
Activity
Merge pull request #648 from developing-today/sweep/639-empty-components