Skip to content

lab-ops-hub: container validator returns empty path, empty port and empty userinfo un-normalized — recorded URL differs from the Worker #639

Closed

Description

@dezren39

Finding

The container validator returns three spellings without the WHATWG normalizations the Worker validator applies, so for the same accepted target the two crawlers record different URLs:

  1. Empty path gets no /. http://example.com and http://example.com:8080 are returned with an empty path; the Worker returns http://example.com/ / http://example.com:8080/.
  2. An empty port keeps its colon. http://example.com:/path is returned verbatim; the Worker drops the empty port and returns http://example.com/path.
  3. An empty userinfo keeps its @. http://@example.com/ is returned verbatim; the Worker serialises the empty userinfo away and returns http://example.com/. (The fetcher already treats a bare @ as carrying no credentials — _split_userinfo, app.py — so this arm is purely about the returned/recorded form.)

validate_crawl_url (pkgs/lab-ops-hub/container/app.py) returns _url_with_encoded_components(raw …) — the caller's spelling with only the character-encoding fixes. No step supplies the / for an empty path, drops an empty port, or drops an empty userinfo. The Worker validator (pkgs/lab-ops-hub/src/lib.ts:130) returns u.toString(), the fully WHATWG-normalised form. urlparse(...).geturl() also drops empty ?/# markers the Worker preserves (http://example.com/path? → container …/path, Worker …/path?) — same root cause, lower stakes; fixing it is optional but should be decided explicitly.

This is distinct from #633 (dot segments, host case, default ports): an empty port is not a default port, and none of #633's three arms touches the empty path or empty userinfo.

Evidence

Verified this run on current main, driving both real validators (Python validate_crawl_url; Worker's validateCrawlUrl under Bun):

Input Container returns Worker returns
http://example.com http://example.com http://example.com/
https://example.com https://example.com https://example.com/
http://example.com:8080 http://example.com:8080 http://example.com:8080/
http://example.com:/path http://example.com:/path http://example.com/path
http://@example.com/ http://@example.com/ http://example.com/
http://example.com/path? http://example.com/path http://example.com/path?
http://example.com/path# http://example.com/path http://example.com/path#

Controls that already agree: //double//slash, %41%42, %zz, :0/, ;params, non-empty userinfo.

Why it matters

The validated URL is the crawl record: the two crawlers keying the same target under different spellings (http://example.com vs http://example.com/) defeats cross-crawler dedup/comparison, and an empty-port colon is a spelling no other URL parser in the pipeline produces.

Acceptance criteria

  • validate_crawl_url returns the WHATWG form for the cases above: / supplied for an empty path, empty port colon dropped, empty userinfo @ dropped.
  • A differential test against the Worker validator covers these spellings (plus a decision, recorded in the test or PR, on the empty ?/# markers).
  • Existing container validator tests stay green; no change to which targets are accepted or refused.

Activity

  1. added a commit that references this issue on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions