Repository navigation
Description
Finding
PR #648 (#648, commit 7bd77d0937, merged as 93065f0d) added _empty_query_fragment_markers to preserve the empty ? / # markers WHATWG keeps (#639), but it re-attaches the markers as a suffix after geturl() at every rebuild site. When the query is empty and the fragment is non-empty, the ? therefore lands after the fragment instead of before the #:
| Input | Container returns (current main) |
Worker / WHATWG URL returns |
|---|---|---|
http://example.com/path?#frag |
http://example.com/path#frag? |
http://example.com/path?#frag |
http://example.com?#frag |
http://example.com/#frag? |
http://example.com/?#frag |
http://example.com/a?#b?c |
http://example.com/a#b?c? |
http://example.com/a?#b?c |
http://%65xample.com/path?#frag |
http://example.com/path#frag? |
http://example.com/path?#frag |
http://8%2e8%2e8%2e8/path?#frag |
http://8.8.8.8/path#frag? |
http://8.8.8.8/path?#frag |
Verified on current main by running validate_crawl_url from pkgs/lab-ops-hub/container/app.py (extracted fresh via gh api ...?ref=main) against the table above, cross-checked against Node's WHATWG URL serializer (the Worker's reference). The percent-decoded-host and canonical-IP rebuilds are affected too, because all three rebuild sites share the suffix pattern.
Evidence
pkgs/lab-ops-hub/container/app.py:784-798—_empty_query_fragment_markers()returns the markers as an unordered suffix string ("?"computed from the pre-#spelling,"#"if the fragment is empty).pkgs/lab-ops-hub/container/app.py:860—_normalize_validated_url:...geturl() + _empty_query_fragment_markers(url).pkgs/lab-ops-hub/container/app.py:937—_url_with_encoded_components: samegeturl() + markerssuffix.pkgs/lab-ops-hub/container/app.py:737-741—_url_with_host: same suffix on all three return arms.geturl()rebuilds a non-empty fragment as#fragat the end of the string, so a suffixed?is parsed back as fragment data, not as the query marker.- PR fix(lab-ops-hub): return empty path, userinfo and ?/# markers WHATWG-normalized (#639) #648's test battery covered
path?#(both markers empty — suffix order happens to be correct),path?x#(non-empty query), and#frag?notquery(?as fragment data), but not the empty-query + non-empty-fragment combination. Its 43,200-spelling differential evidently did not include that tail either. - PR fix(lab-ops-hub): drop the password colon when the password is empty in the returned URL (#651) #653 (open, fixes lab-ops-hub: container validator keeps the password colon when the password is empty (user:@ / :@) — returned URL differs from the Worker #651) stacks on this code and does not touch the marker logic, so it carries the same defect.
Why it matters
This is exactly the class of divergence #639 set out to eliminate: the two crawlers record different URLs for the same accepted target. Worse than a dropped marker, the returned URL is a different URL — the fragment is corrupted (frag? instead of frag) and the present-but-empty query becomes absent — so anything keying on the recorded URL (dedup, redirect handling, downstream fetch of the recorded form) sees a different target than the one that was judged and that the Worker records.
Acceptance criteria
validate_crawl_urlreturns the WHATWG form for empty-query + non-empty-fragment spellings, e.g.http://example.com/path?#frag→http://example.com/path?#frag, including through the percent-decoded-host and canonical-IP rebuild paths.- The empty-fragment marker case stays correct (
path?#,path#,path?x#unchanged) and a?inside the fragment stays data (#frag?notqueryunchanged). - New regression tests pin the
?#<non-empty fragment>combination in the Python suite (and the TS suite if the Worker pin is extended); the differential probe set includes empty-query × non-empty-fragment tails. - Existing Python / TS validator suites still pass; no verdict (accept/refuse) changes.
Activity
Merge pull request #663 from developing-today/sweep/654-empty-query-m…