ClearURLs breaks file uploads on iCloud Drive. #315

Activity

1. hannadevtools commented on Jul 29, 2026

   

   hannadevtools

   on Jul 29, 2026

   More actions

   A controlled A/B capture should narrow this down quickly:

   1. In Firefox, use the same small non-sensitive file and a fresh iCloud Drive session.
   2. Clear the Network panel, enable Persist Logs, then capture one upload with ClearURLs enabled.
   3. Repeat after disabling only ClearURLs.
   4. Compare the singleFileUpload request's final URL/query, request headers, redirects, request body length/content type, and the first differing response.

   The highest-value check is whether a signed query component or Apple-specific request header exists in the working capture but disappears in the failing one. That would distinguish URL filtering from a downstream iCloud lookup failure. It is a hypothesis, not yet a diagnosis.

   An iCloud HAR can contain session cookies and signed upload URLs, so I would avoid attaching the raw file. I built ReqRescue for exactly this handoff: it runs locally in the browser, masks cookies/tokens/email/IP, and exports a Markdown diagnosis. The source is linked from the page. If you run the two captures through it, posting the sanitized Markdown reports (or just the differing request summary) should give maintainers useful evidence without exposing the raw sessions.
2. Skyzenfr commented on Jul 29, 2026

   

   Skyzenfr

   on Jul 29, 2026

   Author

   More actions

   Thanks. I completed the controlled A/B comparison using the same browser, machine, iCloud account, and file.

   Result

   The issue is caused by ClearURLs removing the ref query parameter from Apple's signed upload URL.

   Upload sequence

   Before the actual file upload, iCloud sends a request to:

   /ws/com.apple.CloudDocs/upload/web

   In both captures, Apple's JSON response contains a signed singleFileUpload URL with these query parameters:

   tk, ref, c, z, uuid, dataclass, p, and e

   So Apple correctly provides the ref parameter even while ClearURLs is enabled.

   ClearURLs disabled — working

   The subsequent singleFileUpload request preserves all parameters, including:

   ref=<redacted>

   Result:

   HTTP 200 OK

   ClearURLs enabled — failing

   The preceding /upload/web response still contains:

   ref=<redacted>

   However, the actual outgoing singleFileUpload request contains only:

   tk, c, z, uuid, dataclass, p, and e

   The ref parameter has been removed.

   Result:

   HTTP 400 Bad Request

   The associated CORS OPTIONS request also lacks ref when ClearURLs is enabled, which indicates that the URL has already been modified before the network request is sent.

   Other comparison points

   The following were identical between the working and failing captures:

   • Request method: POST
   • Host and upload endpoint
   • Content-Type: image/png
   • Content-Length: 971673
   • Origin and Referer
   • Remaining request headers
   • Uploaded file and file size
   • No redirect occurred

   This confirms that the missing ref query parameter is the first relevant structural difference.

   The evidence shows that Apple returns a complete signed URL, but ClearURLs removes ref before Firefox sends the upload request. Since ref appears to be part of Apple's signed upload URL, removing it invalidates the request and causes the HTTP 400 response.

   A rule exception for the ref parameter on cws.icloud-content.com / the singleFileUpload endpoint should likely resolve the issue.

   I have not attached the raw HAR files because they contain iCloud session information and signed URLs, but I can provide additional sanitized details if required.
3. hannadevtools commented on Jul 29, 2026

   

   hannadevtools

   on Jul 29, 2026

   More actions

   Excellent isolation — this is enough evidence for a narrow fix and regression test; the raw HAR should not be needed.

   I also checked the current ClearURLs rule feed at https://rules2.clearurls.xyz/data.minify.json. Under globalRules, referralMarketing currently includes:

   (?:%3F)?ref_?
   

   and the exception list does not currently contain icloud.com or icloud-content.com. That matches the observed removal of the plain ref parameter when referral-marketing filtering is enabled.

   The smallest fix appears to belong in the rules dataset rather than the generic request-cleaning code: add an exception scoped to the Apple signed-upload host/path (for example cws.icloud-content.com/.../singleFileUpload), or to the precise Apple upload family maintainers consider safe.

   A useful regression test would assert all four points:

   1. The cleaned singleFileUpload URL preserves ref.
   2. The other signed parameters remain byte-for-byte unchanged.
   3. Both the OPTIONS and POST requests carry the same query keys.
   4. An unrelated URL with a referral ref parameter is still cleaned, so the exception is not global.

   One transparent product-research question for ReqRescue: did you run either capture through ReqRescue, or did you perform this comparison manually? If you used it, I would value one sentence on what helped or what was missing. If you did not, that is equally useful to know; I will not treat the answer as an endorsement.
4. Skyzenfr commented on Jul 29, 2026

   

   Skyzenfr

   on Jul 29, 2026

   Author

   More actions

   Thanks for confirming the matching rule and the likely fix.

   I did not use ReqRescue for this investigation. I performed the comparison manually by examining the two HAR files side-by-side.

   Your suggestion to compare the first differing signed query component was very helpful and ultimately led to identifying the missing ref parameter.

   Your finding about the referralMarketing rule ((?:%3F)?ref_?) also matches exactly what I observed.

   Thank you for investigating this issue. I'm happy to test any fix if needed.
5. hannadevtools commented on Jul 29, 2026

   

   hannadevtools

   on Jul 29, 2026

   Last edited by hannadevtools

   More actions

   Thank you — that manual workflow exposed a real gap in ReqRescue, so I implemented it rather than opening another generic feedback thread.

   There is now a free Compare two HARs (A/B) mode in the deployed app: open the A/B analyzer. Choose the working capture first (A) and the ClearURLs-enabled capture second (B). It compares sanitized request structure locally and ranks missing query keys, status transitions, new/removed requests, and latency regressions; values and HAR bytes never leave the tab.

   For this pair, the expected useful result is the ref key disappearing before singleFileUpload, alongside the 200 → 400 transition. If you are willing to spend 15 seconds on it, could you tell me only whether that appears as the first ranked difference? A “yes” is enough; if not, one sentence about what it ranked instead would be more valuable than an endorsement.

   I also removed the payment/license flow entirely. The analyzer, A/B comparison, PDF, 75 MB per-file limit, and local history are all free and open source.
6. KevinRoebert

   transferred this issue fromClearURLs/Addonon Jul 30, 2026
7. Skyzenfr commented on Jul 30, 2026

   

   Skyzenfr

   on Jul 30, 2026

   Author

   More actions

   Yes.

   The missing ref query parameter is ranked as the first structural difference, exactly as expected.

   The comparison also highlights the HTTP 200 → 400 transition on the singleFileUpload request.

   Additionally, it reports the same missing ref parameter for the preceding OPTIONS request, which further supports that the URL is being modified before the upload request is sent.

   The new A/B comparison mode made this very easy to identify. Thanks for implementing it so quickly.