Repository navigation
[Bug] PathUtils.equalsIgnoreSchemeIfOneIsS3 path comparison is inconsistent between same-scheme and cross-scheme branches #64767
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
master (fe-foundation, org.apache.doris.foundation.util.PathUtils).
What's Wrong?
PathUtils.equalsIgnoreSchemeIfOneIsS3(p1, p2) compares two storage-location URIs treating the s3 scheme as interchangeable with other object-store schemes. Its two branches used inconsistent rules:
- Same scheme →
p1.equalsIgnoreCase(p2): full-string, case-insensitive, trailing slash significant. - Cross-scheme (one is
s3) → comparesnormalize(authority)/normalize(path)withObjects.equals: case-sensitive, trailing slash stripped.
Consequences:
- The result for one URI depends on the other URI's scheme. For example
s3://bucket/path/vss3://bucket/pathare unequal (same-scheme branch), buts3://bucket/path/vscos://bucket/pathare equal (cross-scheme branch). - The same-scheme branch ignores case for the whole string, so it can wrongly equate case-sensitive S3 object keys (
s3://b/A==s3://b/a).
The only caller is HMSTransaction.prepareInsertExistingTable (fe-core), which uses this to decide whether a Hive commit needs a rename — so inconsistent equality can lead to an incorrect rename decision.
What You Expected?
A single, consistent rule regardless of whether the two schemes match: compare authority + path (scheme ignored when equal or when one side is s3), with trailing slashes insignificant and the comparison case-sensitive (object-storage keys are case-sensitive).
How to Reproduce?
// same pair, different "other" scheme -> different answer (inconsistent):
PathUtils.equalsIgnoreSchemeIfOneIsS3("s3://bucket/path/", "s3://bucket/path"); // false
PathUtils.equalsIgnoreSchemeIfOneIsS3("s3://bucket/path/", "cos://bucket/path"); // true
// same-scheme comparison wrongly ignores case:
PathUtils.equalsIgnoreSchemeIfOneIsS3("s3://bucket/A", "s3://bucket/a"); // true (should be false)Anything Else?
Fix proposed in the linked PR. It also hardens several edge cases surfaced during review (opaque URIs, percent-encoded slashes, triple-slash / network-path forms) by falling back to exact string comparison for inputs that are malformed for object storage.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Activity
zclllyybb commentedon Jun 24, 2026
Breakwater-GitHub-Analysis-Slot: slot_bc1fd4b9373a
This content is generated by AI for reference only.
Initial triage: this looks like a real consistency bug in current
master.I refreshed
apache/dorisupstream/mastertoe5f3badd0109e312167f242df5aa53adb86806d8and checked the referenced code path. Infe/fe-foundation/src/main/java/org/apache/doris/foundation/util/PathUtils.java,equalsIgnoreSchemeIfOneIsS3currently has two different equality contracts:equalsIgnoreCasecomparison, so object key case is ignored and trailing slashes remain significant.s3compare normalized authority and path withObjects.equals, so the comparison is case-sensitive and strips trailing slashes.That matches the issue's reproduction. It also matters to production behavior: the production use I found is
HMSTransaction.prepareInsertExistingTableinfe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSTransaction.java, where the result directly drivesneedRename = !PathUtils.equalsIgnoreSchemeIfOneIsS3(targetPath, writePath). Therefore:s3path treats the same location as equal.s3://bucket/Aversuss3://bucket/acan be incorrectly treated as equal, which is unsafe for object-storage keys.I also found the linked public fix PR: #64768. It is currently open as a draft, based on the same
masterhead, and its patch changes the utility to use one structural comparison rule for same-scheme ands3cross-scheme cases. The addedPathUtilsTestcases cover the reported trailing-slash consistency issue, case-sensitive path/authority comparisons, malformed object-storage URI forms, encoded slashes, query/fragment distinctions, ands3a/s3nbehavior.Suggested next steps:
targetPathandwritePathfollow the unified location-equality rule.Missing information is not blocking the code-level conclusion here because the logic bug is visible from the current source and the issue includes direct method-level reproductions. A real-world incident assessment would still need the exact table location, write path, storage scheme pair, Doris build SHA, and FE commit/rename logs.