Repository navigation
Tracking Issue for normalize_lexically #134694
Description
Feature gate: #![feature(normalize_lexically)]
This is a tracking issue for Path::normalize_lexically that is used to normalize a path, including .. parent references, without touching the filesystem.
Public API
// std::path
impl Path {
pub fn normalize_lexically(&self) -> Result<PathBuf, NormalizeError> {
}Steps / History
- API Change Proposal
- Implementation: Implement
normalize_lexically#134696 - Final comment period (FCP)1
- Stabilization PR
Unresolved Questions
- Should
PathBufhave a method that normalizes in-place? Normalizing a path just removes components. - Should this disallow resolving to the empty path or the single
.path. - Should this only allow paths without a root or prefix? It may be surprising that
base_path.join(user_path.normalize_lexically()?)can still escape the base path.
The last two can be summarised as: should this strictly require paths to be sub-paths?
Activity
Rollup merge of rust-lang#134696 - ChrisDenton:normalize-lexically, r…
Rollup merge of rust-lang#134696 - ChrisDenton:normalize-lexically, r…
Unrolled build for #134696
ChrisDenton commentedon May 27, 2025
The basic implementation is now implemented in nightly if people would like to experiment.
Rollup merge of rust-lang#134696 - ChrisDenton:normalize-lexically, r…
clarfonthey commentedon Aug 13, 2025
This could be done for
PathBuf, but&mut Pathcannot actually delete anything, since the resulting path has to be the same length. Maybe it could be done in a way similar topartition_dedupwhere all the components are moved to the end, but that would be confusing and aPathBuf-based method sounds much better API-wise.ChrisDenton commentedon Aug 13, 2025
We could have a function both on
PathandPathBuf. The only issue would be choosing names to distinguish them.normalize_lexicallyis already a bit of a mouthful (somewhat intentionally since it shouldn't be what people reach for by default).clarfonthey commentedon Aug 13, 2025
How would you do a function on
Paththat removes components of the path in-place, given what I mentioned?ChrisDenton commentedon Aug 13, 2025
No, I meant have a function on Path that returns a PathBuf and a separate function on PathBuf that does in-place normalisation.
clarfonthey commentedon Aug 13, 2025
Ah, yeah, that makes sense. I was mostly commenting on the note at the end that taking
&mut selfwould only work onPathBuf, meaning that it would be worthwhile to have both.ChrisDenton commentedon Aug 13, 2025
I've updated it to suggest adding a method to
PathBufinstead. If anyone wants to implement that then feel free, I probably won't get to it for awhile.MikkelPaulson commentedon Feb 11, 2026
I'm not sure that it makes sense for this operation to be fallible. Given an input of
"abc/../../def", an output of"../def"seems perfectly reasonable. Even/../etc, which seems like it should be an error, is silently resolved to/etcon Linux.There is absolutely a strong use case for checking if a path escapes the pwd, but
normalize_lexically:a) isn't a one-stop shop for this because it will happily represent absolute paths; and
b) isn't an efficient solution to that problem because it allocates.
With that in mind, I suggest always returning a
PathBufrather thanResult<PathBuf, _>.On that note, does it make sense to return a
Cow<'_, Path>to avoid allocating when no change is made? This is already done for eg.with_trailing_sep().clarfonthey commentedon Feb 12, 2026
I definitely agree that it would be reasonable for this API to return
Cow<'_, Path>, alongside my previous recommendation to add a version that just applies toPathBufin-place.Also in agreement that it would be reasonable for the final result to contain relative paths, assuming that any
..are at the beginning of the path. I mean, after all,abcis a relative path.41 remaining items
Rollup merge of rust-lang#161924 - ChrisDenton:lexically-slash, r=Joh…
Unrolled build for #161924 in rollup 161959
schneems commentedon Sep 10, 2026
I think the limitation of the function (as implied by "lexically") is that it doesn't perform any disk lookups. i.e. it won't ever produce a
std::io::Error. Like callingstd::fs::absolutewould if the call to CWD fails. Without knowing CWD, you cannot produce a meaningful normalization of../since we have no way to know what is being eaten/folded.To the concrete proposal: There are cases where we cannot produce a meaninfully different output than what we're given like
../for those cases I would prefer to hint "nothing changed" or "could not normalize". Versus if we give them an enum, they have to do the mental math to figure out which variants hold normalized outputs and which don't. I think more than a few lazy programmers might just reach for theInto<PathBuf>and completely miss that their input couldn't be normalized. So, I prefer to differentiate them with an Err.Zooming out: I sat with the core of "what exactly is normalization" and wrote some thoughts on a few different ways to think of it, as well as a straw-man proposal. https://gist.github.com/schneems/e0f484bef06ba452199b3eed776203a3.
I'm also now in Zulip, under "Richard Schneeman" if you want to peel off a part of that and have more of a chat.
eirnym commentedon Sep 11, 2026
There was no intention to make this API to touch filesystem at all.
My point is that lexical normalisation is a part of link traversal, but not all of it. And it's better to move this responsibility to the caller as all situations might be different.
E.g.
..in one context could mean "went beyond safety boundary, in other case it's just a path on a filesystem.If there's an intention to have such api, there's a method of
Pathto check if it's a relative path which fully covers the second safety part in their interpretation.The only path which is in a gray area is empty path which may or may not be considered as equivalent of
..schneems commentedon Sep 11, 2026
@eirnym i still guess I don't totally follow. If you want to represent the traversal there's
std::fs::canonicalize.Really, the safety provided by lexical normailzation is for containment is limited as even a normalized path could have a symlink that points to a prior path. I still think it's valuable in an application to reduce/remove
..as it makes reasoning about the logic easier. It reduces the entropy. But it can also cut the other way i.e. a..can "escape" a symlink such that folding it lexically does not produce an equivalent path as a physical traversal. Ruby'sFile.expand_pathboth follows physical links and folds as it goes. I think that's safer but I think there's some subtle dragons there too as you can wind up in situations where the traversal stops due to a problem like a broken symlink.Most languages seem to have a number of slightly different APIs with slightly different behavior. That's why I'm suggesting we try to lock down what "is" and isn't the job of this current feature. That way we can have strong confidence when a PR comes in to accept or reject it. And if someone needs a thing that isn't provided here, it would be a hint to start the process over with a new API with a new stated purpose and goal. I don't think we can have a single API make everyone 100% happy for all their needs.
There's a few ambiguous cases in the existing implementation beyond this one (spelled out some differences in a few of my links). I think it's more useful to define this behavior and declare it produces something other programs can use.
eirnym commentedon Sep 11, 2026
The only think I'd like that lexical normalisation won't think for me in what context I'd like to use it. I'd like that relative paths were valid, even they follow above current folder.
Skgland commentedon Sep 11, 2026
While most of the choices seam reasonable/acceptable I have a few notes.
Absolute Escape
If lexical normalization should return an Err for any path it should be ̀the absolute root escapes, as that can't be check easily before or after.
I.e.
should result in Err.
Relative Escape
For me a relative path can fail to be normalized .. is perfectly fine at the beginning of a normalized path.
So rather than
I.e.
shouldn't result in Err. If someone wants to reject these checking for .. after normalization can be done easily.
Though if you insist that these should Err as long as the Err provides the what you call a partially normalized path I could live with that.
Plain relative
I don't think it is good that we deviate here from all other referenced implementation
Though I don't feel strongly about it as it should be easy to check for so that one can skip normalization in that case.
Paths with Prefix
The examples are missing e.g. windows paths with prefix.
Basically I think it should act as
1. if the path has a prefix remove it
2. perform normalization as described before
3. if 1. removed a prefix add it back
ChrisDenton commentedon Sep 11, 2026
I've started a discussion with the libs team on zulip. I'd like to keep that discussion focused on the broad objective of this method rather than the specific details. Without some agreement on what
normalize_lexicallyis for we're unlikely to make progress.clarfonthey commentedon Sep 13, 2026
Nomination for meeting discussion, with nomination shifting from #159863 to this.
clarfonthey commentedon Sep 22, 2026
So, collecting feedback from libs meeting:
a/b/..is alwaysa./..is just/. (+ for other prefixes as well, noting thatC:..is notC:\)We haven't figured out trailing slashes, yet, and it feels like this might be a place where we want to have extra options, or just not trim the trailing separator and let users do it themselves.
ChrisDenton commentedon Sep 22, 2026
Just to be explicit, the feeling in the libs meeting is that this function should just normalize the path and not do anything else, like restrict it to the root. E.g.
a/../..should be../.As clarafon said, security of paths is a much more complex issue and we can't handle them all in one function. It's also not specific to lexical normalization. This kind of normalization may be one component of securing paths but it may also not be. Even in the case that it is part of securing paths, this one function can't do everything people want without growing a lot of options so it's better for it to do one thing so it can then be composable.
path: Replace `normalize` with component-based normalization in `Path…
path: Replace `normalize` with component-based normalization in `Path…