[BUG] `fd` cannot handle arbitrary deep directories like GNU `find`

[collinfunk]

Checks

• I have read the troubleshooting section and still think this is a bug.

Describe the bug you encountered:

One of the nice features of GNU find and many other GNU programs is that they avoid arbitrary limits whenever possible. As a result, GNU mkdir can create arbitrarily deep directories which find can then visit:

$ mkdir -p $(yes a/ | head -n  32768 | tr -d '\n')
$ find a | wc -l
32768

This is not the case with fd:

$ ./target/release/fd a | wc -l
2062

Using strace, we can see that fd constructs the file names in full each time and calls statx(2) instead of operating on file descriptors. This means fd reaches PATH_MAX limitations:

$ strace -ff -e trace=statx ./target/release/fd -j 1 '' a  2>&1 | grep '= -1'
[...]
[pid 344540] statx(AT_FDCWD, "[long file name trimmed]"..., AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fdb3c3fbcf0) = -1 ENAMETOOLONG (File name too long)
[...]

It also gives a misleading non-zero exit status after failing to list all of the file names:

$ ./target/release/fd -j 1 '' a > /dev/null
$ echo $?
0

Describe what you expected to happen:

No response

What version of fd are you using?

fd 10.4.2

Which operating system / distribution are you on?

$ uname -srm
Linux 6.18.4-200.fc43.x86_64 x86_64
$ lsb_release -a
LSB Version:	n/a
Distributor ID:	Fedora
Description:	Fedora Linux 43 (Workstation Edition)
Release:	43
Codename:	n/a

[tmccombs]

This would be good to address, but it may be quite difficult to do so, since we currently use walkdir for traversing the filesystem (via the ignore crate), which can't currently handle long paths.

Possibly there is a way to work around that, but it probably woudn't be easy, and may add a lot of complexity.

[KSHITIZ6341]

I looked into this in fd and the issue has two parts:

1. fd currently exits successfully even when the traversal reports filesystem errors such as ENAMETOOLONG.
2. The deeper GNU find parity issue needs descriptor-relative traversal, because the current ignore/walkdir stack still opens directories by full path.

I have a small fd-side patch for (1): traversal errors are now tracked and cause a non-zero exit status, including for normal output, --exec, and --exec-batch, while --show-errors remains display-only.

A full fix for (2) likely needs upstream work in walkdir/ignore using APIs like openat/fstatat instead of path-based read_dir. This seems related to BurntSushi/walkdir#213.

[KSHITIZ6341]

I have a working local prototype for this across the dependency stack.

The fix needs to happen mostly below fd, because the failing path is in traversal:

1. walkdir: use descriptor/capability-relative traversal instead of repeatedly constructing full paths for descendant stats/reads.
2. ignore: update the fd-facing WalkParallel path to use the same long-path-safe traversal, including deep ignore-file loading.
3. fd: bump to the fixed ignore release and keep an exit-status fix so traversal errors do not silently return 0.

What I verified locally:

• fd built against the patched local ignore successfully found a sentinel file whose absolute path was beyond PATH_MAX.
• Deep directory traversal works past PATH_MAX.
• Deep .ignore / fd ignore loading works.
• --follow works for deep symlinked directories.
• --one-file-system uses cached/descriptor-derived metadata instead of restatting long ambient paths.
• Unfollowed deep symlinks correctly report as symlinks in the covered long-path case.
• Traversal errors now make fd return nonzero instead of silently succeeding.

Test coverage I added locally includes long-path regressions in walkdir and ignore, plus an fd regression for traversal errors affecting exit status.

One upstream wrinkle: my prototype uses cap-std, whose dependency stack appears to require a newer Rust version than walkdir currently advertises. So the walkdir PR either needs an MSRV bump discussion or a lower-level implementation with the current MSRV.

[tmccombs]

FWIW, it might be best if walkdir and ignore included a reference to the directory handle, so that users don't need to re-traverse the path to open the file. Although, that may be problematic from a lifetime perspective, and might require a breaking change.

[KSHITIZ6341]

That makes sense to me. Exposing a directory handle/capability from walkdir/ignore would avoid forcing downstream callers to reopen long ambient paths. I agree the API shape is the hard part because of lifetimes and backwards compatibility. For fd, I think the immediate fd-side change should stay limited to reporting traversal failures with a non-zero exit status; the long-path traversal work should happen in walkdir/ignore first.