Expose per-field on-disk byte-range layout from codec readers

[abernardi597]

Expose per-field on-disk byte-range layout from codec readers

Lucene readers know a great deal about how bytes are laid out on disk — which byte ranges of a segment file belong to which field — but that information is not observable today. There is no general way to ask "how many bytes of file X does field Y occupy?" across postings, doc values, KNN vectors, points, etc.

This makes it hard to answer operationally important questions:

• Which fields dominate a segment's on-disk footprint?
• Given a memory-residency signal (e.g. mincore(2)/cachestat(2)), which fields are "cold" (least resident in the page cache) and therefore candidates for cost or latency attention?
• How is a field's storage split across the formats that store it (postings vs. doc values vs. points)?

Today the only ways to get at this are indirect and fragile: summing whole-file sizes (no per-field breakdown), or reflecting into codec internals to recover per-field file pointers (brittle, breaks on every codec change).

Precedent

KnnVectorsReader#getOffHeapByteSize(FieldInfo) (#14426) already established that a reader can report its per-field on-disk size, and that this is useful enough to live on the reader API. That method returns a per-field, per-extension total. This proposal generalizes the same idea in two ways:

1. To any format (not just KNN vectors)
2. From a per-field total to a per-field byte-range layout so a consumer can correlate ranges with offset-addressed signals like page-cache residency, read-access histograms, etc.

Proposed shape

The Data: an Extent and its Attribution

A contiguous byte range, and what it belongs to:

/** A contiguous byte range within a file, and a descriptor of its purpose. */
record Extent(long offset, long length, Attribution attribution) {}

/** What the bytes in an Extent belong to. */
sealed interface Attribution {
  /** Bytes belonging to a single field. */
  record Field(String name) implements Attribution {}
  /** Codec-declared overhead: header, footer, padding, or other field-independent metadata. */
  record Structural()       implements Attribution {}
  /** Bytes no reader described: a gap within a file, or a file no reader reports. */
  record Unclaimed()        implements Attribution {}
}

Modeling attribution as a sealed type (rather than a nullable field name or a set) distinguishes the cases that consumers genuinely need to treat differently (attributed, structural overhead, and not-yet-described) while leaving room to add other kinds later (e.g. a range shared by several fields) without silently breaking existing consumers, who get a compile error until they handle the new case.

The API

public interface DiskAccountable {
  /**
   * On-disk layout of the files this component manages, keyed by file name. Each value is
   * a list of non-overlapping Extents sorted by offset that together cover the whole file
   * [0, fileLength): ranges no reader describes appear as Attribution.Unclaimed rather than
   * being omitted, so every byte is accounted for exactly once. Returns an empty map by default.
   */
  default Map<String, List<Extent>> getDiskLayout() throws IOException {
    return Map.of();
  }
}

The natural home is per-segment, and the file key is built up as the layout bubbles up through the reader tiers — each tier contributing the part of the name it knows:

1. Leaf codec readers (FieldsProducer, DocValuesProducer, KnnVectorsReader, PointsReader, ...) describe the files they manage. A reader has its extension and its segmentSuffix (from SegmentReadState) but not the segment name, so it can name a file by everything except that prefix — e.g. a flat vectors reader's .vec.

2. Per-field formats (PerFieldKnnVectorsFormat, etc.) delegate to one inner reader per format+instance, each constructed with a distinguishing segmentSuffix (Lucene99_0, Lucene99_1, ...). That suffix is what separates the multiple physical files a per-field format produces — _Lucene99_0.vec vs _Lucene99_1.vec.

3. CodecReader merges its child readers' layouts, and SegmentReader — the only tier that knows the segment name and holds the segment's file list — completes each key into a full file name (_5_Lucene99_0.vec) and reconciles the merged result against SegmentCommitInfo#files(): any file no child reader described is added as a whole-file Unclaimed extent. The segment-level result is therefore keyed by globally-unique full file names and covers every file in the segment — including formats that do not (yet) implement DiskAccountable, which simply surface as Unclaimed and can be attributed incrementally as readers adopt the interface.

A consumer walks a SegmentReader's layout, correlates each Extent with whatever offset-addressed signal it cares about, and aggregates per Attribution.

Open questions for discussion

1. Cost. Building a per-file Extent graph for every field of every segment on every call has a GC/CPU cost that grows with field count (which can be high). Should the base API lean toward a visitor/callback style (accumulate into a caller-supplied sink, allocating nothing) rather than materializing an actual Map<String, List<Extent>>? A segment's layout should not change after it is written, so caching by segment core identity (LeafReader#getCoreCacheHelper, which is stable across new deletions) is the intended mitigation, but the base API shape still matters. Interested in preferences here.

2. Attribution kinds. Field / Structural / Unclaimed covers the basics. Is there appetite for modeling a range shared by multiple fields (e.g. deduplicated vectors), or is that better left out until a concrete need exists? Sharing also breaks the one-field-one-contiguous-run intuition — a shared range belongs partly to several fields at once — so any per-field total over shared ranges needs an explicit split policy (equal? by reference count?) rather than a simple sum.

3. Which readers. A useful first cut covers postings, doc values, KNN vectors, and points (BKD). Stored fields and term vectors are block-compressed with interleaved fields and are much harder to attribute per-field; they would surface as Unclaimed until/unless someone tackles them.

4. Full coverage vs. best-effort. This sketch proposes the segment-level result cover [0, fileLength) for every file (gaps and unreported files become Unclaimed), which makes totals reconcile against the directory's file sizes. The alternative — allow gaps, omit unreported files — is cheaper but leaves the caller to reconcile. Preference? Reconciliation is a diff against SegmentCommitInfo#files() in both directions: a file in files() that no reader described becomes a whole-file Unclaimed (the common case). The reverse — a reader reporting a file not in files() — would indicate the reader's view and the commit's file list disagree; the sketch treats that as a bug to surface (fail fast) rather than silently drop, but the API contract for it is worth pinning down.

5. Keying. Lucene already models a file name as (segmentName, segmentSuffix, ext) and constructs it with IndexFileNames#segmentFileName(...); a reader has its segmentSuffix and ext at construction (via SegmentReadState) but not the segment name. So rather than have readers emit an opaque suffix string that a higher tier concatenates, the layout key could align with that existing model — readers key by segmentSuffix/ext, and SegmentReader completes the name with the canonical segmentFileName(...) (the same call the write path used, so it round-trips against files() exactly). This avoids re-implementing name construction and its edge cases. Worth deciding what the key type should be (structured vs. string) at the interface level.

6. Compound files. Under CompoundFormat, child readers describe logical files (.doc, .dvd, ...) at offsets within those logical files, but on disk there is only the physical .cfs (plus .cfe). A raw merge would therefore key by logical files that do not physically exist. SegmentReader is the tier that could reconcile this: it knows whether the segment is compound and holds the compound Directory, which maps each logical file to its offset within .cfs. Options range from translating logical extents into .cfs-relative ones (preserves whole-byte accounting, but needs the compound directory to expose per-entry offsets), to reporting the .cfs as a single Unclaimed file (no per-field breakdown inside compound segments), to leaving CFS out of scope initially. This likely needs input from folks who own CompoundFormat.

Prior art / relationship to existing APIs

• getOffHeapByteSize (Add support for determining off-heap memory requirements for KnnVectorsReader #14426): per-field, per-extension totals for KNN vectors. This is the range-level generalization, across all formats.
• Accountable/ramBytesUsed: RAM, not disk; whole-component, not per-field.
• CheckIndex / SegmentInfos#files(): whole-file sizes, no per-field breakdown.

Happy to put up a draft PR implementing this across the readers above plus the CodecReader/SegmentReader composition, if there's interest in the direction.

[mikemccand]

[Disclaimer: I work with @abernardi597 at Amazon (Customer facing product search, built 100% on Lucene now) where we struggle to understand hot/cold bytes per field in our production Lucene views. I asked @abernardi597 to propose possible solutions here. ]

I like that this proposal is "zero entry", entirely optional for a new Codec component to implement (i.e. just default to Unclaimed), and that we could incrementally build it (implement component by component), phasing in over time. Not burdening Codec authors is crucial -- that has been an objection in past proposals for this feature.

I also like that it's simpler than the in-RAM Accountable API (no nesting/recursion needed).

without silently breaking existing consumers, who get a compile error until they handle the new case.

+1, but let's remember to mark the API @lucene.experimental at first so we reserve the option to make such breaks as we iterate even within new dot (feature) releases (10.6, 10.7, 10.8, ...).

Given a memory-residency signal (e.g. mincore(2)/cachestat(2)), which fields are "cold" (least resident in the page cache) and therefore candidates for cost or latency attention?

+1, this would be really helpful for understanding/debugging/benchmarking cold/warm indices! It's so hard to see this today -- we are flying blind. I can't wait for first light of this metric 🔥

The reverse — a reader reporting a file not in files() — would indicate the reader's view and the commit's file list disagree; the sketch treats that as a bug to surface (fail fast) rather than silently drop, but the API contract for it is worth pinning down.

+1, this really would be a bad bug (IndexWriter's whole deletion infra relies on Codecs being precise about their file names), so it's good to be brittle.

A segment's layout should not change after it is written,

+1 -- it cannot -- Lucene is write-once, unless we try to cover updatable doc-values fields or deletions, which write new generations (new files each time, with an _N that increases) associated with each core segment. I think we shouldn't model those for starters?

a reader has its segmentSuffix and ext at construction (via SegmentReadState) but not the segment name.

I'm confused -- segment name is always available in SegmentReadState, at the lowest level (under SegmentInfo.name). Each low level reader uses that to construct the file names it then opens. I think we should keep things simple and key by the String fileName (what the Codec component passes to Directory.openInput)?

Compound files

Another option is to deprecate/remove CFS files -- I think I had opened an issue for discussion long ago but cannot find it. CFS files are broken for other things too (like MMapDirectory#setPreload), and maybe madvise/prefetch? I would say for this proposal we leave CFS as Unclaimed (your 2nd option) for starters?

The natural home is per-segment

+1, this is a per-segment thing, but what does "home" mean? Is it where this API is exposed? I don't think Lucene should cache these extents by default -- users can do that well (tracks per-segment reader lifecycle well) with the CoreCacheHelper.

[abernardi597]

I'm confused -- segment name is always available in SegmentReadState, at the lowest level (under SegmentInfo.name). Each low level reader uses that to construct the file names it then opens. I think we should keep things simple and key by the String fileName (what the Codec component passes to Directory.openInput)?

In the spirit of keeping the changes minimal I did not want to force codecs to maintain references to the SegmentInfo/file names that otherwise seem to be referenced only during construction. To that extent, I had envisioned leaf readers just describing via file extension (vec, vex, pos, etc), then the wrapping readers prepending information accordingly (e.g. per-field codecs add to the file name). Finally SegmentReader would be able to complete the file names with the SegmentInfo it already references.

this is a per-segment thing, but what does "home" mean? Is it where this API is exposed?

Yes, I mean where the API is exposed at the top level. You could argue for making IndexReader itself DiskAccountable too (aggregating over its leaves), but as you mentioned, consumers likely want to cache per-segment layouts using CoreCacheHelper. At the IndexReader level the segment geometry would be lost unless it caches internally itself, which is arguably too invasive.

[mikemccand]

I'm confused -- segment name is always available in SegmentReadState, at the lowest level (under SegmentInfo.name). Each low level reader uses that to construct the file names it then opens. I think we should keep things simple and key by the String fileName (what the Codec component passes to Directory.openInput)?

In the spirit of keeping the changes minimal I did not want to force codecs to maintain references to the SegmentInfo/file names that otherwise seem to be referenced only during construction. To that extent, I had envisioned leaf readers just describing via file extension (vec, vex, pos, etc), then the wrapping readers prepending information accordingly (e.g. per-field codecs add to the file name). Finally SegmentReader would be able to complete the file names with the SegmentInfo it already references.

It's fine to expect Codec readers to know the filenames they opened: the whole purpose of each Codec reader is to read just the one segment, andSegmentReadState already carefully provides all necessary context so each reader can easily compute its exact filenames (setting the right suffix for per-field embedded readers). Let's not re-invent that same mechanism in reverse (going back up the call stack).

this is a per-segment thing, but what does "home" mean? Is it where this API is exposed?

Yes, I mean where the API is exposed at the top level. You could argue for making IndexReader itself DiskAccountable too (aggregating over its leaves), but as you mentioned, consumers likely want to cache per-segment layouts using CoreCacheHelper. At the IndexReader level the segment geometry would be lost unless it caches internally itself, which is arguably too invasive.

Maybe at least we could here provide the sugar for caching? It'd be a small wrapper class, using Lucene's CoreCacheHelper so things are evicted when the leaf is finally closed.

Is there appetite for modeling a range shared by multiple fields (e.g. deduplicated vectors), or is that better left out until a concrete need exists?

+1 to design for today (postpone appetite for a future apple pie). Let's keep it simple --- phase one with no CFS, single field per extent -- that's hard enough! We already see that per-field accounting for stored fields isn't really possible. This new mechanism cannot be perfect. It's sort of like Gödel's incompleteness theorems heh. Let's target incompleteness rather than inconsistent.

Should the base API lean toward a visitor/callback style (accumulate into a caller-supplied sink, allocating nothing) rather than materializing an actual Map<String, List>?

This is an interesting question. If the underlying API is a visitor (no added GC load), we could implement the full List<Extent> as a wrapper/layer overlay on top. Visitor model might more naturally be extended over time too, e.g. we could add method for multiple fields per extent and by default fall back to single field for back compat. And, we could provide sugar for basic use cases e.g. a visitor that tallies up total disk usage by field. That would work well I think.

Now I'm also wondering why we need "structural" and "unaccounted" types as well -- why insist the entire file is accounted for? If the default impl will fill in ranges for "unaccounted", it seems pointless since caller can equally well see such extents as unaccounted.

[mccullocht]

I am also in general supportive of this, it could be helpful for debugging size issues, and the residency components could also be used if you abstract away the storage layer.

I would prefer a visitor API as well to avoid generating a lot of objects, especially because you can build APIs that are simpler to interact with on top of it.

I agree that we shouldn't force implementations to produce an extent for every byte of the file -- there's going to be some format overhead that isn't really accountable to any field, and callers should be able to measure this by comparing the byte size of the index to the sum of computed extents.