Flat write fails with "at most u16::MAX buffers" for Utf8 arrays with about 65,535 data buffers, even after compaction #11135

What happened?

A Utf8 array with 65,535 or more small data buffers cannot be serialized after compression, so writing it fails:

Array and all descendent arrays can have at most u16::MAX buffers: 65536

I expected compaction to copy the values into a few large buffers so the write succeeds. Two things prevent that.

1. should_compact forces compaction only above u16::MAX data buffers (compact.rs#L48). The serializer's limit also counts the views buffer, the validity buffer and any parent encoding's buffers, so 65,535 data buffers already fail.
2. Forced compaction keeps buffers that are well used. At 65,536 fully referenced buffers, compaction runs and keeps every one of them.

The test below references every buffer and fails the same way with #10446 applied.

Steps to reproduce

Add this test to vortex-btrblocks/src/tests.rs and run cargo test -p vortex-btrblocks --lib many_buffers.

/// Each random 16-byte value sits in its own data buffer.
#[rstest]
#[case::below_forced_compaction(65_535)]
#[case::at_forced_compaction(65_536)]
fn test_compressed_strings_with_many_buffers_serialize(
    #[case] nbuffers: usize,
) -> VortexResult<()> {
    let mut ctx = SESSION.create_execution_ctx();
    let mut rng = StdRng::seed_from_u64(0);
    let values: Vec<[u8; 16]> = (0..nbuffers)
        .map(|_| std::array::from_fn(|_| b'!' + (rng.next_u32() % 94) as u8))
        .collect();
    let input = VarBinViewArray::try_new(
        values
            .iter()
            .enumerate()
            .map(|(i, value)| BinaryView::make_view(value, i as u32, 0))
            .collect(),
        values
            .iter()
            .map(|value| ByteBuffer::copy_from(value))
            .collect(),
        DType::Utf8(Nullability::NonNullable),
        Validity::NonNullable,
        &mut ctx,
    )?
    .into_array();

    let compressed = BtrBlocksCompressorBuilder::from_session(&SESSION)
        .build()
        .compress(&input, &mut ctx)?;
    compressed.serialize(
        &ArrayContext::empty(),
        &SESSION,
        &SerializeOptions::default(),
    )?;

    Ok(())
}

It needs these imports on top of the existing ones:

use vortex_array::ArrayContext;
use vortex_array::arrays::varbinview::BinaryView;
use vortex_array::serde::SerializeOptions;
use vortex_buffer::ByteBuffer;

Both cases fail on develop:

running 2 tests
test tests::test_compressed_strings_with_many_buffers_serialize::case_1_below_forced_compaction ... FAILED
test tests::test_compressed_strings_with_many_buffers_serialize::case_2_at_forced_compaction ... FAILED
...
---- case_1_below_forced_compaction stdout ----
Error: Array and all descendent arrays can have at most u16::MAX buffers: 65536
---- case_2_at_forced_compaction stdout ----
Error: Array and all descendent arrays can have at most u16::MAX buffers: 65537

The values are random so that no string scheme compresses them. With compressible values the compressor picks another encoding and the test passes.

Environment

• Vortex version: develop at 246a397e71d13467aebe4b5d40ca277ac9f3734b and feat(array): name three levels of canonical compaction and use them in the compressor #10446 at 223e4f5511dce14b266f9aac01c61913cfb1f9e3
• Python/Java version: n/a, Rust from each revision's rust-toolchain.toml
• OS: macOS 26.7, arm64

Additional context

The default write strategy splits arrays into chunks before serializing, so most inputs stay below the limit.

To fix this, the check could count every buffer the serializer counts. Above the limit, it would copy the referenced bytes into new buffers however full they are. Arrays below the limit would not change. I don't know which layer should own the check. should_compact only sees the VarBinViewArray, not the buffers of a parent encoding. The serializer sees the whole tree, but by then the strings can be inside a compressed child. I can send a PR for whichever you prefer.

Related: #5166 added the forced compaction, #5162 turned the panic into this error, and #10403 is another way string arrays keep too many buffers.

I used Claude Code to investigate this and write the test. I reviewed the code and ran it at each revision above.