wasi-http p3: reused instance leaks handles until it traps when a content-length response body is dropped after content-length bytes #14769

A wasi:http/handler@0.3 instance that is reused for many requests leaks handles until it traps, if all three of these are true:

1. the guest's response sets content-length;
2. the host reads exactly content-length bytes of the response body and then drops the body without polling it to the end. hyper does this when it serves a response with content-length, so wasmtime serve-style embeddings will hit it;
3. at least 2 requests are in flight on the instance at once.

If any one of these is false, 300,000 requests run fine on one instance.

The trap is raised while the guest creates the next response, so it looks like resources from earlier responses are never released:

Test Case

Repro: https://github.com/avkonst/wasi-repro (./run.sh runs the failing case and the three passing ones). In summary:

• guest: a wasip3::http_compat component with two routes that differ only in whether the response sets content-length.
• host: one instance in one Store, with up to N calls in flight, each started with handler::Prepared inside Store::run_concurrent. Each response body is either read to the end (full) or read until content-length bytes have arrived and then dropped (until-length, like hyper).

Steps to Reproduce

git clone https://github.com/avkonst/wasi-repro && cd wasi-repro
./run.sh   # rust-toolchain.toml pins rustc 1.99.0 + wasm32-wasip2

# or by hand:
W=guest/target/wasm32-wasip2/release/guest.wasm
H=host/target/release/host

# fails
$H $W /with-content-length    300000 until-length 2 direct
# pass
$H $W /with-content-length    300000 until-length 1 direct
$H $W /with-content-length    300000 full         2 direct
$H $W /without-content-length 300000 until-length 2 direct

Expected Results

All 300,000 requests succeed on one instance, as they do when any one of the three conditions is removed.

Actual Results

instance failed after 99998 requests: error while executing at wasm backtrace:
    0:  0x3ee1d - <unknown>!<wasm function 9>
    1:   0x8ead - guest.wasm!wasip3::service::wasi::http::types::Response::new
    2:   0x53db - guest.wasm!wasip3::http_compat::conversions::http_into_wasi_response
    3:   0x5924 - guest.wasm!wasip3::service::exports::wasi::http::handler::_export_handle_cabi::{{closure}}
    4:  0x112ca - guest.wasm!<futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
    5:  0x14a0c - guest.wasm!wit_bindgen::rt::async_support::spawn::Tasks::poll_next
    6:  0x11876 - guest.wasm!wit_bindgen::rt::async_support::TaskState::callback
    7:  0x116a2 - guest.wasm!wit_bindgen::rt::async_support::callback
    8:   0x55bf - guest.wasm!wit_bindgen::rt::async_support::start_task
    9:   0x7e65 - guest.wasm![async-lift]wasi:http/handler@0.3.0#handle

Caused by:
    resource table has no free keys

instance failed after 25394 requests: error while executing at wasm backtrace:
    0:  0x10262 - guest.wasm!wasip3::http_compat::body_writer::BodyWriter::new
    1:   0x538a - guest.wasm!wasip3::http_compat::conversions::http_into_wasi_response
    ...
Caused by:
    cannot allocate another handle: store handle limit exceeded

Results (/with-content-length unless noted):

The ProxyHandler only starts a concurrent request on an instance whose tasks are blocked on I/O, so each instance sees less overlap. On 49.0.3 that only delays the failure.

Versions and Environment

• Wasmtime version or commit: wasmtime, wasmtime-wasi, wasmtime-wasi-http 50.0.0-rc.1 (component-model-async); also 49.0.3
• Guest: wasip3 0.9.0+wasi-0.3.0 (http-compat), wit-bindgen 0.62.0
• rustc 1.99.0, target wasm32-wasip2
• Operating system: Linux
• Architecture: x86_64

Extra Info

• With content-length, wasmtime-wasi-http consumes the guest body with LimitedGuestBodyConsumer (src/p3/body.rs), not UnlimitedGuestBodyConsumer. GuestBody::size_hint is then exact, which is what lets hyper stop after the declared length and drop the body.
• The guest writes the body and then the trailers from a task that http_into_wasi_response spawns (spawn_local, then BodyWriter::send_http_body). That task, or the stream and future ends it holds, may be what never completes once the host drops the body. If so, it may be partly or wholly a wasip3/wit-bindgen issue and not only a host-side one.
• Workaround attempt on 49.0.3: polling the dropped body to the end on a spawned host task made the trap less frequent but did not stop it.

Guest

The whole guest, for reference (the host is in the repo):

//! A `wasi:http/handler@0.3` component with two routes that differ only in
//! whether the response carries a `content-length` header:
//!
//! - `GET /with-content-length`
//! - `GET /without-content-length`

use bytes::Bytes;
use http::header;
use http_body_util::Full;
use wasip3::http::types::{self, ErrorCode};
use wasip3::http_compat::{http_from_wasi_request, http_into_wasi_response};

wasip3::http::service::export!(Guest);

struct Guest;

impl wasip3::exports::http::handler::Guest for Guest {
    async fn handle(request: types::Request) -> Result<types::Response, ErrorCode> {
        let request = http_from_wasi_request(request)?;
        let body = "hello\n";
        let mut response = http::Response::new(Full::new(Bytes::from_static(body.as_bytes())));
        if request.uri().path() == "/with-content-length" {
            response
                .headers_mut()
                .insert(header::CONTENT_LENGTH, body.len().into());
        }
        http_into_wasi_response(response)
    }
}