Web forwarding drops HTTP/2 response trailers, which breaks gRPC (HTTP/2 itself now negotiates fine) #687

Hi, and apologies for reopening a settled topic. I know HTTP/2 support on *.devtunnels.ms has
been raised before, in #263, and again in #463 and #481 which were closed as duplicates of it. #263
was closed as not planned in February 2024 with the note "At this point, HTTP/2 on *.devtunnels.ms
is not planned", and I could not find a rationale recorded anywhere, so I am not sure which part of
it was the expensive part.

I am raising this again because I think the remaining gap is materially smaller than it was when
those issues were filed, and I want to put fresh measurements in front of you rather than just
re-asking.

What changed since #263

#263 reported that an HTTP/2 connection could not be established at all:

Requesting HTTP version 2.0 with version policy RequestVersionOrHigher while unable to establish
HTTP/2 connection.

That no longer reproduces. Today a tunnelled request negotiates HTTP/2 successfully. I tested with
HttpVersionPolicy.RequestVersionExact, which throws rather than silently downgrading, and the
request completed over HTTP/2 with a 200 response.

So HTTP/2 to the edge appears to work. The one remaining thing gRPC needs is that response
trailers are forwarded, and currently they are not.

Why that single gap breaks gRPC

A gRPC unary call returns its status in the grpc-status trailer whenever the response has a body.
When there is no body, the status can ride in the headers instead. So:

• Every gRPC error reply has an empty body, puts grpc-status in the headers, and travels
  through a tunnel perfectly.
• Every gRPC success reply has a body, puts grpc-status in the trailers, and loses its status.

The client then sees a 200 with a payload and no status, and reports:

Grpc.Core.RpcException: Status(StatusCode="Cancelled", Detail="No grpc-status found on response.")

This is an unpleasant failure mode to debug, because everything that goes wrong works, and only
the calls that succeed fail. I spent a while convinced the tunnel was healthy because every error
status I could provoke relayed correctly.

Minimal repro

A plain Kestrel HTTP/2 server that answers in the shape of a gRPC unary reply: a body, then
grpc-status as a trailer.

using Microsoft.AspNetCore.Server.Kestrel.Core;

var builder = WebApplication.CreateBuilder(args);
builder.WebHost.ConfigureKestrel(o =>
{
    o.ListenAnyIP(3979, l =>
    {
        l.Protocols = HttpProtocols.Http2;
        l.UseHttps();
    });
});

var app = builder.Build();

// Body, then grpc-status in the TRAILERS. This is what a successful gRPC call looks like.
app.MapPost("/trailer/{size:int}", async (HttpContext ctx, int size) =>
{
    ctx.Response.StatusCode = 200;
    ctx.Response.ContentType = "application/grpc";
    ctx.Response.DeclareTrailer("grpc-status");

    var frame = new byte[5 + size];
    frame[0] = 0;
    frame[1] = (byte)((size >> 24) & 0xFF);
    frame[2] = (byte)((size >> 16) & 0xFF);
    frame[3] = (byte)((size >> 8) & 0xFF);
    frame[4] = (byte)(size & 0xFF);
    for (var i = 0; i < size; i++) frame[5 + i] = (byte)(i % 251);

    await ctx.Response.Body.WriteAsync(frame);
    await ctx.Response.Body.FlushAsync();

    ctx.Response.AppendTrailer("grpc-status", "0");
    ctx.Response.AppendTrailer("grpc-message", "OK");
});

// Control: no body at all, so grpc-status can ride in the headers.
app.MapPost("/nobody", (HttpContext ctx) =>
{
    ctx.Response.StatusCode = 200;
    ctx.Response.ContentType = "application/grpc";
    ctx.Response.Headers["grpc-status"] = "0";
    return Task.CompletedTask;
});

app.Run();

Host port 3979 as https, then call it directly and through the tunnel with an HTTP/2 client that
reads TrailingHeaders.

Results

The body arrives intact in every case, including at 14 KB. Only the trailers are lost. I also
confirmed separately that request bodies up to 256 KB and request headers up to 9 KB relay fine, so
this does look specific to trailer propagation rather than to size or framing generally.

Our scenario, and why the documented workaround does not fit

We run a gRPC service locally so we can attach a debugger to it, while the service that calls it
runs in a managed cloud environment. We want the cloud caller to reach the locally hosted gRPC
service for a single test account, which is exactly the "debug your web applications from anywhere"
case dev tunnels is built for, except that the protocol is gRPC.

devtunnel connect TUNNELID plus localhost:PORT is suggested in #263, and it does work, because
the SDK forwards over SSH forwarded-tcpip channels and never parses HTTP. But it requires the dev
tunnel CLI to run on the calling machine, and in our case the caller is a container in a managed
environment we do not control. So that path is not available to us.

The alternatives we looked at:

• Third-party tunnelling providers that do support end to end HTTP/2. Our organization's policy does
  not permit routing this traffic through an external provider, so these are not an option for us.
• gRPC-Web, which moves the trailers into the body. That needs a protocol change on both ends, and
  we only own one of them.
• Hosting the service in a shared environment instead of locally. That works, but it gives up the
  local debugger, which is the entire reason we want this.

That leaves us with no viable path, which is why I am asking rather than quietly working around it.

Suggestions

I appreciate I am on the outside of the implementation here, so please take these as questions
rather than proposals.

1. Forward response trailers through web forwarding. If the edge already speaks HTTP/2 to the
   client, propagating the trailer frame may be a contained change. If the internal hop to the host
   is HTTP/1.1, the Trailer header plus chunked trailing headers could carry them across.
2. An opt-in passthrough port mode. A port flag that makes the hosted URL forward the TLS stream
   opaquely, with no HTTP inspection, would support gRPC and anything else HTTP/2 based. I realise
   this likely conflicts with per-port rate limiting, the forwarding cookie, and abuse protection,
   which all need request visibility, so it may only make sense for authenticated, non-anonymous
   tunnels.
3. Failing both, document the limitation explicitly. A line in the docs saying gRPC and HTTP/2
   trailers are not supported on *.devtunnels.ms, with devtunnel connect named as the supported
   path, would have saved me a day. The current behaviour looks like success, so there is nothing to
   search for until you know to look at trailers specifically.

If trailer forwarding is feasible, it would unblock gRPC entirely for us, and I suspect for the
people on the three earlier issues too. We would really appreciate you reconsidering it.

Happy to test any preview build against our setup and report back.