Keepalive PING after graceful GOAWAY disconnects draining RPCs

What version of gRPC and what language are you using?

@grpc/grpc-js 1.14.4 on Node.js 24.15.0. The relevant path is also unchanged on the current master branch.

What operating system are you using?

Linux.

What did you do?

Send a graceful HTTP/2 GOAWAY while a unary RPC is still running, then allow the RPC to finish after the client's keepalive interval.

This is closely related to #2624, but the current failure does not depend on an immediate server shutdown. A graceful GOAWAY by itself leaves the keepalive timer running. The next PING fails, maybeSendPing() calls handleDisconnect(), and grpc-js terminates the accepted in-flight RPC.

goaway-server.js:

const http2 = require('node:http2');

function frameMessage(payload) {
  const frame = Buffer.allocUnsafe(5 + payload.length);
  frame.writeUInt8(0, 0);
  frame.writeUInt32BE(payload.length, 1);
  payload.copy(frame, 5);
  return frame;
}

const server = http2.createServer();
server.on('stream', stream => {
  stream.respond(
    { ':status': 200, 'content-type': 'application/grpc' },
    { waitForTrailers: true }
  );
  stream.on('wantTrailers', () => {
    stream.sendTrailers({ 'grpc-status': '0' });
  });

  setTimeout(() => {
    stream.session.goaway(http2.constants.NGHTTP2_NO_ERROR, stream.id);
  }, 10);
  setTimeout(() => {
    stream.end(frameMessage(Buffer.from('response')));
  }, 20_000);
});

server.listen(50051, '127.0.0.1');

client.js:

const grpc = require('@grpc/grpc-js');

const client = new grpc.Client(
  '127.0.0.1:50051',
  grpc.credentials.createInsecure(),
  {
    'grpc.keepalive_time_ms': 15_000,
    'grpc.keepalive_timeout_ms': 30_000,
    'grpc.keepalive_permit_without_calls': 1
  }
);
const startedAt = Date.now();

client.makeUnaryRequest(
  '/repro.Service/Slow',
  value => value,
  value => value,
  Buffer.from('request'),
  { deadline: Date.now() + 120_000 },
  (error, response) => {
    console.log({
      elapsedMs: Date.now() - startedAt,
      code: error?.code,
      details: error?.details,
      response: response?.toString()
    });
    client.close();
  }
);

Run node goaway-server.js, then node client.js.

The same reproduction can be sped up deterministically by using a 100 ms keepalive and a 500 ms response delay.

What did you expect to see?

The accepted RPC finishes normally after the graceful GOAWAY and returns response at about 20 seconds. The transport is already draining and will not accept a new call, so it should not send more keepalive PINGs.

What did you see instead?

Stock 1.14.4 fails at the keepalive boundary (15,022 ms in the original reproduction) with status 14 / Connection dropped. In the scaled reproduction, stock 1.14.4 fails at about 114 ms; a transport that stops keepalive on GOAWAY returns the response at about 524 ms.

There is a second race when a PING is already in flight as GOAWAY arrives: a later PING cancellation callback can still call handleDisconnect() unless the callback also recognizes the draining state.

Suggested fix

When the session emits goaway:

Mark the transport as draining.
Clear the keepalive timeout.
Make canSendPing() / maybeSendPing() reject future PINGs.
Ignore completion of a PING that was already in flight once the transport is draining.

The existing socket close and error handlers still report genuine disconnects, and RPC deadlines continue to bound hung calls. This only stops probing a connection whose peer explicitly announced graceful draining.

I believe the server is actually behaving incorrectly in this case. The section of the HTTP/2 spec about the PING frame says

Receivers of a PING frame that does not include an ACK flag MUST send a PING frame with the ACK flag set in response, with an identical frame payload.

I can't find any indication that the sender of a GOAWAY frame should behave any differently during the graceful shutdown period.

You're right that a peer must ACK a PING during graceful shutdown, and Node does this: a raw-frame probe confirms a Node server ACKs PINGs received while draining.

The PING in this failure never reaches the server. Node marks the client session closed on a NO_ERROR GOAWAY (onGoawayData calls session.close()), and ping() on a closed session short-circuits in lib/internal/http2/core.js:

if (this.connecting || this.closed) {
   process.nextTick(cb, false, 0.0, payload);
   return;
}

The callback fires with ERR_HTTP2_PING_CANCEL and grpc-js treats both the falsy return and the callback error as connection failures, dropping the in-flight RPC.

PR #3089 stops keepalive PINGs on a closing session on the client and both server handler variants. One regression test fails on unchanged upstream, with only the client fix, and with only the server fix; it passes with both fixes.

With 15-second keepalive the original code fails at ~15s; the fix returns the response at ~20s. Reproduction scripts and traces: https://gist.github.com/gfrancz/e09c65c05ab90dba36f01c2411e3aaf7

We hit a server-only variant of this in production. The trigger is the server's own grpc.max_connection_age_ms: no client keepalive and no peer behaving unusually.

When the connection reaches its maximum age, the server sends GOAWAY and calls session.close(), but its keepalive timer keeps running. The next server ping on the closed session is cancelled locally by Node: ping() returns undefined, and the callback gets ERR_HTTP2_PING_CANCEL. The server then calls session.destroy(), which kills all calls still in flight.

As a result, grpc.max_connection_age_grace_ms is effectively capped at grpc.keepalive_time_ms. With our settings (max age 40 min, grace 10 min, keepalive 60 s), long calls were reset at most 60 s after the GOAWAY instead of after 10 minutes.

Reproduction (only @grpc/grpc-js, no proto file):

const grpc = require('@grpc/grpc-js');

const KEEPALIVE = process.env.KEEPALIVE !== '0';

const service = {
  slow: {
    path: '/test.Test/Slow',
    requestStream: false,
    responseStream: false,
    requestSerialize: (v) => v,
    requestDeserialize: (v) => v,
    responseSerialize: (v) => v,
    responseDeserialize: (v) => v,
  },
};

const start = Date.now();
const log = (msg) => console.log(`${((Date.now() - start) / 1000).toFixed(2).padStart(6)}s  ${msg}`);

const server = new grpc.Server({
  'grpc.max_connection_age_ms': 2000,
  'grpc.max_connection_age_grace_ms': 60000,
  ...(KEEPALIVE ? {'grpc.keepalive_time_ms': 1000, 'grpc.keepalive_timeout_ms': 20000} : {}),
});

server.addService(service, {
  slow: (call, callback) => {
    log('server: call started, responding in 5 s');
    setTimeout(() => callback(null, Buffer.from('done')), 5000);
  },
});

server.bindAsync('127.0.0.1:0', grpc.ServerCredentials.createInsecure(), (err, port) => {
  if (err) throw err;
  log(`server: keepalive ${KEEPALIVE ? 'on (1 s)' : 'off'}, max age 2 s, grace 60 s`);
  const Client = grpc.makeGenericClientConstructor(service);
  const client = new Client(`127.0.0.1:${port}`, grpc.credentials.createInsecure());
  client.slow(Buffer.alloc(0), (err, res) => {
    if (err) log(`client: FAILED with ${grpc.status[err.code]}: ${err.details}`);
    else log(`client: OK (${res.toString()})`);
    client.close();
    server.forceShutdown();
  });
});

npm install @grpc/grpc-js@1.14.6, then GRPC_TRACE=keepalive GRPC_VERBOSITY=DEBUG node repro.js:

  0.01s  server: keepalive on (1 s), max age 2 s, grace 60 s
  0.02s  server: call started, responding in 5 s
D 2026-10-16T14:15:54.515Z | v1.14.6 224479 | keepalive | (1) Ping send failed: Ping returned false
D 2026-10-16T14:15:54.515Z | v1.14.6 224479 | keepalive | (1) Ping failed with error: HTTP2 ping cancelled
  2.02s  client: FAILED with UNAVAILABLE: Connection dropped

Expected: the call finishes within the 60 s grace period. With KEEPALIVE=0 node repro.js, it does:

  0.02s  server: call started, responding in 5 s
  5.02s  client: OK (done)

With the server change from #3089 (!session.closed in canSendPing()) applied to 1.14.6, the repro also passes (client: OK after 5 s).

Environment: Linux x86_64 (WSL2), Node 24.21.0 (production: Node 24.13.1), @grpc/grpc-js 1.14.6 (production: 1.14.3).

@gfrancz

We reproduced this with grpc-js on both sides, so no third-party server is involved: @grpc/grpc-js 1.14.6 server and client, Node v24.13.0.

Setup: server grpc.max_connection_age_ms: 1000, grpc.max_connection_age_grace_ms: 15000; client grpc.keepalive_time_ms: 2000, grpc.keepalive_timeout_ms: 5000, grpc.keepalive_permit_without_calls: 1; one unary call whose handler responds after 10 s.

Client trace (GRPC_TRACE=transport,keepalive GRPC_VERBOSITY=DEBUG):

transport | connection closed by GOAWAY with code 0 and data max_age
keepalive | Sending ping with timeout 5000ms        (~1.05 s after the GOAWAY)
keepalive | Ping send failed: Ping returned false
keepalive | Ping failed with error HTTP2 ping cancelled

The call ends with code=14 Connection dropped about 1 s after the GOAWAY, 13 s before the server's grace period would have ended. With keepalive disabled, the same call survives until the grace period ends, as expected.

This matches @gfrancz's analysis above: no PING reaches the server. In Node 24, Http2Session.prototype.ping() returns early when this.closed is true and schedules the callback with ERR_HTTP2_PING_CANCEL instead of writing a frame. The client session is closed because grpc-js calls session.close() when it handles the GOAWAY (Http2Transport.shutdown()), while the keepalive timer keeps running and canSendPing() only checks session.destroyed. So the failure is local to the client and does not depend on how the server answers PINGs.

We also have a production failure consistent with this path, though our production logs do not include grpc-js traces to confirm it: an in-flight unary write (~1.4 s) failed with UNAVAILABLE: Connection dropped while the server pod kept running and completed the write 1 s later.

#3089 addresses this path. We would welcome a review of it.

@murgatroid99

Hi @murgatroid99, I've also reproduced the above issue and tested the proposed fix. With the 2 previous comments in mind and the comments on the PR itself, what will it take to get #3089 approved and merged?

Navigation Menu

Repository navigation

Keepalive PING after graceful GOAWAY disconnects draining RPCs #3068

What version of gRPC and what language are you using?

What operating system are you using?

What did you do?

What did you expect to see?

What did you see instead?

Suggested fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Keepalive PING after graceful GOAWAY disconnects draining RPCs #3068

Description

What version of gRPC and what language are you using?

What operating system are you using?

What did you do?

What did you expect to see?

What did you see instead?

Suggested fix

Activity

murgatroid99 commented on Aug 17, 2026

gfrancz commented on Sep 21, 2026

n0v1 commented on Oct 16, 2026

azyu commented on Oct 19, 2026

ws-gregm commented on Oct 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions