Skip to content

PickFirstLoadBalancer.updateAddressList silently ignores resolver errors while in TRANSIENT_FAILURE with existing subchannels #3116

Closed as not planned

Description

@hidingmyname

Problem description

Location (grpc/grpc-node @ default branch 6741e13): packages/grpc-js/src/load-balancer-pick-first.ts:520

On latest HEAD, the resolver-error branch of updateAddressList only reports a new picker when there are no children at all:

// In load-balancer-pick-first.ts, updateAddressList:
if (!maybeEndpointList.ok) {
  if (this.children.length === 0 && this.currentPick === null) {
    this.channelControlHelper.updateState(
      ConnectivityState.TRANSIENT_FAILURE,
      new UnavailablePicker(maybeEndpointList.error),
      maybeEndpointList.error.details
    );
  }
  return true;
}

Problem: When pick_first is in sticky TRANSIENT_FAILURE (all subchannels have failed to connect), the failed subchannel children still exist in this.children, so this.children.length === 0 is false. The resolver error is therefore dropped and updateState is never called, leaving the picker reflecting the old connection error instead of the new resolver error. The condition should additionally allow a picker update when the balancer is already in TRANSIENT_FAILURE; it should ignore resolver errors only when the balancer is not in TRANSIENT_FAILURE and already has addresses from a previous good update.

Reproduction steps

  1. Create a PickFirstLoadBalancer and send a valid address update with one endpoint.
  2. Transition the subchannel to TRANSIENT_FAILURE so the balancer enters sticky TF mode.
  3. Send a resolver error via updateAddressList(statusOrFromError({code: Status.UNAVAILABLE, details: 'Resolution failed'}), ...).
  4. Observe that updateState on the ChannelControlHelper is not called — the picker still reflects the old connection error.

Test added on HEAD (packages/grpc-js/test/test-pick-first-resolver-error-in-tf.ts):

it('Should update the picker when a resolver error is received while in TRANSIENT_FAILURE', done => {
  let updateStateCallCount = 0;
  const channelControlHelper = createChildChannelControlHelper(
    baseChannelControlHelper,
    {
      updateState: (connectivityState: ConnectivityState, picker: Picker, errorMessage: string | null) => {
        updateStateCallCount++;
      },
    }
  );
  const pickFirst = new PickFirstLoadBalancer(channelControlHelper);
  pickFirst.updateAddressList(
    statusOrFromValue([{ addresses: [{ host: 'localhost', port: 1 }] }]),
    config,
    {},
    ''
  );
  process.nextTick(() => {
    subchannels[0].transitionToState(ConnectivityState.TRANSIENT_FAILURE);
    const countAfterTF = updateStateCallCount;
    assert.strictEqual(countAfterTF, 2, 'Expected 2 updateState calls before resolver error');

    pickFirst.updateAddressList(
      statusOrFromError({ code: Status.UNAVAILABLE, details: 'Resolution failed: test error' }),
      config,
      {},
      ''
    );

    assert.ok(
      updateStateCallCount > countAfterTF,
      `Resolver error while in TRANSIENT_FAILURE should trigger a picker update. ` +
      `Expected updateState to be called more than ${countAfterTF} times, but was called ${updateStateCallCount} times.`
    );
    done();
  });
});

Failing output on HEAD + test patch:

1) pick_first resolver error handling Should update the picker when a resolver error is received while in TRANSIENT_FAILURE:
    Uncaught AssertionError [ERR_ASSERTION]: Resolver error while in TRANSIENT_FAILURE should trigger a picker update. Expected updateState to be called more than 2 times, but was called 2 times.
    + expected - actual

The control run without the test patch exits 0 (same suite).

Environment

  • Node version: v22.23.4
  • Python (toolchain line): Python 3.9.2
  • Commit: 6741e13f210f119d4d3da37691dd01055541663b (2026-10-20), default branch
  • Package: grpc-js (packages/grpc-js)

Additional context

Already fixed in grpc/grpc-go

The analogous bug was addressed in grpc/grpc-go PR #7498 ("pickfirst: New pick first policy for dualstack"), merged 2024-10-10T04:03:47Z (merge commit 00b9e140ce7). The PR is a larger dualstack feature; the relevant part is the new pickfirstleaf policy's resolverErrorLocked, which only ignores resolver errors when the balancer is not in TransientFailure and already has addresses, and otherwise updates the picker with the resolver error.

How grpc-go fixed it (from balancer/pickfirst/pickfirstleaf/pickfirstleaf.go):

func (b *pickfirstBalancer) resolverErrorLocked(err error) {
	...
	// The picker will not change since the balancer does not currently
	// report an error. If the balancer hasn't received a single good resolver
	// update yet, transition to TRANSIENT_FAILURE.
	if b.state != connectivity.TransientFailure && b.addressList.size() > 0 {
		if b.logger.V(2) {
			b.logger.Infof("Ignoring resolver error because balancer is using a previous good update.")
		}
		return
	}

	b.cc.UpdateState(balancer.State{
		ConnectivityState: connectivity.TransientFailure,
		Picker:            &picker{err: fmt.Errorf("name resolver error: %v", err)},
	})
}

This policy has since become grpc-go's default pick_first, and the same check is still on grpc-go master today (balancer/pickfirst/pickfirst.go:197).

Suggested fix

Also update the picker when the balancer is already in TRANSIENT_FAILURE, matching grpc-go:

diff --git a/packages/grpc-js/src/load-balancer-pick-first.ts b/packages/grpc-js/src/load-balancer-pick-first.ts
--- a/packages/grpc-js/src/load-balancer-pick-first.ts
+++ b/packages/grpc-js/src/load-balancer-pick-first.ts
@@ -517,7 +517,14 @@ export class PickFirstLoadBalancer implements LoadBalancer {
       return false;
     }
     if (!maybeEndpointList.ok) {
-      if (this.children.length === 0 && this.currentPick === null) {
+      /* If the balancer is already in TRANSIENT_FAILURE, update the picker
+       * with the new resolver error. Otherwise, only update the picker if
+       * there are no existing addresses (i.e. the balancer has not yet
+       * received a good resolver update). */
+      if (
+        this.currentState === ConnectivityState.TRANSIENT_FAILURE ||
+        (this.children.length === 0 && this.currentPick === null)
+      ) {
         this.channelControlHelper.updateState(
           ConnectivityState.TRANSIENT_FAILURE,
           new UnavailablePicker(maybeEndpointList.error),

Verified on 6741e13 (Node v22.23.4): the test above fails without this patch and passes with it; no other test results change.

Activity

  1. murgatroid99 commented on Oct 28, 2026

    @murgatroid99
    Member

    I think the current behavior is correct. We expect that errors from resolvers are transient errors, like a failure to connect to a DNS server. So, those errors shouldn't generally indicate that a previously received address list is stale. Therefore, the connection errors are more relevant to the user, and should be prioritized.

    The referenced code in grpc-js was introduced in #2925, which references the specification in grpc/grpc#38953. That has this line, but doesn't otherwise specify how to handle resolver errors:

    The channel does not directly use the list of endpoints returned by the resolver; it only passes that list on to the LB policy. If the resolver returns an error for the list of addresses, the LB policy is responsible for determining how to handle that error.

    The referenced Go PR refers to the dualstack backends gRFC, which doesn't specify at all how to handle resolver errors, as far as I can tell. I would just say that I disagree with the logic in Go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions