Skip to content

channelz related memory leak (ServerHttp2Session) #2068

Closed

Description

@sam-la-compass

Problem Description
Upgrading from @grpc/grpc-js from 1.3.x to 1.5.x seems to introduce a memory leak
There seems to be an excess amount of ServerHttp2Sessions created (and unable to be freed)

We currently run a production service that has a single grpc-js server that creates multiple grpc-js clients. The clients are created and destroyed using lightning-pool.

Channelz is disabled when we initialize the server/clients as well with the 'grpc.enable_channelz': 0

Reproduction Steps
Create a single grpc-js server that calls grpc-js clients as needed from a pool resource. In our case, the server is running and when requests are made, we acquire a client via the pool (factory created once as a singleton) to make a request. These should be able to handle concurrent/multiple requests.
'grpc.enable_channelz': 0 for server/client options.

Environment

  • OS Name: macOS (locally testing) and running on AWS EKS clusters (production)
  • Node Version: 14.16.0
  • Package Name and Version: @grpc/grpc-js@1.5.9

Additional Context
Checking out some heap info, it looks like there is a lot of ServerHttp2Sessions created and being retained. The attached screenshot is the service running for about an hour with limited requests (just idling). I also have another snapshot where the service is running for much longer with many more objects created.

I also read through #1958 and #1941 which seems related but had no luck getting to a solution.

Happy to provide more context or help as neeeded.

Screen Shot 2022-03-24 at 1 38 38 PM

Activity

  1. changed the title [-]channelz related memory leak[/-] [+]channelz related memory leak (ServerHttp2Session)[/+] on Mar 24, 2022
  2. murgatroid99 commented on Mar 24, 2022

    @murgatroid99
    Member

    Ironically, it looks like the bug specifically caused a memory leak when 'grpc.enable_channelz': 0 is set, because of a missing check for that in one place. I published a change that should fix it in version 1.5.10.

  3. sam-la-compass commented on Mar 25, 2022

    @sam-la-compass
    Author

    Thanks a lot for the quick fix! Upgrading to 1.5.10 looks to have fixed the issue

  4. sam-la-compass commented on Mar 25, 2022

    @sam-la-compass
    Author

    False positive, looks like there is still something lingering. Investigating to find more details, will recomment

  5. sam-la-compass commented on Mar 25, 2022

    @sam-la-compass
    Author

    ServerHttp2Session is fixed and no longer being created but the memory leak for our system still exists. Will need to reopen or create another issue once I narrow down the source

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions