You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
channelz related memory leak (ServerHttp2Session) #2068
Problem Description
Upgrading from @grpc/grpc-js from 1.3.x to 1.5.x seems to introduce a memory leak
There seems to be an excess amount of ServerHttp2Sessions created (and unable to be freed)
We currently run a production service that has a single grpc-js server that creates multiple grpc-js clients. The clients are created and destroyed using lightning-pool.
Channelz is disabled when we initialize the server/clients as well with the 'grpc.enable_channelz': 0
Reproduction Steps
Create a single grpc-js server that calls grpc-js clients as needed from a pool resource. In our case, the server is running and when requests are made, we acquire a client via the pool (factory created once as a singleton) to make a request. These should be able to handle concurrent/multiple requests. 'grpc.enable_channelz': 0 for server/client options.
Environment
OS Name: macOS (locally testing) and running on AWS EKS clusters (production)
Node Version: 14.16.0
Package Name and Version: @grpc/grpc-js@1.5.9
Additional Context
Checking out some heap info, it looks like there is a lot of ServerHttp2Sessions created and being retained. The attached screenshot is the service running for about an hour with limited requests (just idling). I also have another snapshot where the service is running for much longer with many more objects created.
I also read through #1958 and #1941 which seems related but had no luck getting to a solution.
Ironically, it looks like the bug specifically caused a memory leak when 'grpc.enable_channelz': 0 is set, because of a missing check for that in one place. I published a change that should fix it in version 1.5.10.
ServerHttp2Session is fixed and no longer being created but the memory leak for our system still exists. Will need to reopen or create another issue once I narrow down the source
Activity
[-]channelz related memory leak[/-][+]channelz related memory leak (ServerHttp2Session)[/+]murgatroid99 commentedon Mar 24, 2022
Ironically, it looks like the bug specifically caused a memory leak when
'grpc.enable_channelz': 0is set, because of a missing check for that in one place. I published a change that should fix it in version 1.5.10.sam-la-compass commentedon Mar 25, 2022
Thanks a lot for the quick fix! Upgrading to 1.5.10 looks to have fixed the issue
sam-la-compass commentedon Mar 25, 2022
False positive, looks like there is still something lingering. Investigating to find more details, will recomment
sam-la-compass commentedon Mar 25, 2022
ServerHttp2Session is fixed and no longer being created but the memory leak for our system still exists. Will need to reopen or create another issue once I narrow down the source