Repository navigation
Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted #1158
Description
Problem description
We use @googlecloud/datastore dependency in our code. Since some version of grpc-js (currently we're on 0.6.9) we started to receive the following error in our production and staging backends, also in our cron jobs that stream over ~100K / 1M records in Datastore (sometimes after ~5 minutes, sometimes after ~30 minutes). Error details as seen in our Sentry:
Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
at Object.callErrorFromStatus (/root/repo/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
at Http2CallStream.<anonymous> (/root/repo/node_modules/@grpc/grpc-js/build/src/client.js:96:33)
at Http2CallStream.emit (events.js:215:7)
at Http2CallStream.EventEmitter.emit (domain.js:476:20)
at /root/repo/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22
at processTicksAndRejections (internal/process/task_queues.js:75:11) {
code: 8,
details: 'Bandwidth exhausted',
metadata: Metadata { internalRepr: Map {}, options: {} },
note: 'Exception occurred in retry method that was not classified as transient'
}
Reproduction steps
Very hard to give reproduction steps. Stack trace is not "async", in a way that it doesn't like to exact place where it was called in the code (like it would have done with return await). We know that in the backend service we're doing all kinds of Datastore calls, but NOT stream. In cron jobs we DO stream as well as other (get, save) api calls.
Environment
- Backend service is in AppEngine (was in Node10, now in Node12 beta, which runs node 12.4.0)
@grpc/grpc-js@0.6.9
We definitely did NOT see this error in 0.5.x, but I don't remember exactly since which version of 0.6.x it started to appear.
Additional context
Error happens quite seldomly, maybe ~1-2 times a days on a backend service that serves ~1M requests a day. But when it fails - it fails hard, it's impossible to try/catch such error, and usually one "occurrence" of such error fails multiple requests from our clients. For example, last night it failed in our staging environment that was running e2e tests (many browsers open in parallel) which produced ~480 errors in one spike. So, looks like this error does not "recover the connection" very quickly.
Another annoying thing of this error is that if it happens inside a long-running cron job that streams some table - we have no way to recover from that error and the whole cron job becomes "failed in the middle" (imagine running a DB migration that fails in the middle in a non-transactional way). So, if our cron job needs to run for ~3 hours and fails after 2 hours - we have no choice but to restart it from the very beginning (paying all the datastore costs).
Activity
kirillgroshkov commentedon Nov 4, 2019
UPD. Don't want to complicate this github issue with different errors, but trying now the same cron job multiple times gives me different error. Don't know how it is related to the original one. Tell me if I should open another issue for it or not. Stack trace:
Currently I have a script that reproduced this error after ~7 minutes 3 times in a row (will try more times, but looks "consistently reproducible"). All this script does is it opens 16 streams in parallel, reading data from Datastore and saving it (streaming) into a gzipped file.
This is when running on my local MacOS.
murgatroid99 commentedon Nov 4, 2019
OK, there are a couple of issues here. Tracking down the source of these errors may be tricky for the reasons you mentioned, but they are definitely the result of errors being sent by the server in some form.
You also mention that you can't
try/catchthese errors. These are errors that the Datastore library is outputting. They may be promise rejections or something but you should be able to handle them.kirillgroshkov commentedon Nov 4, 2019
I'm confident that these errors we're unable to try catch. We have a code like:
This
console.errorfires sometimes on "normal" occasions (non-grpc-related), but it never fired when we had thisBandwidth exhaustederror or any of the previous errors fromgrpc-jsin the past (like the ones that other people experienced and that were closed in 0.6.x versions). So, I'm suspecting there's some problem in how@google-cloud/datastorefires errors. In other words, example of "non-try-catchable" error is:AFAIK - nothing can catch such error, so it'll become
unhandledRejectionoruncaughtException.Correct me if I'm wrong.
Or, is there a way to do an experiment, add some
throw errorin grpc source code innode_modulesand see iftry/catcharound Datastore api call gonna be caught or not?murgatroid99 commentedon Nov 4, 2019
Well, grpc doesn't throw that error. It just passes it along to the client (Datastore, in this case). I am fairly certain that Datastore propagates the error as a promise rejection, but either way, it's not a grpc issue if Datastore is propagating that error improperly.
MiLk commentedon Dec 11, 2019
We started to observe the same issue since yesterday.
We have a client using
grpc-jsthat we updated from0.5.2to0.6.11about 3 weeks ago,and a server using
grpc-gothat we updated fromv1.21.1tov1.25.1about 3 weeks ago.What is strange is that we didn't update anything in the past days, but the issue started to show only now.
We get a lot of
8 RESOURCE_EXHAUSTED: Bandwidth exhaustedand a few13 INTERNALerrors on the client after running for a few hours.We have no error logs on the backend which could help us track down this issue.
We used to have a keepalive enforcement policy set with a 25 seconds
MinTime.It has been temporarily disabled.
MiLk commentedon Dec 13, 2019
So far, our best guess has been that it has been related to the amount of concurrent requests we are doing. But the numbers are not consistent.
After enabling tracing and using
0.6.14, if we try to send 50k requests with a concurrency of 1k, we reach a situation where we start getting errors 13 and 8.At some point we receive:
Here is some logs from around the time it starts failing.
murgatroid99 commentedon Dec 13, 2019
These logs show that the HTTP/2 streams are being closed with RST_STREAM frames with error codes 2 (INTERNAL_ERROR) and 11 (ENHANCE_YOUR_CALM which translates to RESOURCE_EXHAUSTED). It is very likely (but not 100% guaranteed) that these error codes are being sent by the server and not generated on the client. Specifically, they are likely being generated by the server's HTTP/2 implementation, not gRPC. You mentioned that you are using grpc-go on the server, so you may have some luck filing a bug report with the grpc-go repository and including a link to this comment.
MiLk commentedon Dec 13, 2019
Ok, thank you for your help.
MiLk commentedon Dec 13, 2019
After spending more time troubleshooting this, I think the problem might come from the client and not the server.
The client seems to be the one sending the
RST_STREAMframes.I've reproduced the issue locally with the same go server and node client, and I'm using wireshark to check the traffic.
Here a the logs of the client at the time of the issue:
Here are the server logs (using GODEBUG=http2debug=2):
We can see that it says
read RST_STREAMat the time the errors start to occur.I wanted to confirm that the server was not misbehaving and so I ran wireshark.
Here we can see it's coming from the client (port 65517) to the server (8443).
I'm going to continue digging into the client.
MiLk commentedon Dec 13, 2019
After moving from node 12 to node 13, we no longer have the issue.
It's most likely related to nodejs/node@18a1796 and nodejs/node@8a4a193
The fix is scheduled for v10.17.1 and v12.13.2.
nodejs/node#30684
murgatroid99 commentedon Dec 18, 2019
Thank you for sharing those extra details.
shynome commentedon Feb 6, 2020
the error is disappear after I upgrade to
v13.Thank you for sharing those extra details.
ajayhjaiswal commentedon Feb 13, 2020
Thanks guys for information, Does google Cloud support Nodejs 13 for AppEngine deployment for Production use, Sorry for adding different context to original topic but are inter-link.
isaiah-coleman commentedon Apr 21, 2020
I've upgraded to node
v13and haven't had any luck in supressing these errors. Is this the case for anyone else? Running@grpc/grpc-js@0.7.9for referencestephenplusplus commentedon May 21, 2020
This would seem to relate to this issue as well: googleapis/nodejs-datastore#525. So to avoid the error, it basically comes down to finding the right batch size of concurrent requests to send at a time?
14 remaining items
Use "grpc" (C-based implementation) instead of "@grpc/grpc-js" (pure …
paulish commentedon May 25, 2022
Had the same problem. Fixed by the following code:
pagoru commentedon Sep 23, 2022
This solves the issue! Thanks @paulish !
pagoru commentedon Sep 23, 2022
The bug is related with this:
grpc-node/packages/grpc-js/src/subchannel.ts
Lines 414 to 424 in 3f71020
The default value is
Number.MAX_SAFE_INTEGER, but it's not working, if you assign again the value works. Please, check this, the error still occurs and looks simple to fix.'grpc-node.max_session_memory'looks defined and it's not.murgatroid99 commentedon Sep 23, 2022
It seems to be working in my tests. Can you provide a complete example that demonstrates the problem you are experiencing?
pagoru commentedon Sep 26, 2022
I cannot provide an example because it's internal code from my company.
Btw, this are the options assigned (the last one was added on Friday and fixes the problem), maybe another one causes the problem?
murgatroid99 commentedon Sep 26, 2022
Please verify what version of the library you are using. That option was added in version 1.3.0, but the default was not set to
Number.MAX_SAFE_INTEGERuntil version 1.6.3.pagoru commentedon Sep 26, 2022
Using:
murgatroid99 commentedon Sep 26, 2022
OK, that is the right version. Well, if you can provide a code sample that has different behavior when that option is set to
Number.MAX_SAFE_INTEGERthan when it is omitted, then I can debug it, otherwise there is nothing more I can do.sberthier commentedon Oct 14, 2022
It seems that I have a similar problem. I have tested to set
"grpc-node.max_session_memory": Number.MAX_SAFE_INTEGERand it seems to do the job. I can confirm that in following days.I may have an explanation about this issue.
In subchannel.ts, we have
https://github.com/grpc/grpc-node/blob/master/packages/grpc-js/src/subchannel.ts#L414
There is a default value of
Number.MAX_SAFE_INTEGERfor the client side.But in the server.ts side, there isn't
https://github.com/grpc/grpc-node/blob/master/packages/grpc-js/src/server.ts#L345
It may be why inject
grpc-node.max_session_memoryin server config solves our pb.murgatroid99 commentedon Oct 14, 2022
Thank you for pointing that out. I will fix that inconsistency.
murgatroid99 commentedon Oct 14, 2022
I have published grpc-js 1.7.2 with
Number.MAX_SAFE_INTEGERas the default value for that option on the server too.Merge #13237
murgatroid99 commentedon Apr 1, 2024
Since there have been no new comments since that fix went out, I assume this is resolved.