Storage: lots of "socket hang up" errors

@ovaris

From @ovaris on September 21, 2017 8:31

Environment details

OS:
Node.js version: 8.5.0
npm version: 5.3.0
google-cloud-node/storage version: 1.2.1

I have a utlity nodejs script that checks existence of few thousands of files in cloud storage.
I run script locally, so not in Cloud environment.
I'm executing those checks (bucket.file(fileName).exists()) in batch of 20, so not all checks are fired concurrently.
I'm seeing lots of these errors when trying to run script:

{ Error: socket hang up
    at TLSSocket.onHangUp (_tls_wrap.js:1140:19)
    at Object.onceWrapper (events.js:314:30)
    at emitNone (events.js:110:20)
    at TLSSocket.emit (events.js:207:7)
    at endReadableNT (_stream_readable.js:1059:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
  code: 'ECONNRESET',
  path: null,
  host: 'accounts.google.com',
  port: 443,
  localAddress: undefined }

and these:

{ Error: read ECONNRESET
    at _errnoException (util.js:1026:11)
    at TLSWrap.onread (net.js:606:25) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }

aaand these:

{ Error: socket hang up
    at createHangUpError (_http_client.js:345:15)
    at TLSSocket.socketOnEnd (_http_client.js:437:23)
    at emitNone (events.js:110:20)
    at TLSSocket.emit (events.js:207:7)
    at endReadableNT (_stream_readable.js:1059:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9) code: 'ECONNRESET' }

I have added this fix (suggested here: googleapis/google-cloud-node#2254):

const gcs = storage();
//https://github.com/GoogleCloudPlatform/google-cloud-node/issues/2254
gcs.interceptors.push({
    request: function(reqOpts) {
        reqOpts.forever = false;
        return reqOpts
    }
});

I have tried to reduce the check batch size, but it didn't have any effect.

Copied from original issue: googleapis/google-cloud-node#2623

@stephenplusplus

From @stephenplusplus on September 21, 2017 14:10

Can you share a small gist or repo that I can use to try to reproduce?

@ovaris

From @ovaris on September 21, 2017 14:57

Here's gist, hopefully it makes sense: https://gist.github.com/ovaris/c30c492db06fbe73875ed71d00dcb547

@stephenplusplus

From @stephenplusplus on September 21, 2017 15:46

It's a bit hard to go from that, since I can't tap into the firebase instance. Can you try to replicate the problem without any external dependencies?

Here's simple test that replicates error, if I put 1000 times, it works, but with 3000 times it starts to fail:

import {chunk, times} from 'lodash';
import * as storage from '@google-cloud/storage';

const gcs = storage();
//https://github.com/GoogleCloudPlatform/google-cloud-node/issues/2254
gcs.interceptors.push({
    request: function(reqOpts) {
        reqOpts.forever = false;
        return reqOpts
    }
});

const bucket = gcs.bucket("xxx");


const checkFileExists = (fileName) =>
    bucket.file(fileName).exists()
        .then(([exists]) => exists || fileName)
        .catch((error) => {
            console.error('Error checking ', fileName);
            console.error(error);
            return `error: ${fileName}`;
        });


(async () => {
    const checkPromises = [];
    times(3000, () => checkPromises.push(checkFileExists(`xxx/1.png`)));

    const chunks = chunk(checkPromises, 20);
    let i = 1;
    let results = [];
    for (const chunk of chunks) {
        const chunkResults = await Promise.all(chunk);
        results = results.concat(chunkResults);
        i++;
    }
    const missing = results.filter((exists) => exists !== true);
    if (missing.length > 0) {
        console.log('Missing: \n');
        missing.forEach((file, i) => console.log(`${i}: ${file}`));
        process.exit(1);
    }
    console.log('YAY!');
    process.exit(0);
})();

This is a stupid user error and can be closed. Of course code above will fire ALL requests at the same time

ovaris, what code did you use to fix this issue? I'm having a similar problem where my code is firing 1000 database uploads at a time - hence the database returns socket hang up errors because of the excessive load

Glad to hear this was resolved!

Getting some ESOCKETTIMEDOUT errors when using createWriteStream. This problem was also posted on stackoverflow:

https://stackoverflow.com/questions/47328578/getting-esockettimeout-econnreset-or-socket-hang-up-with-large-file-when-stream

I am opening at most 5 streams in parallel.

@surjikal

@surjikal could you try using the latest version (v1.5.2) and if the problem persists, please open a new issue with reproduction steps and I'll check it out.

Cool will do, thanks.

still getting this issue with the last version. Any workaround?

@stephenplusplus

@kinwa91 @stephenplusplus this one has been going on for a while now, and I'm concerned about the whole downgrading to 2.1 thing. Can y'all prioritize an investigation for this tomorrow?

fix: remove timeout rule from streaming uploads (#365)

deps: update @google-cloud/common (#596)

This should be resolved tomorrow morning, once a new release of @google-cloud/storage goes out. If you want to test in the meantime, please install from master and let me know how it goes!

Tomorrow came earlier than expected-- v2.4.2 is out now! Please update and report back any lingering issues.

With v2.4.2 I was able to stream write 6 out of 6 files to GCS. With earlier versions, at least 2-3 of those would have failed, so this is much better.

The strange thing is that I still see a socket timeout error in the logs, but it looks as though the file was successfully written to GCS, so I'm not sure what's going on there. I'll keep an eye on it and report back.

Here's the error I saw:

Error: ESOCKETTIMEDOUT at ClientRequest.<anonymous> (/srv/node_modules/request/request.js:816:19) at Object.onceWrapper (events.js:313:30) at emitNone (events.js:106:13) at ClientRequest.emit (events.js:208:7) at TLSSocket.emitTimeout (_http_client.js:721:34) at Object.onceWrapper (events.js:313:30) at emitNone (events.js:106:13) at TLSSocket.emit (events.js:208:7) at TLSSocket.Socket._onTimeout (net.js:422:8) at ontimeout (timers.js:498:11)

request should be eliminated from the file.createWriteStream({resumable:false}) path. Are you using {resumable: true}? In either case, feel free to share additional details if the problem keeps up. Thanks for the update!

Ugh, so stupid. I didn't deploy after making my updates. I'll deploy and check again. Sorry for the false alarm.

I ran another test today with 7 files and they all successfully streamed to GCS. Great work.

Phew, glad to hear. Thanks for the update!

fix: remove timeout rule from streaming uploads (#365)

Navigation Menu

Repository navigation

Storage: lots of "socket hang up" errors #27

Environment details

40 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Storage: lots of "socket hang up" errors #27

Description

Environment details

Activity

lukesneeringer commented on Sep 21, 2017

lukesneeringer commented on Sep 21, 2017

lukesneeringer commented on Sep 21, 2017

ovaris commented on Sep 22, 2017

ovaris commented on Sep 29, 2017

ashleyhenderson commented on Oct 7, 2017

stephenplusplus commented on Oct 8, 2017

surjikal commented on Jan 3, 2018

stephenplusplus commented on Jan 3, 2018

surjikal commented on Jan 3, 2018

jeremymarc commented on Mar 20, 2018

40 remaining items

JustinBeckwith commented on Feb 3, 2019

stephenplusplus commented on Feb 6, 2019

stephenplusplus commented on Feb 6, 2019

micahwedemeyer commented on Feb 6, 2019

stephenplusplus commented on Feb 6, 2019

micahwedemeyer commented on Feb 7, 2019

micahwedemeyer commented on Feb 7, 2019

stephenplusplus commented on Feb 7, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions