Skip to content

Getting ESOCKETTIMEOUT when streaming through file.createReadStream() #2317

New issue
New issue
Closed
Closed
Getting ESOCKETTIMEOUT when streaming through file.createReadStream()#2317
Assignees
stephenplusplus
Labels
api: storageIssues related to the Cloud Storage API.

Description

@MuthuD
MuthuD
opened on May 17, 2017

Hi,

We transform a large binary file (>150MB up to GBs) and publish the data to pubsub.

OS: Google App Engine (Assuming its windows)
Node.js version: LTS
npm version: LTS
google-cloud-node version: HEAD

Steps to reproduce

  1. require google-cloud
  2. let file = bucket.file([largebinaryfile]);
  3. let readStream = file.createReadStream();
  4. file.pipe(transformStream).pipe(writeStream)

readStream: Reads large file from bucket directly from bucket
transformStream: Does some operation
writeStream: publishes transformed data to cloud pubsub

Problem,
readStream receives ESOCKETTIMEDOUT error more often, how to handle this is this value configurable through code?

Thanks

Activity

  1. stephenplusplus
    added
    api: storageIssues related to the Cloud Storage API.
    on May 17, 2017

  2. stephenplusplus commented on May 17, 2017

    @stephenplusplus
    stephenplusplus
    on May 17, 2017
    Contributor

    Can you double check the version of google-cloud-node that you're using? Only the latest release will retry on errors such as a socket timeout.

  3. MuthuD commented on May 17, 2017

    @MuthuD
    MuthuD
    on May 17, 2017
    Author

    Hi Stephen,

    Thanks for the response.

    I'm using version @google-cloud/storage: "0.7.0"

    I will try with version 1.1.1 and let you know if there is any issue.

    Thanks

  4. stephenplusplus commented on May 22, 2017

    @stephenplusplus
    stephenplusplus
    on May 22, 2017
    Contributor

    @MuthuD how did it go?

  5. stephenplusplus
    closed this as completedon Jun 1, 2017

  6. Scarysize commented on Sep 11, 2018

    @Scarysize
    Scarysize
    on Sep 11, 2018

    Hey there, thought I would chime in here before opening a new issue in the nodejs-storage repo. I got the exact same issue on version 2.0.1.

    Here is the code producing the error:

    const readline = require('readline');

const storage = new Storage({projectId: config.projectId});
const bucket = storage.bucket(config.bucketName);
const remoteFile = bucket.file('events.txt');

const lineReader = readline.createInterface(remoteFile.createReadStream());

lineReader.on('line', /* do stuff */);

    The code runs on GCE:

    • OS: Linux 4.9.0-8-amd64 SMP Debian 4.9.110-3+deb9u3 (2018-08-19) x86_64 GNU/Linux
    • node: v8.11.4

    It reliably times out after 60 seconds of streaming.

    Let me know if you need any further information.

  7. JustinBeckwith commented on Sep 11, 2018

    @JustinBeckwith
    JustinBeckwith
    on Sep 11, 2018
    Contributor

    Thanks! Opening this on the nodejs-storage repo is absolutely the way to go though 👍

  8. Scarysize commented on Sep 12, 2018

    @Scarysize
    Scarysize
    on Sep 12, 2018

    👍 Okay, added a comment to a similar issue: googleapis/nodejs-storage#27 (comment)

  9. JustinBeckwith
    assigned
    stephenplusplus
    on Feb 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: storageIssues related to the Cloud Storage API.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions