How to set destination while using uploadManyFiles in TransferManager

I have a local directory and I want to upload its contents to cloud storage, but when using transferManager.uploadManyFiles with either the directory name, or array of files, the destination of the file in cloud storage is taken from the local file path.

local directory structure:

.
└── data
    └── avatars
        └── <image files>

here's my code

import { Storage, TransferManager } from "@google-cloud/storage";

const storage = new Storage();
const transferManager = new TransferManager(storage.bucket("bucket-name"));

const res = await transferManager.uploadManyFiles("data/avatars", {
  prefix: "avatars",
  passthroughOptions: {
    destination: "", // ???
  },
  skipIfExists: true,
});

when uploaded with following code, it becomes

.
└── avatars
    └── data
        └── avatars
            └── <image files>

the prefix property works, but I want to upload files directly to avatars/
there seem to be a passthroughOptions that can take a destination, but when using uploadManyFiles we don't know the exact file names, so can't make the full path

Shouldn't it be that while using prefix, property, the local file path shouldn't be considered?

For a temporary fix, I edited library source to ignore local paths when prefix is supplied, it works, I'd still like to know the intended way of doing so

@arnav-kr

Hi @arnav-kr thanks for the question. The original intent was to mirror the same directory structure from local to storage. Obviously as you bumped into this can be improved. Currently destination isn't directly settable in uploadMany

nodejs-storage/src/transfer-manager.ts

Lines 95 to 100 in 18eef67

    
           export interface UploadManyFilesOptions { 
        
             concurrencyLimit?: number; 
        
             skipIfExists?: boolean; 
        
             prefix?: string; 
        
             passthroughOptions?: Omit<UploadOptions, 'destination'>; 
        
           }

.

We can probably add some additional options to offer a greater degree of flexibility such as not mirroring the entire path structure and only using prefix. I can take a look at making these improvements in the near future.

Probably in options, something like a destinationBuilder would make it more flexible, than specifing a string, as this way we can get the filename and process accodingly

...
destinationBuilder: (path, additionalParams) => {
  let dest;
  // perform operation on path
  return dest;
}
...

Thanks ❤️

Can you add an example code for that feature? Unit tests are not understandable for that case

@xephtar

@xephtar can you please open a new issue as a feature request so we can better track it.

Navigation Menu

Repository navigation

How to set destination while using uploadManyFiles in TransferManager #2492

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	export interface UploadManyFilesOptions {
	concurrencyLimit?: number;
	skipIfExists?: boolean;
	prefix?: string;
	passthroughOptions?: Omit<UploadOptions, 'destination'>;
	}

How to set destination while using uploadManyFiles in TransferManager #2492

Description

Activity

arnav-kr commented on Jun 26, 2024

ddelgrosso1 commented on Jun 26, 2024

arnav-kr commented on Jun 26, 2024

arnav-kr commented on Jul 15, 2024

xephtar commented on Aug 20, 2024

ddelgrosso1 commented on Aug 21, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions