Server-Side Bucket to Bucket #24

Activity

1. willmcgugan commented on Feb 21, 2018

   

   willmcgugan

   on Feb 21, 2018

   Member

   More actions

   S3FS isn't aware of bucket-to-bucket transfers. So it will essentially do a download/upload.

   It should be possible to detect where a bucket-to-bucket transfer would be possible. To be honest, I didn't even know that was possible. Will look in to that...
2. willmcgugan

   added

   enhancement

   on Feb 21, 2018
3. geoffjukes commented on Feb 22, 2018

   

   geoffjukes

   on Feb 22, 2018

   ContributorAuthor

   More actions

   @willmcgugan I didn't either, until recently, It's actually pretty simple:

       s3.meta.client.copy_object(
           CopySource={
               'Bucket': src,
               'Key': key
           },
           Bucket=tgt,
           Key=new_key if new_key else key,
           StorageClass='STANDARD_IA'
       )
   

   where src and tgt are Boto3 Bucket instances (with thier appropriate connections).

   http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.copy

   Thanks for looking into it!
4. ELind77 commented on Apr 30, 2020

   

   ELind77

   on Apr 30, 2020

   Last edited by ELind77

   More actions

   Any progress on this?

   Any guidance on the change that would need to be made? Currently, it looks like copy is doing the right thing, but it's not really clear to me what happens when the src_path is an S3FS object from another bucket. Is there a way to get the bucket name from the src_path without causing problems or inconsistencies with the API?

   Context:
   

   s3fs/fs_s3fs/_s3fs.py

   Line 790 in 7f95af6

   self.client.copy_object(