Best way to move files between S3 buckets?

Best way to move files between S3 buckets?

I'd like to copy some files from a production bucket to a development bucket daily.

For example: Copy productionbucket/feed/feedname/date to developmentbucket/feed/feedname/date

Because the files I want are so deep in the folder structure, it's too time consuming to go to each folder and copy/paste.

I've played around with mounting drives to each bucket and writing a windows batch script, but that is very slow and it unnecessarily downloads all the files/folders to the local server and back up again.

Answer by Steffen Opel for Best way to move files between S3 buckets?

Update

As pointed out by alberge (+1), nowadays the excellent AWS Command Line Interface provides the most versatile approach for interacting with (almost) all things AWS - it meanwhile covers most services' APIs and also features higher level S3 commands for dealing with your use case specifically, see the AWS CLI reference for S3:

• sync - Syncs directories and S3 prefixes. Your use case is covered by Example 2 (more fine grained usage with --exclude, --include and prefix handling etc. is also available):

  The following sync command syncs objects under a specified prefix and bucket to objects under another specified prefix and bucket by copying s3 objects. [...]

  aws s3 sync s3://mybucket s3://mybucket2  

For completeness, I'll mention that the lower level S3 commands are also still available via the s3api sub command, which would allow to directly translate any SDK based solution to the AWS CLI before adopting its higher level functionality eventually.

---

Initial Answer

Moving files between S3 buckets can be achieved by means of the PUT Object - Copy API (followed by DELETE Object):

This implementation of the PUT operation creates a copy of an object that is already stored in Amazon S3. A PUT copy operation is the same as performing a GET and then a PUT. Adding the request header, x-amz-copy-source, makes the PUT operation copy the source object into the destination bucket. Source

There are respective samples for all existing AWS SDKs available, see Copying Objects in a Single Operation. Naturally, a scripting based solution would be the obvious first choice here, so Copy an Object Using the AWS SDK for Ruby might be a good starting point; if you prefer Python instead, the same can be achieved via boto as well of course, see method copy_key() within boto's S3 API documentation.

PUT Object only copies files, so you'll need to explicitly delete a file via DELETE Object still after a successful copy operation, but that will be just another few lines once the overall script handling the bucket and file names is in place (there are respective examples as well, see e.g. Deleting One Object Per Request).

Answer by dk. for Best way to move files between S3 buckets?

If you have a unix host within AWS, then use s3cmd from s3tools.org. Set up permissions so that your key as read access to your development bucket. Then run:

s3cmd cp -r s3://productionbucket/feed/feedname/date s3://developmentbucket/feed/feedname  

Answer by Alex Dean for Best way to move files between S3 buckets?

We had this exact problem with our ETL jobs at Snowplow, so we extracted our parallel file-copy code (Ruby, built on top of Fog), into its own Ruby gem, called Sluice:

https://github.com/snowplow/sluice

Sluice also handles S3 file delete, move and download; all parallelised and with automatic re-try if an operation fails (which it does surprisingly often). I hope it's useful!

Answer by bantic for Best way to move files between S3 buckets?

Here is a ruby class for performing this: https://gist.github.com/4080793

Example usage:

$ gem install aws-sdk  $ irb -r ./bucket_sync_service.rb  > from_creds = {aws_access_key_id:"XXX",                  aws_secret_access_key:"YYY",                  bucket:"first-bucket"}  > to_creds = {aws_access_key_id:"ZZZ",                aws_secret_access_key:"AAA",                bucket:"first-bucket"}  > syncer = BucketSyncService.new(from_creds, to_creds)  > syncer.debug = true # log each object  > syncer.perform  

Answer by Matt Dell for Best way to move files between S3 buckets?

.NET Example as requested:

using (client)  {      var existingObject = client.ListObjects(requestForExisingFile).S3Objects;       if (existingObject.Count == 1)      {          var requestCopyObject = new CopyObjectRequest()          {              SourceBucket = BucketNameProd,              SourceKey = objectToMerge.Key,              DestinationBucket = BucketNameDev,              DestinationKey = newKey          };          client.CopyObject(requestCopyObject);      }  }  

with client being something like

var config = new AmazonS3Config { CommunicationProtocol = Protocol.HTTP, ServiceURL = "s3-eu-west-1.amazonaws.com" };  var client = AWSClientFactory.CreateAmazonS3Client(AWSAccessKey, AWSSecretAccessKey, config);  

There might be a better way, but it's just some quick code I wrote to get some files transferred.

Answer by sgimeno for Best way to move files between S3 buckets?

To move/copy from one bucket to another or the same bucket I use s3cmd tool and works fine. For instance:

s3cmd cp --recursive s3://bucket1/directory1 s3://bucket2/directory1  s3cmd mv --recursive s3://bucket1/directory1 s3://bucket2/directory1  

Answer by alberge for Best way to move files between S3 buckets?

The new official AWS CLI natively supports most of the functionality of s3cmd. I'd previously been using s3cmd or the ruby AWS SDK to do things like this, but the official CLI works great for this.

http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

aws s3 sync s3://oldbucket s3://newbucket  

Answer by Ankit Valia for Best way to move files between S3 buckets?

I know this is an old thread but for others who reach there my suggestion is to create a scheduled job to copy content from production bucket to development one.

You can use If you use .NET this article might help you

http://www.codewithasp.net/2015/03/aws-s3-copy-object-from-one-bucket-or.html

Answer by esharp for Best way to move files between S3 buckets?

Here's a simple ruby class to copy every object from one bucket to another bucket: https://gist.github.com/edwardsharp/d501af263728eceb361ebba80d7fe324

Answer by lony for Best way to move files between S3 buckets?

For me the following command just worked:

aws s3 mv s3://bucket/data s3://bucket/old_data --recursive  

Answer by Justin Workman for Best way to move files between S3 buckets?

Actually as of recently I just use the copy+paste action in the AWS s3 interface. Just navigate to the files you want to copy, click on "Actions" -> "Copy" then navigate to the destination bucket and "Actions" -> "Paste"

It transfers the files pretty quick and it seems like a less convoluted solution that doesn't require any programming, or over the top solutions like that.

Answer by James for Best way to move files between S3 buckets?

If you have a lot of objects (millions or maybe even billions), and you want this to finish in your lifetime, I've written a C# .NET command-line utility to do this in a highly parallel way. It will track tens of thousands of concurrent object copies, restart where it left off if it gets killed for any reason, etc. I've successfully used it to replicate about 100 million objects between regions. I wrote it as I went, so I'm not sure exactly how long it would have taken if I ran it again with an empty destination, but I'm guessing less than a week). It also tracks cost as it goes. It can also be used to verify that the objects are synchronized for fairly low cost (unless interrupted, it queries each object just once on each side in batches of 1000 objects at a time).

Its two major limitations are:

1. it only works with objects that don't require multipart upload (<4GB), but it logs all of those in an error log. These objects were not a requirement for my needs, but the source is there if you want to enhance it.
2. it relies on the ETag being consistent, which may not be the case depending on how your objects were originally created--multipart uploads even of small objects cause ETags to be different than non-multipart uploads.

Here it is: S3BucketSync

Fatal error: Call to a member function getElementsByTagName() on a non-object in D:\XAMPP INSTALLASTION\xampp\htdocs\endunpratama9i\www-stackoverflow-info-proses.php on line 72