s3cmd to s3-cli : Halving our deployment time!

40 minutes.

That’s the painful amount of time our developers spent screen watching or playing Pokemon Go while the code deployed to around 15 servers.

compiling-new

On a bit of probing, I realized; that the single-most time consuming activity which took 30 of the 40 minutes was syncing files to Amazon S3.

We were using s3cmd, a free command line tool for retrieving, uploading and managing data in Amazon S3. It is a very nice tool to sync files to S3, however, it is very slow.

So I decided to try some alternate tool which could reduce the time required for the sync. While searching for “alternatives for s3cmd”, my mediocre Googling skills introduced me to s3-cli.

As mentioned in its README,  s3-cli is inspired by s3cmd and attempts to be a drop-in replacement.  It is completely compatible with the existing s3cmd configuration file and supports most of the s3cmd commands. The most important advantage is that, while syncing directories, unlike s3cmd which uploads one file at a time, s3-cli uploads files in parallel thus making this activity really fast!

I installed it using npm on my server and the only change to the code was to replace the command “s3cmd” with the command “s3-cli”.

Screenshot from 2016-07-18 23:03:54

I ran the deployment script again and lo and behold! The result of making this small change was that our deployment time was reduced to

20 minutes!

Happy coding!

Advertisements

2 Comments

Add yours →

  1. Do you know what is the optimisation in the tool that causes this performance improvement?

    Like

    • Don’t know much about the internals of the tool, but as mentioned in its README, it uploads multiple files in parallel unlike s3cmd which uploads one file at a time. Also it uses multi-part uploads for large files, where these large files are broken down into smaller files and uploaded in parallel thus making it faster.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: