Mass Updating Amazon Web Services S3 Permissions

Recently I ran into an issue where I needed to update the Access Control Lists(ACLs) for about 1.5 Million pictures. Amazon currently doesnt provide an easy way to do this and current tools such as 3Hub and such cant handle such a large volume.

By far the biggest flaw when it comes to mass updating in current S3 utilities is the lack of threading. With 3Hub I was able to update the permissions on around 400k images in a 30 hour period. I did not want to leave my laptop at work yet again so I decided to just write a script that could do this quicker and run on a server.
Here is the code that I used to accomplish this.
The config file is pretty straightforward and goes into a directory called config
Now obviously the script above would need to be adapted, in my case I had a list of ids from 1 to 9999. My method above is just a quick and dirty way to thread the application without having to pull a list of the IDs and feed it out. In our development environment which contains about 300,000 images I was able to update all of the permissions in about an hour.
Something to note –  If you run this script and update the ACLs it replaces your ACls with the above. So if have 5 different ACLs be sure to set them all above so that they are applied. If you need to only change a specific ACL and not the others then this script would need to be adapted to do that.

Leave a Reply

Your email address will not be published. Required fields are marked *