Skip to main content

Mass Updating Amazon Web Services S3 Permissions

Recently I ran into an issue where I needed to update the Access Control Lists(ACLs) for about 1.5 Million pictures. Amazon currently doesnt provide an easy way to do this and current tools such as 3Hub and such cant handle such a large volume.

By far the biggest flaw when it comes to mass updating in current S3 utilities is the lack of threading. With 3Hub I was able to update the permissions on around 400k images in a 30 hour period. I did not want to leave my laptop at work yet again so I decided to just write a script that could do this quicker and run on a server.
Here is the code that I used to accomplish this.
The config file is pretty straightforward and goes into a directory called config
Now obviously the script above would need to be adapted, in my case I had a list of ids from 1 to 9999. My method above is just a quick and dirty way to thread the application without having to pull a list of the IDs and feed it out. In our development environment which contains about 300,000 images I was able to update all of the permissions in about an hour.
Something to note –  If you run this script and update the ACLs it replaces your ACls with the above. So if have 5 different ACLs be sure to set them all above so that they are applied. If you need to only change a specific ACL and not the others then this script would need to be adapted to do that.

Sticky Session Load Balancing with mod_balancer

When load balancing across multiple application servers that are cache dependent you can use sticky sessions to easily ensure a browser request is routed to the same applicaton server each time. To do this we use the mod_headers module along with mod_balancer and mod_proxy.
When a request comes in the BALANCER_SESSION_ROUTE is checked (this is set as “route” in your site cookie). If this is unset then your request is assigned a new route id and your request is routed on its merry way. If your BALANCER_SESSION_ROUTE does not match the BALANCER_WORKER_ROUTE then you are assigned a new route id. This can happen if a server is removed from the pool.
Implementing this is into an existing setup is actually pretty simple. Lets use an example configuration.
We append the routeid to the end of each of our routes. We then must insert this route into the site cookie using mod_headers
Next we make ProxyPass follow the route using sticky sessions.
Here is an example of a full configuration.
And thats it, now you need to restart apache and run a quick test using wget.
Here we can see that the routeID is properly being stored in the cookie. An alternative method for checking would be to use firebug or developer tools in chrome and checking the contents of your site cookie. Using this method you should see the same routeID as well and can see that it does not change as you flip through pages.