Disabling delete access on Amazon S3

When writing code which accesses S3 you should, of course, use IAM credentials specifically for the purpose. The tricky part can be granting the minimal access. In this case I wanted the code to be able to read and update files, but not delete them. After struggling with this for a while I realized I was missing the part which allowed the code to see the S3 buckets. Here’s the final policy:

 

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*",
      "Condition": {}
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*",
        "s3:Put*"
      ],
      "Resource": "arn:aws:s3:::my-bucket-name/*",
      "Condition": {}
     }
  ]
}

Design Inspiration

A collection of UI designs:

http://www.mobile-patterns.com/

http://pttrns.com/

http://androidpttrns.com/

http://ui.theultralinx.com/

http://designm.ag/inspiration/33-ios-mobile-app-screen-views-for-design-inspiration/

http://mycolorscreen.com/

http://www.jqmgallery.com/

http://tapfancy.com/

http://beautifulpixels.com/

http://www.mobiletuxedo.com/category/ui-patterns/

http://inspired-ui.com/

http://www.mobile-patterns.com/

http://creattica.com/mobile/latest/

http://www.lovelyui.com/

http://iphone.appstorm.net/roundups/design/40-more-pixel-perfect-iphone-app-interfaces/

http://www.tappgala.com/

http://www.behance.net/search?search=app

http://theultralinx.com/category/recent/art-design/graphic-design/ui-screenshots

http://iphone.meer.li/

 

Designer’s gallery:

http://www.scoutzie.com

Performance of EC2 versus S3 versus CloudFront

I’ve been trying to work out the fastest way to serve up thumbnail images for an auto-complete feature.

Test parameters: Cleared the browser cache, downloaded approximately 126 thumbnails, average 1.3Kb, to my office in San Francisco. Measured download duration in Chrome’s debugger.

Results:

So although EC2 is currently faster, that’s under zero load and I’m not confident it would stay that way once the load picked up. I’m assuming Cloudfront is already under stable load and is very scalable so these are the numbers we should see in production.

Here’s the raw data with tracert output for added interest.
Zero-load Amazon EC2 Micro instance in Oregon data center
Avg download time – total: 66ms StdDev 8
Avg download time less latency: 8ms StdDev 13
Tracing route to awseb-ios-html-demo-296746925.us-west-2.elb.amazonaws.com [50.1
12.94.116] over a maximum of 30 hops:

1    <1 ms    <1 ms    <1 ms  192.168.x.x
2    27 ms    19 ms    29 ms x-x-x-1.hsd1.ca.comcast.net [x.x.x.x]
3     8 ms     9 ms     9 ms  te-7-3-ur02.sffolsom.ca.sfba.comcast.net [68.87.196.173]
4    12 ms    11 ms    12 ms  te-1-2-0-0-ar01.oakland.ca.sfba.comcast.net [68.86.143.54]
5    17 ms    23 ms    23 ms  he-2-12-0-0-cr01.sacramento.ca.ibone.comcast.net [68.86.90.129]
6    16 ms    13 ms    15 ms  pos-0-8-0-0-cr01.sanjose.ca.ibone.comcast.net [68.86.85.78]
7    13 ms    14 ms    13 ms  te-1-3.car1.SanJose1.Level3.net [4.71.118.37]
8    15 ms    13 ms    25 ms  vlan70.csw2.SanJose1.Level3.net [4.69.152.126]
9    12 ms    13 ms    13 ms  ae-72-72.ebr2.SanJose1.Level3.net [4.69.153.21]
10    33 ms    28 ms    28 ms  ae-7-7.ebr1.Seattle1.Level3.net [4.69.132.50]
11    30 ms    29 ms    30 ms  ae-11-51.car1.Seattle1.Level3.net [4.69.147.131]
12    29 ms    28 ms    28 ms  AMAZON.COM.car1.Seattle1.Level3.net [4.53.144.74]
13    29 ms    28 ms    62 ms  205.251.225.24
14    34 ms    38 ms    35 ms  205.251.232.78
15    36 ms    35 ms    35 ms  205.251.232.215
16    36 ms    35 ms    35 ms  205.251.232.221
17    35 ms    34 ms    35 ms  ec2-x-x-x-x.us-west-2.compute.amazonaws.com [x.x.x.x]

Amazon S3
Avg download time – total: 173ms StdDev 135
Avg download time less latency: 14ms StdDev 43

Tracing route to s3-website-us-west-2.amazonaws.com [205.251.235.157] over a maximum of 30 hops:

1    <1 ms    <1 ms    <1 ms  192.168.x.x
2   196 ms    20 ms    17 ms  x-x-x-1.hsd1.ca.comcast.net [x.x.x.x]
3     9 ms     8 ms     9 ms  te-7-3-ur02.sffolsom.ca.sfba.comcast.net [68.87.196.173]
4    17 ms    29 ms    20 ms  te-1-2-0-0-ar01.oakland.ca.sfba.comcast.net [68.86.143.54]
5    20 ms    11 ms    11 ms  he-2-15-0-0-cr01.sacramento.ca.ibone.comcast.net [68.86.91.225]
6    17 ms    15 ms    15 ms  pos-0-6-0-0-cr01.sanjose.ca.ibone.comcast.net [68.86.86.202]
7    13 ms    13 ms    13 ms  xe-10-2-0.edge1.SanJose1.Level3.net [4.71.118.1]
8    13 ms    23 ms    12 ms  vlan80.csw3.SanJose1.Level3.net [4.69.152.190]
9    13 ms    14 ms    13 ms  ae-82-82.ebr2.SanJose1.Level3.net [4.69.153.25]
10    30 ms    27 ms    38 ms  ae-7-7.ebr1.Seattle1.Level3.net [4.69.132.50]
11    28 ms    30 ms    28 ms  ae-x-x-x.car1.Seattle1.Level3.net [x.x.x.x]

Amazon Cloudfront
Avg download time – total: 92ms StdDev 4
Avg download time less latency: 27ms StdDev 5

Tracing route to dv4f02orrismj.cloudfront.net [205.251.215.88] over a maximum of 30 hops:

1     1 ms     1 ms    <1 ms  192.168.x.x
2    23 ms    29 ms    14 ms x-x-x-1.hsd1.ca.comcast.net [x.x.x.x]
3     9 ms     8 ms    10 ms  te-7-3-ur02.sffolsom.ca.sfba.comcast.net [68.87.196.173]
4    11 ms    11 ms    11 ms  te-1-10-0-2-ar01.oakland.ca.sfba.comcast.net [68.85.154.66]
5    18 ms    11 ms    11 ms  he-2-15-0-0-cr01.sacramento.ca.ibone.comcast.net [68.86.91.225]
6    12 ms    17 ms    14 ms  pos-0-8-0-0-cr01.sanjose.ca.ibone.comcast.net [68.86.85.78]
7    12 ms    13 ms    11 ms  xe-5-2-0.edge1.SanJose1.Level3.net [4.71.118.49]
8    14 ms    13 ms    14 ms  ae-1-60.edge1.SanJose3.Level3.net [4.69.152.16]
9    14 ms    13 ms    14 ms  AMAZON.COM.edge1.SanJose3.Level3.net [4.53.208.22]
10    13 ms    22 ms    12 ms  205.251.229.155
11    65 ms    13 ms    14 ms  205.251.230.89
12    12 ms    13 ms    14 ms  server-x.x.x.x-88.sfo5.r.cloudfront.net [x.x.x.x]

Auto-complete – performance tuning

While working on a prototype JQuery Mobile application I implemented a quick-and-dirty auto-complete function in PHP. This uses a regex select statement on a table of about 2500 rows – not particularly elegant, but ok for a prototype. However, I couldn’t stop thinking over the weekend that I wanted to implement something a bit faster and more elegant, possibly using memcached. The first rule of performance tuning? Baseline measurements.

To my surprise the php method was taking on average 15 microseconds, which seems pretty good for what I’m assuming is a table scan of 2500 rows (admittedly with a local MySQL database). Looking at the query plan, perhaps my regex is somehow making use of the unique index on the name column:

id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE mytable index NULL name_UNIQUE 47 NULL 10 Using where

In that case I’ll let MySQL do it’s thing and help it along its way by enabling query caching. Here’s how: http://www.howtogeek.com/howto/programming/speed-up-your-web-site-with-mysql-query-caching/

After a quick check with:

SHOW STATUS LIKE 'Qc%';

…to ensure I was getting some cache hits, checked my timing and found that a significant number of method calls dropped from 15 to 5 microseconds, presumably correlating to the cache hits.

Proportionally re-sizing transparent images in PHP

Should be easy, right? No.

After trying a few differnt approaches (and losing  transparency) I used Maxin’s solution: https://github.com/maxim/smart_resize_image

REGEX for auto-complete

I’ve been working on some auto-complete code for a mobile app and struggling (as usual) with regex. The query needs to search approximately 6000 multi-word rows for potential matches. My first attempt replaced the spaces in the search term with | then used the following regex:

^.*($search).*\$

This gives far to many hits for my purposes. For example:

Mon would give hits on monitor, monday but also lemon which I don’t want.

Changing to:

[[:<:]]$search

worked a bit better, but as soon as I put multiple words in my search term it broke, missing the most obvious results.  The solution was to remove the code replacing spaces with | that I’d added earlier and it worked perfectly. It now takes multiple word search terms and does perfect auto-complete matching.

Martin L would match Aston Martin Lagonda

 

Should I be using regex in a db query> No, but it’ll do for a prototype.

 

Hadoop on Windows 7 – gotchas

There are a number of existing guides on getting pseudo-distributed mode and HDFS configured on Windows:

http://www.zdnet.com/blog/microsoft/wheres-hadoop-for-microsofts-windows-server/12809

http://v-lad.org/Tutorials/Hadoop/14%20-%20start%20up%20the%20cluster.html

http://hayesdavis.net/2008/06/14/running-hadoop-on-windows/

Despite following the above carefully, I ran into a number of issues:

1) I could not get anything other than 0.20.0 working because of a bug in the way file permissions are set – http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.user/25837. There is a workaround here but I wasn’t prepared to go to these lengths.

2) I ran into issues using the username in my HDFS location as my the sshd user and my local user ended up trying to write to different locations. Maybe something else is wrong here, but this was an easy solution.

Here are my configurations for Hadoop 0.20.0:

core-site.xml:

<property>
    <name>hadoop.tmp.dir</name>
    <value>/cygwin/tmp/hadoop</value>
</property>
<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:54310</value>
</property>

hdfs-site.xml:

<property>
 <name>hadoop.tmp.dir</name>
 <value>/cygwin/tmp/hadoop</value>
</property>
<property>
 <name>fs.default.name</name>
 <value>hdfs://localhost:54310</value>
</property>
<property>
 <name>mapred.job.tracker</name>
 <value>hdfs://localhost:54311</value>
</property>
<property> 
 <name>dfs.replication</name>
 <value>1</value>
</property>
<property>
 <name>mapred.child.java.opts</name>
 <value>-Xmx512m</value>
</property>
<property>
 <name>dfs.permissions</name>
 <value>false</value>
</property>

mapred-site.xml

<property>
 <name>mapred.job.tracker</name>
 <value>localhost:8021</value>
</property>

BTW – here are the default web URLs for the management pages:

http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/

Follow

Get every new post delivered to your Inbox.