AWS EBS Performance

Here is a very good presentation on Amazon EBS performance from reInvent in 2013:
http://www.slideshare.net/AmazonWebServices/ebs-webinarfinal

Storage tiering on AWS

Here is the replay of the session I just presented with an AWS partner (App Associates) and customer (Riso):http://cloudconclave.blogspot.com/2013/11/aws-storage-tiering-for-enterprise.html

I opened the session by 'testing' the audiences understanding of AWS storage tiers:
1. Assuming a 16K block size, what storage option provides average throughput of 1 to 2 MBPS ?
2. What storage option has single threaded through put of around 17 MBPS ?
3. Assuming a 16K block size, what EBS volume provides an average of 16-20 MBPS through put ?
4. Once again assuming 16K block size and also assuming 4 1K PIOPS volumes, for what EC2 instances will you start to see network saturation ?
5. What storage option produces approximately 100-145 MBPS read and write through put?
6. For which storage option is it possible to transfer approximately 3 TB a day a day over a WAN ?




















Answers:
1. Standard IOPS
2. S3
3. 1K PIOPS
4. Any instance with .5 Gpbs network connection. For example, m2.2xlarge
5. Hi1.4xlarge (high IO) ephemeral storage
6. AWS Storage gateway

Running Microsoft and Oracle Stacks on Elastic Block Store

This session is focused on using Elastic Block Storage (EBS) with Oracle Databases on AWS:

http://www.youtube.com/watch?v=D9eUmhvgMwY

Moving EBS volumes between Linux and Windows

EBS volumes are portable between instances running different operating systems but this does not mean the underlying file system format will be compatible.  EBS is a block level storage device.  The volume must be formatted with a file system which may or may not run across different EC2 instances based upon the OS of the instance.

Moving between Linux and Windows file formatted EBS formatted volumes were most of the compatible issues may arise. NTFS and EXT4 are some of the most common file system formats.  

An option is to used NTFS as your 'master' file system.  Then use a utility like ntfsprogs (http://en.wikipedia.org/wiki/Ntfsprogson) to use the EBS volume for both Windows and Linux instances. If you don't need writing on Linux, you can mount the NTFS drive and read it on Linux . Linux can natively read NTFS but it can not write to NTFS drives natively.  

The other option is to use EXT4 (or another Linux file system) as your 'master' file system. For example, EXT4 is not natively supported on Windows.  Good discussion here:

http://superuser.com/questions/465393/how-to-mount-read-write-an-ext4-parti

tion-on-windows-solution-back-to-ntfs

Also, ext2fsd  (http://www.ext2fsd.com)  a proven solution for reading EXT4 from Linux.  For those that don't need NTFS, sharing filesystems between Windows and Linux can be done using exFAT. Another potential solution that supports reading and writing to NTFS on Linux is: http://sourceforge.net/projects/ntfs-3g/

Redshift block size

Typical database block sizes range from 2 KB to 32 KB. Amazon Redshift uses a block size of 1 MB, which is more efficient and further reduces the number of I/O requests needed to perform any database loading or other operations that are part of query execution.

Oracle Database PIOPS and striping

A common question on PIOPS and striping:

Question: Striping with PIOPS - Is it necessary to stripe data when using PIOPS?

Response: It depends on how many PIOPS you are looking to achieve.  Since max PIOPS is per volume is 2000 PIOPS, if you need more then 2000 PIOPS then you would need to stripe.  Less than 2000 PIOPS then striping is not required.