AWS EBS Performance

Here is a very good presentation on Amazon EBS performance from reInvent in 2013:
http://www.slideshare.net/AmazonWebServices/ebs-webinarfinal

AWS PIOPS SSD backed

The question of what the technology backing EBS Provisioned IOPS volumes comes up often.  PIOPS volumes are backed by Solid-State Drives (SSDs), Provisioned IOPS volumes support up to 30 IOPS per GB which enables you to provision 4000 IOPS on a volume as small as 134 GB. 

More details here:
https://aws.amazon.com/ebs/details/

Oracle Database AMIs

An easy way to Bring Your Own License (BYOL) to AWS and start running the Oracle Database is to utilize the AWS Marketplace AMIs.  There are six AMIs for Oracle 12c and 11g.  Standard Edition One, Standard Edition and Enterprise Edition versions are available for both 12c and 11g. All these Marketplace AMIs, use PIOPS with a provisioning of 500 IOPS.  The OS system is Red Hat Enterprise Linux 6.5. The AWS Marketplace AMIs can be found here:

https://aws.amazon.com/marketplace/search/results/ref=sp_navgno_search_box?page=1&searchTerms=oraclempbyol

EC2 EBS optimized instances throughput

EC2 EBS-Optimized Instances are listed here:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html

It is important to understand the maximum throughput in MB/s and maximum 16 KB IOPS when architecting Oracle databases on AWS.  

Network throughput for an Oracle database : Smaller number of large PIOPS volumes or large number of smaller PIOPS volumes

Individual 4000 PIOP volumes at 16K  I/O's can theoretically transfer 64MB/s.  Smaller PIOP volumes (e.g 1000) can burst up to 40MB/s.   In this case, especially for bursty workloads, it may be more cost effective to use smaller PIOPS volumes (but more of them) in a raid0 configuration, than to use fewer higher rated PIOPS volumes.

Oracle Database size and network throughput : IOPS and network throughput

When configuring an Oracle Database on AWS EC2, you need to consider both storage (EBS) IOPS and network throughput.   With PIOPS, you can achieve up to 4K IOPS per EBS volumes.  However, the EC2 instances (assuming EBS optimized or 10 Gbps cluster compute) can be a potential bottle neck.  For example, the m2.2xlarge instance type has a maximum throughput of 0.5Gb.  Which means this instance type is limited to approximately 3750 * 16 KiB IOPS. Therefore, one 4K PIOPS volume would start to saturate the network.   Take into account the IOPS and instance network throughput when designing your Oracle Database on AWS EC2.

AWS Storage Tiering for Enterprise Workloads

This a replay of the AWS reInvent session that covers storage tiering options for running Oracle Enterprise workloads such as Oracle Databases on AWS:

http://www.youtube.com/watch?v=i5Il80bhQCI

Slides:
http://www.slideshare.net/tomlaszewski/aws-storage-tiering-for-enterprise-workloads

AWS EBS PIOPS : block size and IOPS

Having spent more time in the database world than in the web development world, I am accustomed to measuring (database) performance/through put in terms of IOPS or TPS.  The web/video/image world like to use MB/sec.  Why I am saying this? Because it relates to the conversation about getting a certain level of PIOPS (based upon a 16 KB block) on AWS EBS and how this effects MB/sec.  MB/sec, I am beginning to understand, and maybe move to the 'dark side', is the ultimate measure of disk 'performance'.  

Example: A 2000 Provisioned IOPS volume can handle:

•2000 16KB read/write per second, or 1000 32KB read/write per second, or 500 64KB read/write per second 

•You will get consistent 32 MB/sec throughput (with 16KB or higher IOs)

•Perform an index creation action and sends I/O of 32K, IOPS becomes 1000, you still get 32MB/sec throughput

•On best effort, you may get up to 40 MB/sec throughput 

So, you may be better off using a 64 KB block size but your PIOPS will show up as lower but your MB/sec could be better.

AWS storage performance

A common question is "How would the client compare the performance of storage on Amazon vs. local data center? "  Well here are some numbers for AWS storage:

a.   Instance storage Disk : Equivalent to local disk for a server

b.   Instance storage SSD

                                                             i.     ~120,000 random read IOPS (4 KB blocks)

                                                            ii.     ~10,000-85,000 random write IOPS (4 KB blocks)

c.    EBS Standard (100 IOPS)

                                                             i.     Reads typically <20ms writes typically <10ms

                                                            ii.     Best effort to 10’s of MB/sec

d.   EBS PIOPS (4K IOPS),

                                                             i.     On best effort, you may get up to 40 MB/sec throughput for 2K PIOPS disk

e.   S3 : loading :http://cloudconclave.blogspot.com/2013/05/aws-s3-load-throughput.html

f.     Glacier : Long time archiving. Takes 3-5 hours to retrieve.

g.   Storage Gateway : On premise to AWS S3 ‘replication’.  So, will depend on the pipe (internet, AWS DirectConnect) from on premise to AWS.

AWS Storage options, cost, and SLA

A common question when moving from on premise to AWS is what are the storage options
 and how much they will cost.  A great place to calculate the cost is the Simple Monthly Calculator: http://calculator.s3.amazonaws.com/calc5.html. 

Before you can calculate the cost you need to be aware of the options, cost of each option, when to use it, and SLAs:

a.   EBS: http://aws.amazon.com/pricing/ec2/ : Data Transfer, EBS standard and EBS PIOPS. Sample pricing:

                                                             i.     Data Transfer, EBS Standard, EBS PIOPS : http://aws.amazon.com/pricing/ebs/

                                                            ii.     Price is for allocated storage

                                                          iii.     SLA : annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume

                                                          iv.     Common use cases: RDBMS (PIOPS), Application Server files (IOPS)

b.   S3 : http://aws.amazon.com/pricing/s3/

                                                             i.     First TB per month of storage: .095

                                                            ii.     Data transfer : Up to 10 TB .120 per GB + request prices (.005 per 1K requests)

                                                          iii.     Price is for used storage

                                                          iv.     SLA: 99.999999999% durability, 99.99% availability

                                                           v.     Common use cases: backups, multi media content, web logs, EMR jobs

c.    Glacier: http://aws.amazon.com/glacier/pricing/

                                                             i.     First TB per month of storage: .01

                                                            ii.     Price is for used storage

                                                          iii.     SLA: 99.999999999% durability

                                                          iv.     Common use cases: archiving, long term storage

d.   Instance storage : Included in price of instance

                                                             i.     Common use cases: web server files (instance disk storage), data warehouses (Redshift runs on instance storage), RDBMS (instance SSD..with redundancy), temporary files

AWS EBS PIOPS increased to 4K

This announcement will provide additional throughput for high transaction volume Oracle databases running on EC2:

• Jeff Barr's Blog post: http://aws.typepad.com/aws/2013/05/provision-up-to-4k-iops-per-ebs-volume.html
• What's new announcement: https://aws.amazon.com/about-aws/whats-new/2013/05/07/announcing-4000-iops-per-piops-volume-and-marketplace-support/
• EBS detail page: http://aws.amazon.com/ebs/
• AWS Marketplace Provisioned IOPS detail page: https://aws.amazon.com/marketplace/piops
• EBS Provisioned IOPS Documentation: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/AmazonEBS.html

SSD for Oracle Databases

Here are a couple of articles/web site with the pros and cons of running your Oracle database on SSD:

http://www.pythian.com/blog/de-confusing-ssd-for-oracle-databases/
http://www.slideshare.net/gwenshap/ssd-collab13

If your Oracle database is less than 2 TB you could run it on the  hi1.4xlarge instance type.  This instance type has  2 SSD-based volumes each with 1024 GB of instance storage.  More on AWS instance types here: http://aws.amazon.com/ec2/instance-types/

You could also just use Oracle RDS on AWS.  With PIOPS for RDS, you can get up to 25K IOPS on your Oracle database. 
Instagram uses SSD : http://m.cio.com/article/716829/SSDs_Boost_Instagram_39_s_Speed_on_Amazon_EC2

EC2 EBS Optimized Instances

Because Oracle databases are typically terabytes in size and required 1000's of IOPS, you will typically use EBS PIOPS volumes with EBS-optimized EC2 instances.   This leads the often asked question of "What does EBS optimized EC2 instances really mean?"  The short answer is that storage pipe to the EBS (PIOPS) volume is dedicated and does not compete with network traffic like it does in a non EBS-optimized instance.  A more detailed answer can be found here:
http://perspectives.mvdirona.com/2012/08/01/EBSProvisionedIOPSOptimizedInstanceTypes.aspx

On a related note, additional instance types now offered EBS Optimized functionality: http://aws.amazon.com/about-aws/whats-new/2013/03/19/announcing-ebs-optimized-support-for-additional-instance-types/

Oracle Database PIOPS and striping

A common question on PIOPS and striping:

Question: Striping with PIOPS - Is it necessary to stripe data when using PIOPS?

Response: It depends on how many PIOPS you are looking to achieve.  Since max PIOPS is per volume is 2000 PIOPS, if you need more then 2000 PIOPS then you would need to stripe.  Less than 2000 PIOPS then striping is not required. 

AWS Provisioned I/Os

Provisioned IOPS volumes are designed to deliver within 10% of the provisioned IOPS performance 99.9% of the time. Therefore, PIOPS are very good choice when running RDBMS systems like Oracle on EC2 or RDS.

AWS EBS PIOPS volume creating tablespace

Creating you EBS volume and attaching it to your EC2 instance through the AWS console, CLI or APIs is just the first step in using this volume in your Oracle database.  If on Linux, you then need to format, create a mount point, and then mount your volume as follows:

1.mkfs -t ext3 /dev/sdg

2. mkdir /mnt/oracle1000

3. mount /dev/sdg /mnt/oracle1000

(Remember to give permissions - chmod - to oracle user on your mount point)

 More information on mounting EBS volumes can be found here:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

To check if the device has been mounted properly, you can issue these commands:
-  df -l
-  mount

To insure that your EBS volumes get mounted at start up time, you can add the device mapping to the /etc/fstab file

Then the Oracle tablespace needs to be created.  This is done using the following command:

This is for creating a tablespace larger then the Oracle standard limits:
create bigfile tablespace datapiops1k
datafile '/mnt/oracle1000/bigfiledata.dbf'
size 600G;