AWS storage performance

A common question is "How would the client compare the performance of storage on Amazon vs. local data center? "  Well here are some numbers for AWS storage:

a.   Instance storage Disk : Equivalent to local disk for a server

b.   Instance storage SSD

                                                             i.     ~120,000 random read IOPS (4 KB blocks)

                                                            ii.     ~10,000-85,000 random write IOPS (4 KB blocks)

c.    EBS Standard (100 IOPS)

                                                             i.     Reads typically <20ms writes typically <10ms

                                                            ii.     Best effort to 10’s of MB/sec

d.   EBS PIOPS (4K IOPS),

                                                             i.     On best effort, you may get up to 40 MB/sec throughput for 2K PIOPS disk

e.   S3 : loading :http://cloudconclave.blogspot.com/2013/05/aws-s3-load-throughput.html

f.     Glacier : Long time archiving. Takes 3-5 hours to retrieve.

g.   Storage Gateway : On premise to AWS S3 ‘replication’.  So, will depend on the pipe (internet, AWS DirectConnect) from on premise to AWS.

AWS Storage options, cost, and SLA

A common question when moving from on premise to AWS is what are the storage options
 and how much they will cost.  A great place to calculate the cost is the Simple Monthly Calculator: http://calculator.s3.amazonaws.com/calc5.html. 

Before you can calculate the cost you need to be aware of the options, cost of each option, when to use it, and SLAs:

a.   EBS: http://aws.amazon.com/pricing/ec2/ : Data Transfer, EBS standard and EBS PIOPS. Sample pricing:

                                                             i.     Data Transfer, EBS Standard, EBS PIOPS : http://aws.amazon.com/pricing/ebs/

                                                            ii.     Price is for allocated storage

                                                          iii.     SLA : annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume

                                                          iv.     Common use cases: RDBMS (PIOPS), Application Server files (IOPS)

b.   S3 : http://aws.amazon.com/pricing/s3/

                                                             i.     First TB per month of storage: .095

                                                            ii.     Data transfer : Up to 10 TB .120 per GB + request prices (.005 per 1K requests)

                                                          iii.     Price is for used storage

                                                          iv.     SLA: 99.999999999% durability, 99.99% availability

                                                           v.     Common use cases: backups, multi media content, web logs, EMR jobs

c.    Glacier: http://aws.amazon.com/glacier/pricing/

                                                             i.     First TB per month of storage: .01

                                                            ii.     Price is for used storage

                                                          iii.     SLA: 99.999999999% durability

                                                          iv.     Common use cases: archiving, long term storage

d.   Instance storage : Included in price of instance

                                                             i.     Common use cases: web server files (instance disk storage), data warehouses (Redshift runs on instance storage), RDBMS (instance SSD..with redundancy), temporary files

Oracle Linux and HVM AMIs

OEL is not supported on cr1.8xlarge. These instances run as HVM-base instances.  All Oracle AMIs are PVM.    For HVM you have to go with Windows, SUSE or RHEL for now. You can spin one of these up and install Oracle DB on them.  There are ways to convert a PV AMI to  HVM AMI, but whether Oracle will support you is another question.   This applies to the other cluster compute instances types as well: CC2 and CG1.  However, it is mostly likely someone wants to run the cr1 instance type as this is the largest memory footprint by far (244 GB).  This size instance is great for in memory databases or large EBusiness Suite/Fustion installations.  

AWS data transfer costs

Some times instance reside in different AZs and these instances need to share data. Instances also use S3 for things like backups. Here is breakdown of data transfer cost for some typical scenarios: 

1. Instance to Instance in same AZ 
Cost: Free
2. Instance to Instance in different AZ
Cost: Inter-AZ charge for both inbound and outbound
3. Instance to Instance in different account in same or different AZ
Cost: Inter-az charges are applicable for data transfer charges between instances in the different AZ's (same region) and in different accounts
4. Instance to S3 in the same region
Cost: Free

DBA and developer access to Oracle hosted on AWS

Here are three common methods used to limit access to the AWS environment for DBAs and developers:

1. Bastion host : A bastion can be used as a jump box' / proxy server. Developers and DBAs would be given access using SSH and than use other credentials to log into the web, application, and database servers. More on bastion host security can be found here: http://cloudconclave.blogspot.com/2013/05/aws-bastion-host-as-single-point-of.html.  There is the cost of the EC2 instance that is the bastion host and data transfer out costs.
2. VPN with customer gateway and virtual private gateway.  In the case, you create a VPN tunnel.  The costs here are the VPN hardware on your side (customer gateway), cost of virtual private gateway (VPG), and costs of VPN connections and data transfer out of AWS. More on VPN costs here (this assumes this option): http://cloudconclave.blogspot.com/2013/05/vpn-costs-for-connections-and-data.html
3. OpenVPN : You do not incur the cost of hardware on your side and the VPG on the AWS side.  You still have the cost of the data transfer out.  You would also incur the cost of the EC2 instances that is running an open source VPN software stack (in this case OpenVPN).

I am sure there are other methods as well.

Could also use these constructs to provide secure integration from your on premise or third party applications (SFTP for flat file integration, VPN for web services).

VPN costs for connections and data

VPC has no cost associated. However, if you want to extend your data center or provide a secure IPSec tunnel through a VPN client, you need to add in costs for the VPN connection and data transfer costs over the VPN tunnel.   This is the first place to look at to decipher VPN connection cost: http://aws.amazon.com/vpc/pricing/.  The cost is $0.05 per VPN Connection-hour + standard AWS data transfer charges for all data transferred via the VPN Connection. For example, if you have 2 VPN connections the cost for those connections would be:  .05x24x365.25x2(2 connections) / 12 = ~73.05 + data transfer.

Now for the data transfer out piece:  On data transfer out for VPN connection, the cost is .12 per GB for less then 10 TB a month per connection..  So, two connection each at 1 TB a month would be $245.76 a month.  

AWS SSL certificates

All your SSL server certificates are managed by AWS Identity and Access management (IAM). By default, IAM allows 10 server certificates per AWS account. If you try to upload a new server certificate after reaching this limit, you'll get an error. You can request for more certificates using this form - IAM Limit Increase Contact Us Form.

AWS 'Exalytics' on AWS

Exalytics is YAOES (Yet Another Oracle Engineered System). 
https://blogs.oracle.com/emeapartnerbiepm/entry/lastest_bic2g_and_exalytics_demonstration

This web page mentions: The Business Intelligence Challenge to Go (BIC2g) is a demonstration, training, and POC business intelligence platform for Oracle Partners. It is a comprehensive platform, containing the full EPM, BI, and BI Apps (including full ETL with DW) product suites, with extensive (40+) demos. It is available in the Amazon Cloud (not as a download).

This demonstration is actually the test drive for OBIEE (which is the core software stack on the Exalytics machine) on the AWS web site here :

http://aws.amazon.com/solutions/global-solution-providers/oracle/labs/#Oracle_Fusion_Middleware

Auto Scaling based upon schedule

Sometime you may have nightly, monthly, year end, or periodic (calendar based) schedules where you would like to scale out your AWS infrastructure.  Here is more information on schedule based auto scaling:
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/schedule_time.html

Auto Scaling Script

Auto Scaling with VPC 

This example uses the CLI version 1.0.61.2 (API 2011-01-01).  There is now a newer version of the CLI.

Overall notes:

1. Create an image (aka: gold image) that has a health check on it that consists of a simple static HTML page such as a ping.html or an index.html. 

2. The ELB (need an ELB before creating Auto Scaling group) hat has the instances running from your gold image AMI created in step 1. Could potentially (nothing is preventing you from doing) create a auto scale launch configuration (step 6 in process below) with an AMI that is different, with a different instance type as instance type is another parameter of your launch configuration, than the instance that will be part of your auto scaling group.  Both are possible but most will have the same AMI (gold image, so you know you have a health check) and the same instance type.

3. Before you create the ELB you should have instances running of the  gold image to make sure the ELB is working properly.  To provide true HA, make sure to put the instances in two AZs. (for example: us-west-2a, us-west-2b)

4. When creating elb that is using subnets, limitation on ELB is that only one subnet per AZ is allowed.  

5. When creating autoscaling group (as-create-auto-scaling-group)

 need to make sure to specific the VPC subnets and AZs.  The VPC subnets need to be in the AZs specified in the AZ parameter for command.

Steps:

Prework:

1. Identify the VPC : For example, VPC: vpc-9b120cf2

2. Add subnets : In this case, using two public subnets. I had to create the second one (subnet) which was private by default and I had to add an igw to make it public.

3. Subnets -

            A. Subnet: subnet-9f120cf6

            CIDR: 10.0.0.0/24   VPC: vpc-9b120cf2   Availability Zone: us-west-2a

            B. Subnet: subnet-8c130de5

            CIDR: 10.0.5.0/24   VPC: vpc-9b120cf2   Availability Zone: us-west-2b

4. Luanch two instances from AMI that has Apache installed with ping.html (or some other file) as a health check. One instance in subnet A and the other in subnet B. 

5. Create an ELB with the two instances across subnets.  ELB is healthy with two instances running in two AZs in two different public subnets. Limitation on ELB is that one subnet per AZ. 

6. Auto Scaling:

A. as-create-launch-config vpcautoscaling-as-lc --image-id ami-dcd344ec --instance-type t1.micro --key AutoScalingKey

B. as-create-auto-scaling-group vpcautoscaling-as-grp --launch-configuration vpcautoscaling-as-lc  --min-size 4 --max-size 12 --load-balancers VpcautoscalingAutoScalingELB --vpc-zone-identifier subnet-9f120cf6,subnet-8c130de5 --availability-zones us-west-2a,us-west-2b

C. as-describe-auto-scaling-groups --headers

output of command:

INSTANCE  INSTANCE-ID  AVAILABILITY-ZONE  STATE      STATUS   LAUNCH-CONFIG

INSTANCE  i-3af6e708   us-west-2b         InService  Healthy  vpcautoscaling-as-lc

INSTANCE  i-3cf6e70e   us-west-2a         InService  Healthy  vpcautoscaling-as-lc

INSTANCE  i-3ef6e70c   us-west-2a         InService  Healthy  vpcautoscaling-as-lc

INSTANCE  i-38f6e70a   us-west-2b         InService  Healthy  vpcautoscaling-as-lc

D. as-put-scaling-policy vpcautoscaling-scale-out-policy --auto-scaling-group vpcautoscaling-as-grp --adjustment=30 --type PercentChangeInCapacity

ARN: arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:69270ce4-3350-48f4-9d6f-71bc64225554:autoScalingGroupName/vpcautoscaling-as-grp:policyName/vpcautoscaling-scale-out-policy

E.  as-put-scaling-policy vpcautoscaling-scale-in-policy --auto-scaling-group vpcautoscaling-as-grp --adjustment=1 --type PercentChangeInCapacity

ARM - arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:d9a6780b-3319-4b00-98f1-e98dcfc68d82:autoScalingGroupName/vpcautoscaling-as-grp:policyName/vpcautoscaling-scale-in-policy

F. mon-put-metric-alarm --alarm-name AddCapacity --metric-name CPUUtilization --namespace "AWS/EC2" --statistic "Average" --evaluation-periods 6 --period 120 --threshold 80 --comparison-operator GreaterThanOrEqualToThreshold --dimensions "AutoScalingGroupName=vpcautoscaling-as-grp"  --alarm-actions arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:69270ce4-3350-48f4-9d6f-71bc64225554:autoScalingGroupName/vpcautoscaling-as-grp:policyName/vpcautoscaling-scale-out-policy

G. mon-put-metric-alarm --alarm-name RemoveCapacity --metric-name CPUUtilization --namespace "AWS/EC2" --statistic "Average" --evaluation-periods 2 --period 120 --threshold 40 --comparison-operator LessThanOrEqualToThreshold --dimensions "AutoScalingGroupName=vpcautoscaling-as-grp"  --alarm-actions arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:d9a6780b-3319-4b00-98f1-e98dcfc68d82:autoScalingGroupName/vpcautoscaling-as-grp:policyName/vpcautoscaling-scale-in-policy

H. as-describe-policies --auto-scaling-group vpcautoscaling-as-grp —headers