Traffic costs between EC2 instances, AZs, and VPCs

Often times when running on AWS you will have multiple AZs, accounts and VPCs.  This is a simple summary of EC2 data transfer costs:

1. Within the same AZ  : Free: It is within the same AZ and over private IP.
2. Between two AZs :   Cost. Will incur inter-az charges for both inbound and outbound

3. Between Accounts in different VPCs :  Cost. Inter-az charges are applicable for data transfer charges between instances in the different AZ's (same region) and in different accounts.
4. Between one EC2 classic instance and VPC EC2 instance in a same AZ : Cost
5. Instances to S3 in the same region: Free
6. Instance to S3 in different region: Cost. cross region charges

VPC and AZs when using ELB and Auto Scaling

Here are a couple of things to be aware of when using Auto Scaling, VPC, and ELB with Auto Scaling:

1. There is a limitation on ELB in regards to AZs and subnets.   You can only have one subnet per AZ.  You will most likely be creating an ELB (and using VPC) when using Auto Scaling so it is important to understand this.
2. When creating an autoscaling group (as-create-auto-scaling-group), make sure to specific the VPC subnets and AZs if you are using .  The VPC subnets need to be in the AZs specified in the AZ parameter for the as-create-auto-scaling-group command. No checking is done to make sure this is true so it is up to you to make sure these AZs actually exist in the subnets you specify. 

AWS data transfer costs

Some times instance reside in different AZs and these instances need to share data. Instances also use S3 for things like backups. Here is breakdown of data transfer cost for some typical scenarios: 

1. Instance to Instance in same AZ 
Cost: Free
2. Instance to Instance in different AZ
Cost: Inter-AZ charge for both inbound and outbound
3. Instance to Instance in different account in same or different AZ
Cost: Inter-az charges are applicable for data transfer charges between instances in the different AZ's (same region) and in different accounts
4. Instance to S3 in the same region
Cost: Free

Auto Scaling Script

Auto Scaling with VPC 

This example uses the CLI version 1.0.61.2 (API 2011-01-01).  There is now a newer version of the CLI.

Overall notes:

1. Create an image (aka: gold image) that has a health check on it that consists of a simple static HTML page such as a ping.html or an index.html. 

2. The ELB (need an ELB before creating Auto Scaling group) hat has the instances running from your gold image AMI created in step 1. Could potentially (nothing is preventing you from doing) create a auto scale launch configuration (step 6 in process below) with an AMI that is different, with a different instance type as instance type is another parameter of your launch configuration, than the instance that will be part of your auto scaling group.  Both are possible but most will have the same AMI (gold image, so you know you have a health check) and the same instance type.

3. Before you create the ELB you should have instances running of the  gold image to make sure the ELB is working properly.  To provide true HA, make sure to put the instances in two AZs. (for example: us-west-2a, us-west-2b)

4. When creating elb that is using subnets, limitation on ELB is that only one subnet per AZ is allowed.  

5. When creating autoscaling group (as-create-auto-scaling-group)

 need to make sure to specific the VPC subnets and AZs.  The VPC subnets need to be in the AZs specified in the AZ parameter for command.

Steps:

Prework:

1. Identify the VPC : For example, VPC: vpc-9b120cf2

2. Add subnets : In this case, using two public subnets. I had to create the second one (subnet) which was private by default and I had to add an igw to make it public.

3. Subnets -

            A. Subnet: subnet-9f120cf6

            CIDR: 10.0.0.0/24   VPC: vpc-9b120cf2   Availability Zone: us-west-2a

            B. Subnet: subnet-8c130de5

            CIDR: 10.0.5.0/24   VPC: vpc-9b120cf2   Availability Zone: us-west-2b

4. Luanch two instances from AMI that has Apache installed with ping.html (or some other file) as a health check. One instance in subnet A and the other in subnet B. 

5. Create an ELB with the two instances across subnets.  ELB is healthy with two instances running in two AZs in two different public subnets. Limitation on ELB is that one subnet per AZ. 

6. Auto Scaling:

A. as-create-launch-config vpcautoscaling-as-lc --image-id ami-dcd344ec --instance-type t1.micro --key AutoScalingKey

B. as-create-auto-scaling-group vpcautoscaling-as-grp --launch-configuration vpcautoscaling-as-lc  --min-size 4 --max-size 12 --load-balancers VpcautoscalingAutoScalingELB --vpc-zone-identifier subnet-9f120cf6,subnet-8c130de5 --availability-zones us-west-2a,us-west-2b

C. as-describe-auto-scaling-groups --headers

output of command:

INSTANCE  INSTANCE-ID  AVAILABILITY-ZONE  STATE      STATUS   LAUNCH-CONFIG

INSTANCE  i-3af6e708   us-west-2b         InService  Healthy  vpcautoscaling-as-lc

INSTANCE  i-3cf6e70e   us-west-2a         InService  Healthy  vpcautoscaling-as-lc

INSTANCE  i-3ef6e70c   us-west-2a         InService  Healthy  vpcautoscaling-as-lc

INSTANCE  i-38f6e70a   us-west-2b         InService  Healthy  vpcautoscaling-as-lc

D. as-put-scaling-policy vpcautoscaling-scale-out-policy --auto-scaling-group vpcautoscaling-as-grp --adjustment=30 --type PercentChangeInCapacity

ARN: arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:69270ce4-3350-48f4-9d6f-71bc64225554:autoScalingGroupName/vpcautoscaling-as-grp:policyName/vpcautoscaling-scale-out-policy

E.  as-put-scaling-policy vpcautoscaling-scale-in-policy --auto-scaling-group vpcautoscaling-as-grp --adjustment=1 --type PercentChangeInCapacity

ARM - arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:d9a6780b-3319-4b00-98f1-e98dcfc68d82:autoScalingGroupName/vpcautoscaling-as-grp:policyName/vpcautoscaling-scale-in-policy

F. mon-put-metric-alarm --alarm-name AddCapacity --metric-name CPUUtilization --namespace "AWS/EC2" --statistic "Average" --evaluation-periods 6 --period 120 --threshold 80 --comparison-operator GreaterThanOrEqualToThreshold --dimensions "AutoScalingGroupName=vpcautoscaling-as-grp"  --alarm-actions arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:69270ce4-3350-48f4-9d6f-71bc64225554:autoScalingGroupName/vpcautoscaling-as-grp:policyName/vpcautoscaling-scale-out-policy

G. mon-put-metric-alarm --alarm-name RemoveCapacity --metric-name CPUUtilization --namespace "AWS/EC2" --statistic "Average" --evaluation-periods 2 --period 120 --threshold 40 --comparison-operator LessThanOrEqualToThreshold --dimensions "AutoScalingGroupName=vpcautoscaling-as-grp"  --alarm-actions arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:d9a6780b-3319-4b00-98f1-e98dcfc68d82:autoScalingGroupName/vpcautoscaling-as-grp:policyName/vpcautoscaling-scale-in-policy

H. as-describe-policies --auto-scaling-group vpcautoscaling-as-grp —headers

AWS RDS cross region replication

RDS offers a simple way to replicate across AZs (Multi-AZ feature) for fault tolerance and high availability.  RDS cross region synchronization / replication provides additional fault tolerance.  This is an interesting situation because RDS is a 'black box'/PaaS/DaaS environment. Therefore, no access to the OS level is provided. This means this replication solutions like MySQL Replication or Oracle Goldengate can not be used.  Until RDS offers snapshots like the new EBS region snapshotting, you will have to use tools like BitTorrent or CloudBeam to move database dump files across regions.You could use DB links with materialized views as this article mentions: http://aws.amazon.com/articles/4173109646282306. DB Links are slow but if you don't have a low RTO and RPO you could use this...Or come up with your own creative solution.

AWS Four DR scenarios

There are four DR scenarios that highlight usage of AWS in a DR and a 'HA-lite' situation:

•  Backup and Restore - For systems running on AWS, customers also back up into Amazon S3. Snapshots of Elastic Block Store (EBS) volumes and backups of Amazon RDS are stored in Amazon S3. Alternatively, you can copy files directly into Amazon S3, or you can choose to create backup files and copy them to Amazon S3.
• Pilot Light for Simple Recovery into AWS - This scenario is similar to a Backup and Restore scenario, however, you must ensure that you have the most critical core elements of your system already configured and running in AWS (the pilot light). When the time comes for recovery, you would then rapidly provision a full scale production environment around the critical core. The database data would be replicated to S3. You would typically have some pre-configured servers bundled as Amazon Machine Images (AMIs), which are ready to be started up at a moment’s notice.  These servers could be EC2 instances that have been stopped.
• Warm Standby Solution - A warm standby solution extends the pilot light elements and preparation. It further decreases the recovery time because in this case, some services are always running. By identifying your business-critical systems, you would fully duplicate these systems on in another AWS zone or region and have them always on.  This would insure that the EC2 capacity is available in a disaster.
•  Active-Active Solution - A multi-site solution runs in another AZ or zone in an active-active configuration. The data replication method that you employ will be determined by the recovery point (RPO) you choose.