CloudWatch custom metric examples

Creating custom metric requires some custom coding or scripting using the AWS Command Line Interface or SDK.   As the name implies these metric are custom so some custom coding / scripting is required to be done.  Here are some places to start that will provide examples of creating custom metric:
CloudWatch custom metric for MySQL using PHP:
http://aws.typepad.com/aws/2011/05/how-to-use-the-new-cloudwatch-user-defined-metrics-feature-with-php-1.html

Linux script sample custom CloudWatch metric for memory, disk space, and swap space:
http://aws.amazon.com/code/8720044071969977

More examples using the CloudWatch Command Interface to create custom metric:
http://blogs.clogeny.com/custom-metrics-in-amazon-cloudwatch/

Sample code for instance memory reporting:
#!/bin/bash export AWS_CLOUDWATCH_HOME=/home/ec2-user/CloudWatch-1.0.12.1 export AWS_CREDENTIAL_FILE=$AWS_CLOUDWATCH_HOME/credentials export AWS_CLOUDWATCH_URL=https://monitoring.amazonaws.com export PATH=$AWS_CLOUDWATCH_HOME/bin:$PATH export JAVA_HOME=/usr/lib/jvm/jre # get ec2 instance id instanceid=`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id` memtotal=`free -m | grep 'Mem' | tr -s ' ' | cut -d ' ' -f 2` memfree=`free -m | grep 'buffers/cache' | tr -s ' ' | cut -d ' ' -f 4` let "memused=100-memfree*100/memtotal" mon-put-data --metric-name "FreeMemoryMBytes" --namespace "System/Linux" --dimensions "InstanceId=$instanceid" --value "$memfree" --unit "Megabytes" mon-put-data --metric-name "UsedMemoryPercent" --namespace "System/Linux" --dimensions "InstanceId=$instanceid" --value "$memused" --unit "Percent"

Auto Scaling : scaling group and policy basics

When setting up auto scaling, you set the plan and then tie this plan to the auto scaling group.  The plan is set (tied to the auto scaling group) using different commands for each of the three types of plans.  The three option plans are:

• Maintain a Fixed Number of Running EC2 Instances
  Use this scaling plan if you would like Auto Scaling to maintain the minimum ( or the desired, if specified) number of instances in your Auto Scaling group at all times. You can manually change the number of running instances in your Auto Scaling group at any time.
  For this plan,  you don't need to do anything for it to take effect as you have already set the minimum and maximum number of instances in the auto scaling group.  If a desired capacity was set in the auto scaling group, this will be the minimum capacity. You can change the desired capacity by using the as-set-desired-capacity command.  You instances could be behind an ELB using this option plan.  The desired capacity could also be changed used using an SQS queue that is updated when the load on a process hits a certain threshold. 
• Scale Based on Demand
  Use this scaling plan if you need to scale dynamically in response to changes in the demand for your application. When you scale based on demand, you must specify when and how to scale.
  For this type of plan, you would control through CloudWatch and ELB (ELB is optional, CloudWatch is not).  First, you use the as-put-scaling-policy to set the scaling adjustment and how to scale up and down. Second, you attach the auto scaling policy to the CloudWatch alarm using the mon-put-metric-alarm.  See more below on setting the auto scaling policy.  When scaling on demand, you will must use CloudWatch to trigger the auto scaling policy.
• Scale Based on a Schedule
  Use this scaling plan if you want to scale your application on a pre-defined schedule. You can specify the schedule for scaling one time only or provide details for scaling on a recurring schedule.
  For this type of policy, you would use the as-put-scheduled-update-group-action on the auto scaling group (to tie the plan to the auto scaling group).   More can be found here: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/schedule_time.html.  You not use CloudWatch and you may or may not have the instances being scaled on a schedule behind an ELB. For example, with Peoplesoft month end payroll processing instances these would most likely not be behind an ELB.
  
  

Scale Based on Demand : Setting the auto scaling policy and 

The as-put-scaling-policy is used to set the scaling adjustment (in percentage or absolute number).  The scaling adjustment valid values are  ChangeInCapacity,ExactCapacity, and PercentChangeInCapacity.

ChangeinCapacity is used to increase or decrease existing capacity a certain amount. For example, let's say that the current capacity of your Auto Scaling group is set to five instances. You then create a scaling policy on your Auto Scaling group, specify the type as ChangeInCapacity and the adjustment as five. When the policy is executed, Auto Scaling will add five more instances to your Auto Scaling group. 

ExactCapacity is used to change the capacity to fixed number of instances. Let's say that the capacity of your Auto Scaling group is set to five instances. You then create a scaling policy on your Auto Scaling group and specify the type as ExactCapacity and the adjustment as two. When the policy is executed, your Auto Scaling group will have two running instances.

PercentChangeInCapacity (the most common one I see used) is used to increase or decrease the desired capacity by a percentage of the desired capacity. For example, let's say that the desired capacity of your Auto Scaling group is set to ten instances. You then create a scaling policy on your Auto Scaling group, specify the type as PercentChangeInCapacity, and the adjustment as twenty.  The capacity will be increase by two as two is twenty percent of 10.

The next step is to set the CloudWatch alarm that is invoke the policy.  More details here: http://cloudconclave.blogspot.com/2013/08/understanding-auto-scaling-policy-and.html

Understanding auto scaling policy and CloudWatch alarms

Taking a look at an example auto scaling policy you can see that there is nothing to indicate when to scale up or down:

as-put-scaling-policy reachforce-scale-in-policy --auto-scaling-group reachforce-as-grp --adjustment=20 --type PercentChangeInCapacity

ARM - arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:d9a6780b-3319-4b00-98f1-e98dcfc68d82:autoScalingGroupName/reachforce-as-grp:policyName/reachforce-scale-in-policy

This policy does specify the "how"; scale 20% (adjustment) for a percentage change in capacity (how much you will scale up or down as a percentage of current capacity).

This is controlled using a CloudWatch metrics alarm.  

mon-put-metric-alarm --alarm-name AddCapacity --metric-name CPUUtilization --namespace "AWS/EC2" --statistic "Average" --evaluation-periods 6 --period 120 --threshold 80 --comparison-operator GreaterThanOrEqualToThreshold --dimensions "AutoScalingGroupName=reachforce-as-grp"  --alarm-actions arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:69270ce4-3350-48f4-9d6f-71bc64225554:autoScalingGroupName/reachforce-as-grp:policyName/reachforce-scale-out-policy

This command ties a cloudwatch metrics to the auto scaling policy above.  The parameters that define the what and when are:
1. What : -metric-name CPUUtilization -- : CPU utilization is the most common metric used.  You also see the threshold of 80% for the CPU utilization.  So, when the CPU is on average over 80%.
2. When: Controlled by the evaluation periods, period and threshold:
A. Evaluation periods : The number of times to check if the alarm metric has been meet before triggering the alarm/policy.
B. Period : Is the length of time to check the alarm metric (in this case CPU utilization).  This value is in seconds.  So, in this case for 2 minutes.

If you look in the AWS CloudWatch console, you see this metrics threshold defined as:
the value is  80 for 12 minutes. This is because you specified 120 seconds for the periods
and 6 for the number of periods.
You also specify evaluation periods and period on the scale down alarm so you don't get constant
scaling up and down of the auto scaling group.

Auto Scaling Script

Auto Scaling with VPC 

This example uses the CLI version 1.0.61.2 (API 2011-01-01).  There is now a newer version of the CLI.

Overall notes:

1. Create an image (aka: gold image) that has a health check on it that consists of a simple static HTML page such as a ping.html or an index.html. 

2. The ELB (need an ELB before creating Auto Scaling group) hat has the instances running from your gold image AMI created in step 1. Could potentially (nothing is preventing you from doing) create a auto scale launch configuration (step 6 in process below) with an AMI that is different, with a different instance type as instance type is another parameter of your launch configuration, than the instance that will be part of your auto scaling group.  Both are possible but most will have the same AMI (gold image, so you know you have a health check) and the same instance type.

3. Before you create the ELB you should have instances running of the  gold image to make sure the ELB is working properly.  To provide true HA, make sure to put the instances in two AZs. (for example: us-west-2a, us-west-2b)

4. When creating elb that is using subnets, limitation on ELB is that only one subnet per AZ is allowed.  

5. When creating autoscaling group (as-create-auto-scaling-group)

 need to make sure to specific the VPC subnets and AZs.  The VPC subnets need to be in the AZs specified in the AZ parameter for command.

Steps:

Prework:

1. Identify the VPC : For example, VPC: vpc-9b120cf2

2. Add subnets : In this case, using two public subnets. I had to create the second one (subnet) which was private by default and I had to add an igw to make it public.

3. Subnets -

            A. Subnet: subnet-9f120cf6

            CIDR: 10.0.0.0/24   VPC: vpc-9b120cf2   Availability Zone: us-west-2a

            B. Subnet: subnet-8c130de5

            CIDR: 10.0.5.0/24   VPC: vpc-9b120cf2   Availability Zone: us-west-2b

4. Luanch two instances from AMI that has Apache installed with ping.html (or some other file) as a health check. One instance in subnet A and the other in subnet B. 

5. Create an ELB with the two instances across subnets.  ELB is healthy with two instances running in two AZs in two different public subnets. Limitation on ELB is that one subnet per AZ. 

6. Auto Scaling:

A. as-create-launch-config vpcautoscaling-as-lc --image-id ami-dcd344ec --instance-type t1.micro --key AutoScalingKey

B. as-create-auto-scaling-group vpcautoscaling-as-grp --launch-configuration vpcautoscaling-as-lc  --min-size 4 --max-size 12 --load-balancers VpcautoscalingAutoScalingELB --vpc-zone-identifier subnet-9f120cf6,subnet-8c130de5 --availability-zones us-west-2a,us-west-2b

C. as-describe-auto-scaling-groups --headers

output of command:

INSTANCE  INSTANCE-ID  AVAILABILITY-ZONE  STATE      STATUS   LAUNCH-CONFIG

INSTANCE  i-3af6e708   us-west-2b         InService  Healthy  vpcautoscaling-as-lc

INSTANCE  i-3cf6e70e   us-west-2a         InService  Healthy  vpcautoscaling-as-lc

INSTANCE  i-3ef6e70c   us-west-2a         InService  Healthy  vpcautoscaling-as-lc

INSTANCE  i-38f6e70a   us-west-2b         InService  Healthy  vpcautoscaling-as-lc

D. as-put-scaling-policy vpcautoscaling-scale-out-policy --auto-scaling-group vpcautoscaling-as-grp --adjustment=30 --type PercentChangeInCapacity

ARN: arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:69270ce4-3350-48f4-9d6f-71bc64225554:autoScalingGroupName/vpcautoscaling-as-grp:policyName/vpcautoscaling-scale-out-policy

E.  as-put-scaling-policy vpcautoscaling-scale-in-policy --auto-scaling-group vpcautoscaling-as-grp --adjustment=1 --type PercentChangeInCapacity

ARM - arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:d9a6780b-3319-4b00-98f1-e98dcfc68d82:autoScalingGroupName/vpcautoscaling-as-grp:policyName/vpcautoscaling-scale-in-policy

F. mon-put-metric-alarm --alarm-name AddCapacity --metric-name CPUUtilization --namespace "AWS/EC2" --statistic "Average" --evaluation-periods 6 --period 120 --threshold 80 --comparison-operator GreaterThanOrEqualToThreshold --dimensions "AutoScalingGroupName=vpcautoscaling-as-grp"  --alarm-actions arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:69270ce4-3350-48f4-9d6f-71bc64225554:autoScalingGroupName/vpcautoscaling-as-grp:policyName/vpcautoscaling-scale-out-policy

G. mon-put-metric-alarm --alarm-name RemoveCapacity --metric-name CPUUtilization --namespace "AWS/EC2" --statistic "Average" --evaluation-periods 2 --period 120 --threshold 40 --comparison-operator LessThanOrEqualToThreshold --dimensions "AutoScalingGroupName=vpcautoscaling-as-grp"  --alarm-actions arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:d9a6780b-3319-4b00-98f1-e98dcfc68d82:autoScalingGroupName/vpcautoscaling-as-grp:policyName/vpcautoscaling-scale-in-policy

H. as-describe-policies --auto-scaling-group vpcautoscaling-as-grp —headers

AWS Auto Scaling for batch processing

Can I use Auto Scaling for scaling out a set of servers that run periodically (auto scaling of zero) ?  This is not the typical auto scaling use case but it is possible.  This would most likely be used for use cases such as batch processing where instances would be spun up to support processing that happens periodically.  Here are some ways to solve this use case:

• Auto Scaling, SQS and CloudWatch : You can but it would require to use AWS SQS with Auto Scaling and you would set up an alarm that uses SQS instead of CloudWatch or a custom metrics.  See more here:
  http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/as-using-sqs-queue.html
  It is possible to set up an auto scaling group with min EC2 size of ):
  http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/US_BasicSetup.html. You can issue this command: as-create-auto-scaling-group my-test-asg --launch-configuration my-test-lc --availability-zones us-east-1a --min-size 0 --max-size 10 --desired-capacity 1
• OpsWorks : OpsWorks provides a graphical tool for scheduling time-based auto scaling (e.g. “start an instance every Monday at 8:00am and shut it down at 5:00p.m.”).
• Auto Scaling and Cron : The auto scaling API supports a cron-style "recurrence" attribute for scheduled actions:
  
  
  http://docs.aws.amazon.com/AutoScaling/latest/APIReference/API_PutScheduledUpdateGroupAction.html

Cloudwatch and benchmarking

Question: Can Cloudwatch provide enough metrics for an Oracle DB benchmarks? What are other tools that can be used?

Response: Cloud really does not provide much in terms of Oracle DB metrics.  Oracle AWR is the best option.  There is also a new OEM AWS plug for AWS:

http://www.oracle.com/technetwork/oem/grid-control/downloads/oem-aws-plugin-1852739.html

Of course, there is the performance pack that is part of OEM.