AWS Import/Export file limits

AWS import/Export can handle volumes of larger then 1TB to be stored on Amazon EBS volumes. However, there is a intermediate step using S3.  If your storage device’s capacity is less than or equal to the maximum Amazon EBS volume size of 1TB, its contents will be loaded directly into an Amazon EBS snapshot. So, in theory no size limit. AWS does not mount the file system on your storage device, nor is a file system required to be present. AWS Import/Export performs a block for block copy from your device to an Amazon EBS Snapshot. If your storage device’s capacity exceeds 1TB, a device image will be stored within your specified Amazon S3 log bucket. You can then create a RAID of EBS volumes using software such as Logical Volume Manager, and copy the image from Amazon S3 to this new volume

Hadoop Cluster and Amazon EMR

Accenture calculated the total cost of ownership of a bare-metal Hadoop cluster and derived the capacity of nine different cloud-based Hadoop clusters at the matched TCO. The performance of each option was then compared by running three real-world Hadoop applications.  The full report is here:

http://www.accenture.com/us-en/Pages/insight-hadoop-deployment-comparison.aspx

There report compares the performance of both a bare-metal Hadoop cluster and Amazon ElasticMapReduce (Amazon EMR).

Auto Scaling : scaling group and policy basics

When setting up auto scaling, you set the plan and then tie this plan to the auto scaling group.  The plan is set (tied to the auto scaling group) using different commands for each of the three types of plans.  The three option plans are:

• Maintain a Fixed Number of Running EC2 Instances
  Use this scaling plan if you would like Auto Scaling to maintain the minimum ( or the desired, if specified) number of instances in your Auto Scaling group at all times. You can manually change the number of running instances in your Auto Scaling group at any time.
  For this plan,  you don't need to do anything for it to take effect as you have already set the minimum and maximum number of instances in the auto scaling group.  If a desired capacity was set in the auto scaling group, this will be the minimum capacity. You can change the desired capacity by using the as-set-desired-capacity command.  You instances could be behind an ELB using this option plan.  The desired capacity could also be changed used using an SQS queue that is updated when the load on a process hits a certain threshold. 
• Scale Based on Demand
  Use this scaling plan if you need to scale dynamically in response to changes in the demand for your application. When you scale based on demand, you must specify when and how to scale.
  For this type of plan, you would control through CloudWatch and ELB (ELB is optional, CloudWatch is not).  First, you use the as-put-scaling-policy to set the scaling adjustment and how to scale up and down. Second, you attach the auto scaling policy to the CloudWatch alarm using the mon-put-metric-alarm.  See more below on setting the auto scaling policy.  When scaling on demand, you will must use CloudWatch to trigger the auto scaling policy.
• Scale Based on a Schedule
  Use this scaling plan if you want to scale your application on a pre-defined schedule. You can specify the schedule for scaling one time only or provide details for scaling on a recurring schedule.
  For this type of policy, you would use the as-put-scheduled-update-group-action on the auto scaling group (to tie the plan to the auto scaling group).   More can be found here: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/schedule_time.html.  You not use CloudWatch and you may or may not have the instances being scaled on a schedule behind an ELB. For example, with Peoplesoft month end payroll processing instances these would most likely not be behind an ELB.
  
  

Scale Based on Demand : Setting the auto scaling policy and 

The as-put-scaling-policy is used to set the scaling adjustment (in percentage or absolute number).  The scaling adjustment valid values are  ChangeInCapacity,ExactCapacity, and PercentChangeInCapacity.

ChangeinCapacity is used to increase or decrease existing capacity a certain amount. For example, let's say that the current capacity of your Auto Scaling group is set to five instances. You then create a scaling policy on your Auto Scaling group, specify the type as ChangeInCapacity and the adjustment as five. When the policy is executed, Auto Scaling will add five more instances to your Auto Scaling group. 

ExactCapacity is used to change the capacity to fixed number of instances. Let's say that the capacity of your Auto Scaling group is set to five instances. You then create a scaling policy on your Auto Scaling group and specify the type as ExactCapacity and the adjustment as two. When the policy is executed, your Auto Scaling group will have two running instances.

PercentChangeInCapacity (the most common one I see used) is used to increase or decrease the desired capacity by a percentage of the desired capacity. For example, let's say that the desired capacity of your Auto Scaling group is set to ten instances. You then create a scaling policy on your Auto Scaling group, specify the type as PercentChangeInCapacity, and the adjustment as twenty.  The capacity will be increase by two as two is twenty percent of 10.

The next step is to set the CloudWatch alarm that is invoke the policy.  More details here: http://cloudconclave.blogspot.com/2013/08/understanding-auto-scaling-policy-and.html