AWS re:Invent enterprise sessions

DAT202 - Using Amazon RDS to Power Enterprise Applications Amazon Relational Database Service (Amazon RDS) makes it cheap and easy to deploy, manage, and scale relational databases using a familiar MySQL, Oracle or MS SQL server database engine. Amazon RDS can be an excellent choice for running many large, off-the-shelf enterprise applications from companies like JD Edwards, Oracle, PeopleSoft, and Siebel. Sign up for this session to learn how to best leverage Amazon RDS for use with enterprise applications and learn about best practices and data migration strategies.

DAT401 - Advanced Data Migration Techniques for Amazon RDS Migrating data from the existing environments to AWS is a key part of the overall migration to Amazon RDS for most customers. Moving data into Amazon Relational Database Service (Amazon RDS) from existing production systems in a reliable, synchronized manner with minimum downtime requires careful planning and the use appropriate tools and technologies. Because each migration scenario is different in terms of source and target systems, tools, and data sizes, you'll need to customize your data migration strategy to achieve the best outcome. In this session we will do a deep dive into various methods, tools and technologies that can be put to use for a successful and timely data migration to Amazon RDS.

STG301 - AWS Storage Tiers for Enterprise Workloads - Best Practices Enterprise environments utilize many different kinds of storage technologies from different vendors to fulfill various needs in their IT landscape. These are often very expensive and procurement cycles quite lengthy. They also need specialized expertise in each vendor's storage technologies to configure them and integrate them into the ecosystem, again resulting in prolonged project cycles and added cost. AWS provides end-to-end storage solutions that will fulfill all these needs of Enterprise Environments that are easily manageable, extremely cost effective, fully integrated and totally on demand. These storage technologies include Elastic Block Store (EBS) for instance attached block storage, Amazon Simple Storage Service (Amazon S3) for object (file) storage and Amazon Glacier for archival. An enterprise database environment is an excellent example of a system that could use all these storage technologies to implement an end-to-end solution using striped PIOPS volumes for data files, Standard EBS volumes for log files, S3 for database backup using Oracle Secure Backup and Glacier for long-time archival from S3 based on time lapse rules. In this session, we will explore the best practices for utilizing AWS storage tiers for enterprise workloads

STG305 - Disaster Recovery Site on AWS - Minimal Cost Maximum Efficiency Implementation of  disaster recovery (DR) site is crucial for business continuity of any enterprise. Due to the fundamental nature of features like elasticity, scalability and geographic distribution, DR implementation on AWS can be done at 10-50% of the conventional cost. In this session, we will do a deep dive into proven DR architectures on AWS and best practices, tools and techniques get the most out of them.

STG303 - Running Microsoft and Oracle Stacks on Elastic Block Store Run your Enterprise applications on Elastic Block Store (EBS). This session will discuss how you can leverage the block storage platform (EBS) as you move your Microsoft (SQL Server, Exchange, SharePoint) and Oracle (Databases, E-business Suite, Business Intelligence) workloads onto Amazon Web Services (AWS). The session will cover high availability, performance, and backup/restore best practices

ENT303 - Migrating Enterprise Applications to AWS - Best Practices, Tools and Techniques In this session we will discuss strategies, tools and techniques for migrating enterprise software systems to AWS. We'll consider applications like Oracle eBusiness Suite, SAP, PeopleSoft, JD Edwards and Siebel. These applications are complex by themselves; they are frequently customized; they have many touch points on other systems in the enterprise; and they often have large associated databases. Nevertheless, running enterprise applications in the cloud affords powerful benefits. We'll identify success factors and best practices.

AWS Life Sciences event

App Associates AWS Life Sciences event in Boston on Sept 19TH:

http://www.appsassociates.com/events-register/roadmap-to-cloud-for-life-sciences/

AWS RDS : Changing OS time zone

This entry helps you change the database time zone: http://cloudconclave.blogspot.com/2013/08/aws-rds-timezone.html

However, it does not address changed the operating system time zone.  With RDS, the OS timezone can’t be changed (UTC, by design).  Each database engine supports different functions that query OS (for Oracle it is sysdate),   database engine, or session level parameters for timezone information . 

Here’s a quick synopsis of each DB engine’s datetime function details which identifies which calls are OS, database instance, or session based and which include timezone information or if its implied:

·         MySQL: http://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html

·         Oracle: http://docs.oracle.com/cd/B19306_01/server.102/b14225/ch4datetime.htm#g1010984

·         SQL Server: http://msdn.microsoft.com/en-us/library/ms186724.aspx

Oracle on AWS at partner event in SF

AWS partner event in downtown SF with an Oracle Solutions on AWS session:
http://awssipartnereventsf-awsemail1.eventbrite.com

Oracle WebLogic and Database on AWS

Presented at reInvent in 2012.  Full session here:
http://www.youtube.com/watch?v=R1AjAKz73M8

McGraw-Hill success story starts here:

http://www.youtube.com/watch?feature=player_detailpage&v=R1AjAKz73M8#t=2617

Oracle WebLogic and RAC on AWS architecture slide:

http://www.youtube.com/watch?feature=player_detailpage&v=R1AjAKz73M8#t=3192

AWS RDS timezone

Changing the time zone can be found here:

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.Oracle.CommonDBATasks.html

Near the bottom you will find this:

Setting the Database Time Zone

You can alter the time zone of a database by running the rdsadmin_util procedure as shown in the following table. You can use any time zone name or GMT offset that is accepted by Oracle.

Oracle Method	Amazon RDS Method
alter database set time_zone = '+3:00';	exec rdsadmin.rdsadmin_util.alter_db_time_zone('+3:00');

After you alter the time zone, you must reboot the DB Instance for the change to take effect.

There are additional restrictions on setting time zones listed in the Oracle documentation.

AWS : Storing Session State

AWS SimpleDB, Memcache and DynamoDB can all be used.  DynamoDB is a good option as there is already a session provider for DynamoDB :

http://aws.typepad.com/aws/2012/09/managing-aspnet-session-state-with-dynamodb.html

For SQL Server, you can also look at session management in SQL server for persistence  and use built in .net session provider modules.

You can also manage session state using AWS RDS for SQL Server, MySQL or Oracle.

AWS Dedicated Instances

Here is the general/high level information on dedicated instances:

http://aws.amazon.com/dedicated-instances/

Dedicated instances look like any other instance in the AWS Console.  You create a dedicated instance using the console and the only two things you need to do:

1. Launch within a VPC (dedicated instances can only run in a VPC).
2. Select the tenancy as 'dedicated'. 

You can move instances between dedicated and 'regular AWS' and 'regular AWS' and dedicated by snapshotting just like you would do with 'regular AWS' instances.

More on the recent price reduction:

http://aws.typepad.com/aws/2013/07/ec2-dedicated-instance-price-reduction.html

Oracle Database on AWS Best Practices Session time and location set

Session ID: CON4728
Session Title: Best Practices for Running Oracle Database Instances on Amazon Web Services EC2
Venue / Room: Marriott Marquis - Salon 7
Date and Time: 9/23/13, 15:15 - 16:15

Full session details here:
http://cloudconclave.blogspot.com/2013/08/openworld-oracle-on-aws-sessions.html

AWS IAM user creation and access

IAM user set up can be found here:
http://docs.aws.amazon.com/IAM/latest/UserGuide/Using_SettingUpUser.html

Give them 'Power User' if they need to create instances and fully utilize AWS services.

They will use a different URL (not aws.amazon.com) then you use to access your account.  More here:
http://docs.aws.amazon.com/IAM/latest/UserGuide/WhatUsersNeedToKnow.html

Migrate Your Business Critical Oracle Applications to the AWS Cloud Webinar

Smarter Agent, AWS and smartShift Technologies provides an overview of how they fully migrated their customer's, Smarter Agent, Oracle database to AWS to provide a scalable and cost effective hybrid implementation.

http://www.youtube.com/watch?v=t2UcCdnNsRc&feature=youtu.be

ELB load balancing algorithm

ELB does round robin load balancing.  However, if you are testing ELB and all of your traffic is going to the same instance, don't be surprised.  This is because Amazon ELB behaves little strange when incoming traffic is originated from Single or  Specific IP ranges, it does not efficiently do round robin and sticks the request to some EC2's only. 

EBS data transfer costs to snapshot Oracle DB to S3

Obviously, there is a cost to snapshot your EBS volumes of your Oracle database and store them in S3 for backup and recovery.  This is the standard S3 cost of $.095 GB a month.  The cost can go down to $.055 when you store more data.  There is no cost to transfer the data in and out of S3 to and from EC2. 

Multiple SSL certificates for ELB

You can have two certificates on two different ELB ports. You CAN'T have two certs on a single port. Most folks would want to use port 443 for SSL traffic so using another port may not be an option. You can use SAN or wildcard cert that can handle more than a single domain and assign that to a single port. 

Certificates are managed using IAM. More details here:http://docs.aws.amazon.com/IAM/latest/UserGuide/ManagingServerCerts.html

Oracle MySQL Connect session time set

The session I am co-presenting at has been set for this time and place:

Session ID: CON4513
Session Title: Best Practices for Deploying MySQL on Amazon Web Services
Venue / Room: Hilton - Powell
Date and Time: 9/21/13 (Saturday), 16:00 - 17:00

Hope to see you there.

AWS Services that need a IGW, NAT instance or VPN server to access in VPC

The the following services can NOT be accessed via a private IP address in your VPC. Therefore, they require the use of the AWS Internet Gateway (or NAT instance):

1. EMR : Because access to and from the AWS cloud is a requirement of the cluster, you must connect an Internet gateway to the VPC subnet hosting the cluster. If your application has components you do not want connected to the Internet gateway you can launch those components in other subnets you create within your VPC. In addition, because of the need to access the AWS cloud, you cannot use Network Address Translation (NAT) when you are running Amazon EMR on a VPC.
2. S3 : This is straight forward as S3 is accessed via a URL.  Therefore, the requests hits the IGW and accesses the S3 bucket.  NAT can not be used here.
3. DynamoDB : The AWS API endpoints are external to a VPC and the instance requires an Internet connection in order to reach them. You can either assign an Elastic IP and route the traffic directly out through the Internet Gateway, or use a NAT instance. The latter makes it possible for instances in private subnets to get access to the Internet. These instances will not need any public IP addresses. Instead, they go out to the Internet through a NAT instance in your public subnet.

Requires IGW or VPN server:
4. VPC to VPC : You can use an open source VPN server like OpenVPN (this allows you to not open up your instances by placing them in a public subnet and using elastic IPs).  You could use IGW with elastic IPs attached to the instances.

Security on AWS

Good overview of top security tips for running on AWS:
http://stratumsecurity.com/2012/12/03/practical-tactical-cloud-security-ec2/

AWS instance types and instance live migration

Interesting comment by Netflix CEO when asked about 'big tech trends' in the next five years:
1. Instance types : No need to select instance types.
2. Live migration of EC2 instances

Interesting that the response was AWS focused.  See video here:
http://www.youtube.com/watch?feature=player_detailpage&v=8FJ5DBLSFe4&t=2441

Boot strapping with user data and CloudFormation

For simple bootstrapping, user data text/scripts may be adequate.  Keep in mind the limit on size is 16K for user data.

s3cmd is often used to load the bootstrap scripts for S3. More on this can be found here:

http://s3tools.org/s3cmd

https://github.com/s3tools/s3cmd

A very good document on using user data, CloudFormation, Chef, Puppet and other tools to bootstrap EC2 instances can be found here:

https://s3.amazonaws.com/cloudformation-examples/BoostrappingApplicationsWithAWSCloudFormation.pdf

Data warehousing load times

Some times when people are moving to AWS for their databases they are looking for load times in the range of 2 TB per hour (this is 650+ MB/second).  

1. Green Plum : http://www.emc.com/collateral/hardware/data-sheet/h7419-greenplum-dca-ds.pdf : 10 TB to 60 TB per hour claimed in this document.

2. Exadata : http://www.oracle.com/ocom/groups/public/@otn/documents/webcontent/173705.pdf  Data Load Rate: Up to 12 TB/hour

Numbers listed in this book:  http://books.google.com/books?id=Whx9nldv8UEC&pg=PA102&lpg=PA102&dq=oracle+database+loads+terabytes+per+hour&source=bl&ots=VEXelUIhd2&sig=9LW4T0PgottDF2kpgvUobd2FonQ&hl=en&sa=X&ei=PHjyUcPKMuKbygHuyIDgBg&ved=0CHkQ6AEwBw#v=onepage&q=oracle%20database%20loads%20terabytes%20per%20hour&f=false

Of course,  Greenplum and Exadata are significant monetary investments.

Paraccel claims to achieve 2 TB an hour: http://www.paraccel.com/resources/Analyst-Reports/TDWI_Checklist-Report_Analytic-Databases-for-Big-Data.pdf

EMR : Pig, Hive

EMR has support for Pig and Hive:

http://aws.amazon.com/about-aws/whats-new/2013/08/02/amazon-elastic-mapreduce-now-supports-hive-0.11.0-and-pig-0.11.1/

Traffic costs between EC2 instances, AZs, and VPCs

Often times when running on AWS you will have multiple AZs, accounts and VPCs.  This is a simple summary of EC2 data transfer costs:

1. Within the same AZ  : Free: It is within the same AZ and over private IP.
2. Between two AZs :   Cost. Will incur inter-az charges for both inbound and outbound

3. Between Accounts in different VPCs :  Cost. Inter-az charges are applicable for data transfer charges between instances in the different AZ's (same region) and in different accounts.
4. Between one EC2 classic instance and VPC EC2 instance in a same AZ : Cost
5. Instances to S3 in the same region: Free
6. Instance to S3 in different region: Cost. cross region charges

Auto Scaling : scaling group and policy basics

When setting up auto scaling, you set the plan and then tie this plan to the auto scaling group.  The plan is set (tied to the auto scaling group) using different commands for each of the three types of plans.  The three option plans are:

• Maintain a Fixed Number of Running EC2 Instances
  Use this scaling plan if you would like Auto Scaling to maintain the minimum ( or the desired, if specified) number of instances in your Auto Scaling group at all times. You can manually change the number of running instances in your Auto Scaling group at any time.
  For this plan,  you don't need to do anything for it to take effect as you have already set the minimum and maximum number of instances in the auto scaling group.  If a desired capacity was set in the auto scaling group, this will be the minimum capacity. You can change the desired capacity by using the as-set-desired-capacity command.  You instances could be behind an ELB using this option plan.  The desired capacity could also be changed used using an SQS queue that is updated when the load on a process hits a certain threshold. 
• Scale Based on Demand
  Use this scaling plan if you need to scale dynamically in response to changes in the demand for your application. When you scale based on demand, you must specify when and how to scale.
  For this type of plan, you would control through CloudWatch and ELB (ELB is optional, CloudWatch is not).  First, you use the as-put-scaling-policy to set the scaling adjustment and how to scale up and down. Second, you attach the auto scaling policy to the CloudWatch alarm using the mon-put-metric-alarm.  See more below on setting the auto scaling policy.  When scaling on demand, you will must use CloudWatch to trigger the auto scaling policy.
• Scale Based on a Schedule
  Use this scaling plan if you want to scale your application on a pre-defined schedule. You can specify the schedule for scaling one time only or provide details for scaling on a recurring schedule.
  For this type of policy, you would use the as-put-scheduled-update-group-action on the auto scaling group (to tie the plan to the auto scaling group).   More can be found here: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/schedule_time.html.  You not use CloudWatch and you may or may not have the instances being scaled on a schedule behind an ELB. For example, with Peoplesoft month end payroll processing instances these would most likely not be behind an ELB.
  
  

Scale Based on Demand : Setting the auto scaling policy and 

The as-put-scaling-policy is used to set the scaling adjustment (in percentage or absolute number).  The scaling adjustment valid values are  ChangeInCapacity,ExactCapacity, and PercentChangeInCapacity.

ChangeinCapacity is used to increase or decrease existing capacity a certain amount. For example, let's say that the current capacity of your Auto Scaling group is set to five instances. You then create a scaling policy on your Auto Scaling group, specify the type as ChangeInCapacity and the adjustment as five. When the policy is executed, Auto Scaling will add five more instances to your Auto Scaling group. 

ExactCapacity is used to change the capacity to fixed number of instances. Let's say that the capacity of your Auto Scaling group is set to five instances. You then create a scaling policy on your Auto Scaling group and specify the type as ExactCapacity and the adjustment as two. When the policy is executed, your Auto Scaling group will have two running instances.

PercentChangeInCapacity (the most common one I see used) is used to increase or decrease the desired capacity by a percentage of the desired capacity. For example, let's say that the desired capacity of your Auto Scaling group is set to ten instances. You then create a scaling policy on your Auto Scaling group, specify the type as PercentChangeInCapacity, and the adjustment as twenty.  The capacity will be increase by two as two is twenty percent of 10.

The next step is to set the CloudWatch alarm that is invoke the policy.  More details here: http://cloudconclave.blogspot.com/2013/08/understanding-auto-scaling-policy-and.html

Understanding auto scaling policy and CloudWatch alarms

Taking a look at an example auto scaling policy you can see that there is nothing to indicate when to scale up or down:

as-put-scaling-policy reachforce-scale-in-policy --auto-scaling-group reachforce-as-grp --adjustment=20 --type PercentChangeInCapacity

ARM - arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:d9a6780b-3319-4b00-98f1-e98dcfc68d82:autoScalingGroupName/reachforce-as-grp:policyName/reachforce-scale-in-policy

This policy does specify the "how"; scale 20% (adjustment) for a percentage change in capacity (how much you will scale up or down as a percentage of current capacity).

This is controlled using a CloudWatch metrics alarm.  

mon-put-metric-alarm --alarm-name AddCapacity --metric-name CPUUtilization --namespace "AWS/EC2" --statistic "Average" --evaluation-periods 6 --period 120 --threshold 80 --comparison-operator GreaterThanOrEqualToThreshold --dimensions "AutoScalingGroupName=reachforce-as-grp"  --alarm-actions arn:aws:autoscaling:us-west-2:649163059618:scalingPolicy:69270ce4-3350-48f4-9d6f-71bc64225554:autoScalingGroupName/reachforce-as-grp:policyName/reachforce-scale-out-policy

This command ties a cloudwatch metrics to the auto scaling policy above.  The parameters that define the what and when are:
1. What : -metric-name CPUUtilization -- : CPU utilization is the most common metric used.  You also see the threshold of 80% for the CPU utilization.  So, when the CPU is on average over 80%.
2. When: Controlled by the evaluation periods, period and threshold:
A. Evaluation periods : The number of times to check if the alarm metric has been meet before triggering the alarm/policy.
B. Period : Is the length of time to check the alarm metric (in this case CPU utilization).  This value is in seconds.  So, in this case for 2 minutes.

If you look in the AWS CloudWatch console, you see this metrics threshold defined as:
the value is  80 for 12 minutes. This is because you specified 120 seconds for the periods
and 6 for the number of periods.
You also specify evaluation periods and period on the scale down alarm so you don't get constant
scaling up and down of the auto scaling group.

VPC and AZs when using ELB and Auto Scaling

Here are a couple of things to be aware of when using Auto Scaling, VPC, and ELB with Auto Scaling:

1. There is a limitation on ELB in regards to AZs and subnets.   You can only have one subnet per AZ.  You will most likely be creating an ELB (and using VPC) when using Auto Scaling so it is important to understand this.
2. When creating an autoscaling group (as-create-auto-scaling-group), make sure to specific the VPC subnets and AZs if you are using .  The VPC subnets need to be in the AZs specified in the AZ parameter for the as-create-auto-scaling-group command. No checking is done to make sure this is true so it is up to you to make sure these AZs actually exist in the subnets you specify. 

ELB primer : A good place to start

If you are just starting off with ELB or even if you have worked with it for some time, here is a helpful blog post:

http://harish11g.blogspot.com/2012/07/aws-elastic-load-balancing-elb-amazon.html

I like these points in particular: 

Point 4) Amazon ELB is not designed for sudden load spikes /Flash traffic 

Note: Not for traffic that changes ever few seconds or even every few minutes.

Point 8) Amazon ELB cannot do Multi AWS Region Load Balancing

Note: Use Route53

Point 9) Amazon ELB sticks request when traffic is generated from Single IP

Note: I see this a lot in training classes as students are always hitting from same IP address.

Point 12) Amazon ELB can easily support more than 20K+ Concurrent reqs/sec

Note: In most cases, one ELB can support multiple systems.

AWS RDS caching query results

The first place to look when attempting to speed query performance when running AWS RDS MySQL is the MySQL query cache size:
http://survivalguides.wordpress.com/2012/07/11/change-the-query-cache-size-amazon-aws-rds/

With support of MySQL 5.6 for AWS RDS, MySQL RDS now has support for memcached:
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.MySQL.Options.html

Running web application on AWS

In most cases, you will not just be running just an Oracle database on AWS.  You will be running a web based application.  For Oracle focused developers and engineers, running web applications on AWS is not an area of expertise. Here are the best documents, documentation, and diagrams to review for developing web based applications on AWS:

http://media.amazonwebservices.com/architecturecenter/AWS_ac_ra_web_01.pdf

http://media.amazonwebservices.com/AWS_Web_Hosting_Best_Practices.pdf

http://docs.aws.amazon.com/gettingstarted/latest/wah-linux/web-app-hosting-intro.html

Support for session state and database failover when application server goes down are a couple of details that most Oracle folks ask about.  Information can be found here:

Session state: Good blog post here: http://harish11g.blogspot.com/2012/10/web-session-synchronization-aws-ec2.html . The last option is DynamoDB which is the method used most often.

Database failover:

1. Using RDS this is automatic: When automatic failover occurs, your application can remain unaware of what's happening behind the scenes. The CNAME record for your DB instance will be altered to point to the newly promoted standby. Your client library must be able to close and reopen the connection in the event of a failover.
2. Using Oracle on EC2 : You would use Elastic Ips and use this pattern : http://aws.amazon.com/articles/2127188135977316 . You cloud use ENIs so private Ips could be used http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#create-a-low-budget-high-availability-solution

OpenWorld Oracle on AWS sessions

Here are the two sessions focused on Oracle on AWS at Oracle OpenWorld:

https://oracleus.activeevents.com/2013/connect/search.ww?eventRef=openworld#loadSearch-event=null&searchPhrase=Amazon&searchType=session&tc=0&sortBy=titleSort&p=&i(11180)=20800

I will be co-presenting this session: Best Practices for Running Oracle Database Instances on Amazon Web Services.

MySQL Session at MySQL Connect

Here are the two MySQL on AWS session an MySQL Connect in September:

https://oracleus.activeevents.com/2013/connect/search.ww?eventRef=mysqlconnect#loadSearch-event=null&searchPhrase=aws&searchType=session&tc=0&sortBy=&p=&i(11180)=20802

I will be co-presenting the Best Practices for Deploying MySQL on Amazon Web Services session.

Blocking traffic from specific countries

There is nothing 'out of  the box' from AWS.  You can do this with CloudFront’s private content feature, but still need to use a their-party geo-ip database like Maxmind.  There is a tutorial here last: http://aws.typepad.com/aws/2012/01/guest-post-geo-blocking-content-with-amazon-cloudfront.html.