On premise application replication and snap shots to AWS for DR

When devising an disaster recovery approach that used AWS for DR of an on premise environment, there are literally hundreds of options for keeping the databases in sync. Each relational database vendor will have five of their own options and their partner ecosystems will offer many more.  For the application, it is different situation.  Their are not many features built into business application  and application servers.  Depending on what DR scenario you are using http://cloudconclave.blogspot.com/2012/11/aws-four-dr-scenarios.html you will be using either replication or snapshots to keep your AWS application in sync with your on premise environment.  
For warm standby and active-active you will need replication:

• RSync : Rsync is cheap (free) and easy.  Rsync can be chatty, does not have compression built in, and you have to code parallelism. 
• BitTorrent : BitTorrent is most often associated with moving large media files over the Internet in a fast matter. BitTorrent now has file syncing capabilities so can be used to sync application server files.  
• Amazon Storage Gateway " attach these volumes as iSCSI devices to your on-premises application servers.  These files will be stored in S3 where they can be instantiate as EBS volumes.

Riverbed Whitewater - http://www.riverbed.com/products-solutions/products/cloud-storage-whitewater/ https://aws.amazon.com/marketplace/pp/B007O0FXW4/ref=srh_res_product_title?ie=UTF8&sr=0-2&qid=1366938137611#product-detail

Attunity CloudBeam - http://www.attunitycloudbeam.com/solutions/disaster-recovery  https://aws.amazon.com/marketplace/pp/B00B5PB8IM/ref=srh_res_product_title?ie=UTF8&sr=0-3&qid=1366938015695

For backup and restore and pilot light you can use snapshots. 

• FTP or SFTP : FTP and SFTP are the two most common methods to transfer files over the internet.
• HTTP or HTTPs : HTTP and HTTPS can be used but they are slower than S/FTP.  However, if you don't want to require or can not have an FTP client in use than HTTP(s) can be used.  Some companies may not want to open up another port to the outside world have an FTP server running.
• AWS import/export : If files don't need to be synced that often, FedEx, UPS, or Post Office can be the least expensive method with the least amount of hassle (limited coding, no restarting needed)
• Attunity CloudBeam to move files into S3 and than instantiate as EBS volumes.

Oracle EBusiness Suite does offer rapid clone http://docs.oracle.com/cd/E18727_01/doc.121/e12841/T120505T120517.htm but this is not for moving production system data files but for setting test, dev, or moving a system to another machine. Weblogic also offers a cloning feature :http://docs.oracle.com/cd/E12839_01/core.1111/e10105/clone.htm

Determining if you are running Oracle VM on EC2

The question has come up on how to determine whether an EC2 instances started from an Oracle AMI is running OVM or Amazon Xen.  You can not find this out on the AWS management console.  You must use the AWS EC2 CLI.  The ec2-describe-instances command shows 'ovm' next to virtualization type (paravitual) or 'xen' of Amazon xen.

Oracle on AWS partner accreditation

In June, partners will be able to take on line training to get accreditation as an Oracle on AWS partner:
http://aws.amazon.com/partners/overview/partner-training/

AWS resource tagging

Tagging of resources is great for billing and managing your resources. You also need to be aware of the EC2 and networking services and resources that do not support tagging:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions

AWS EC2 public IP address ranges

This question gets asked a lot:
https://forums.aws.amazon.com/ann.jspa?annID=1701

A technical book in five days

They wrote a book in five days!  I would like to more of details...how much content did they already have, how many editors did they use, how much prep was already done??

http://www.openstack.org/blog/2013/03/we-did-it-zero-to-book-in-five-days/

Horizontal Scaling of Oracle on AWS

Just came across this solution.
Parelastic: http://www.parelastic.com (MySQL only at this time)

I really enjoyed the 'sharding sucks' mantra.  Being an 'old school' Oracle guy, the new world answer to  database scaling is sharding...I think that is very naive.

I know of these two companies in this space as well:

Scalearc (MySQL and SQL Server)

Scalebase : http://www.scalebase.com (MySQL only at this time)

CloudFormation, OpsWorks and Elastic Beanstalk

Now that AWS has announced OpsWorks and also has CloudFormation and Elastic Beanstalk there are questions on which application and infrastructure management tools is best for you.  Through in thinks like Chef and Puppet and things get really interesting. More can be found here:

https://aws.amazon.com/application-management/

SSD for Oracle Databases

Here are a couple of articles/web site with the pros and cons of running your Oracle database on SSD:

http://www.pythian.com/blog/de-confusing-ssd-for-oracle-databases/
http://www.slideshare.net/gwenshap/ssd-collab13

If your Oracle database is less than 2 TB you could run it on the  hi1.4xlarge instance type.  This instance type has  2 SSD-based volumes each with 1024 GB of instance storage.  More on AWS instance types here: http://aws.amazon.com/ec2/instance-types/

You could also just use Oracle RDS on AWS.  With PIOPS for RDS, you can get up to 25K IOPS on your Oracle database. 
Instagram uses SSD : http://m.cio.com/article/716829/SSDs_Boost_Instagram_39_s_Speed_on_Amazon_EC2

Importing on premise Oracle Database into AWS RDS

There is an article here but does not get into specifics on the performance of each option. http://aws.amazon.com/articles/4173109646282306

Oracle data pump is one of the fastest ways to move data between Oracle databases.  However, the approach shown in the above article uses database links which can be very slow.  A more performant method would be to use data pump file mode. Unfortunately, AWS Oracle RDS does not off access to the OS.  There is a workaround to this by using a little known feature of the Oracle database called Oracle Managed File System

http://docs.oracle.com/cd/B13789_01/server.101/b10739/omf.htm#i1006122

Oracle RDS has access to this Oracle Managed File (OMF) system. Once the file exported from the on premise database and moved to the Oracle RDS OMF you can use the  UTL_FILE capability in the database and import a file into the RDS instance, and then do impdb (data pump import) from there.

Things to keep in mind:

1.       Since the file sits on the Oracle managed file system, it consumes the space allocated for the database. So you will need to make sure you allocate enough space for the instance

2.       Once you do the impdb operation, you can delete the file but till then it will consume space on the instance

3.       File sizes will need to be less than 400-500 GB since the largest instance currently is 1 TB, and you will need space for both the file as well as the database

4.       As you do the import, you will consume IOPS for reading the dump file and then writing to the database.

AWS Boston May User Group meeting

Jeff Barr from AWS will be at this meeting:

http://www.meetup.com/The-Boston-Amazon-Web-Services-Meetup-Group/events/112500012/

Customer Gateway IP address

Most enterprise Oracle customers will create an IPSec VPN tunnel from their data center to AWS.  A customer gateway is required at the customer location/data center and a virtual private gateway on the AWS side.   The customer gateway IP address will be the public external IP address of the on-premise device (supported devices can be found here:http://aws.amazon.com/vpc/faqs/#C8). For example, if the public IP of the customer device is 203.200.25.8 and you plan to use this device to connect to AWS via VPC, then you will need to use this IP as the customer gateway.

Oracle Enterprise Applications on AWS web cast

Oracle Enterprise Applications on AWS Join us for a Webinar on May 3 Space is limited. Reserve your Webinar seat now at: https://www2.gotomeeting.com/register/973888266 Come join us and get a good introduction to running Oracle databases & applications on Amazon Web Services platform.  Hear the answers to commonly asked questions like: Which Oracle products are available and possible to run on AWS?  What does support, pricing and licensing models look like? How to size Oracle on AWS? Title: Oracle Enterprise Applications on AWS Date: Friday, May 3, 2013 Time: 10:00 AM - 11:00 AM PDT After registering you will receive a confirmation email containing information about joining the Webinar. System Requirements PC-based attendees Required: Windows® 7, Vista, XP or 2003 Server Mac®-based attendees Required: Mac OS® X 10.6 or newer Mobile attendees Required: iPhone®, iPad®, Android™ phone or Android tablet

AWS RDS database encryption

There a couple of ways to encrypt you data at rest when using AWS RDS:

1. Oracle RDS TDE support : http://aws.amazon.com/about-aws/whats-new/2013/04/18/amazon-rds-oracle-encryption
2. Oracle RDS with Ciphercloud : http://www.ciphercloud.com/database-encryption-and-tokenization.aspx.  AWS specific here (and mentions HIPPA) : http://www.ciphercloud.com/aws-encryption-ciphercloud.aspx

Oracle TDE also supports encryption at the network tier. 

EBS volume encryption

Here are two companies that offer EBS volume encryption:

1. SafeNet ProtectV

     A. Support for boot volume encryption

     B. Good integration with Hardware Security Modules (HSM)

     C. Somewhat limited OS support  

2. Trend Micro SecureCloud

      A. Good support for RAID arrays

      B. Supports IAM roles for EC2 Instances, HSM support 
      C. Support for root volume encryption

           D. Supports AWS Storage Gateway as well as EBS volumes

Oracle RDS timezone

The default time zone for your RDS instance is UTC and cannot be changed on the DB. You can set the desired timezone (EST) per database connection. The time zone will only be valid for the connection therefore it needs to be done for each connection (and each time you connect).

The following page outlines  how to do this http://aws-musings.com/amazon-relational-database-service-rds-the-timezone-problem

Oracle Enterprise Manager Cloud Control 12c - Monitoring Essentials test drive lab

App Associates and AWS just announced a new test drive lab.  This test drive lab demonstrates how OEM 12c can be used to monitor and manage on premise Oracle environments in the cloud.

http://awslabs.appsassociates.com/oemdb-registration.php

AWS RDS cross region replication

RDS offers a simple way to replicate across AZs (Multi-AZ feature) for fault tolerance and high availability.  RDS cross region synchronization / replication provides additional fault tolerance.  This is an interesting situation because RDS is a 'black box'/PaaS/DaaS environment. Therefore, no access to the OS level is provided. This means this replication solutions like MySQL Replication or Oracle Goldengate can not be used.  Until RDS offers snapshots like the new EBS region snapshotting, you will have to use tools like BitTorrent or CloudBeam to move database dump files across regions.You could use DB links with materialized views as this article mentions: http://aws.amazon.com/articles/4173109646282306. DB Links are slow but if you don't have a low RTO and RPO you could use this...Or come up with your own creative solution.

Oracle WebLogic with AWS Auto Scaling

The question of using Oracle WebLogic with AWS Auto Scaling and propagation of session state is often asked.  Before getting into the details, let's look at the two ways of handling session state at the server layer (using cookies in the browser can be used as well):
1. Session stickiness : session data issue is to send all requests in a user session consistently to the same backend server. 
2. Session in database : Another solution is to keep the per-session data in a database.  Of course, AWS ElastiCache, SimpleDB or DynamoDB, or RDS.

If the session is stored in a database, nothing needs to be done when using Auto Scaling with WebLogic; for obvious reasons.

When sticky sessions is used, nothing needs to be done, but the reason is not so obvious so let's discuss it.

When a request comes into a WebLogic cluster via a load balancer (AWS ELB for example) or through Apache (mod_proxy_balancer plug in) the first time,  WebLogic creates an HTTP session on the primary node and also puts session state on a backup node. (You can provide guidance/control where to put the failure/backup).  If the server goes down that contains the primary WebLogic node, the new primary node will know where the backup session is stored and the session state will automatically get replicated to the new node.  So when AWS Auto Scaling is used there is nothing that needs to be done from an AWS perspective. 

Oracle specific protocol support on AWS ELB

Some Oracle customers are running Oracle specific protocols and when they move to AWS they would like to know if AWS services (such as ELB) support these products.  AWS ELB supports the HTTP, TCP, HTTPS and SSL protocols.  Oracle UCM IDC protocol is not supported with AWS ELB.  In looking at common on premise software and hardware load balancers (F5, Riverbed etc), I do not see any that support Intradoc (IDC).  The more important question to ask is: "What load balancer do you use on premise to load balance IDC traffic?"  If it is a software load balancer, you could run this on an EC2 instances.>

Another protocol that some Oracle customers run is Sockets Direct Protocol (SDP). SDP is an industry-standard wire protocol between InfiniBand network peers. When used over an InfiniBand network, SDP reduces TCP/IP overhead by eliminating intermediate replication of data and transferring most of the messaging burden away from the CPU and onto the network hardware.  This protocol is not supported on AWS. 

AWS shared disk options

Here the four options most often discussed when considering NAS/shared disk/storage on AWS:

1. S3 : Sometimes NAS isn't the right solution to the problem; it's just something that's relatively easy to implement.
2. GlusterFS, Lustre, openAFS : implementation of a distributed filesystem (GlusterFS, Lustre, openAFS, etc).  Write performance can be below writing to EBS.
3. S3-back 'filesystem' : Use a S3-backed "filesystem" (such as s3fs or Danilo's yas3fs), which is definitely easier to implement. However, write performance could become an issue.
4. NFS : You could just run NFS on another EC2 instances. However, this will not provide the fault tolerance and scalability that is built into a solution such as GlusterFS, or a solution such a Zadara. With Zadara you can have a central repository/shared file system in a NFS mount that will be accesible from EC2 machines.  You can mount Zadara from EC2 via NFS or iSCSI.
5. Of course, when you are running an Oracle database you will probably not use one of these options.  This would be like putting your on premise Oracle database storage on NFS. 

Oracle Data Guard on EC2

Oracle Data Guard can be used to set up one or several slave databases which will be the foundation of a highly available environment. It maintains the standby databases as transaction- consistent copies of the primary database.  These instances can be placed in several availability zones.Then, if the production database becomes unavailable because of a planned or an unplanned outage of this instance or of the full availability zone, Data Guard can switch any standby database to the production role, minimizing the downtime associated with the outage.   The Active Data Guard module enables read-only access to the standby databases, thereby allowing customers to run read queries and reports on the standby instances, and to perform the backups from a standby instance.

You can have zero data loss by using the appropriate data protection mode:http://docs.oracle.com/cd/B28359_01/server.111/b28294/protection.htm#CHDEDGIF

Oracle on AWS costs of Oracle on AWS and licensing management

Here are two common questions in regards to Oracle pricing and licensing on AWS:

1. How can they determine the pricing for an instance launched from an Oracle AMI e.g. https://aws.amazon.com/amis/oracle-database-11g-release-2-11-2-0-1-enterprise-edition-64-bit?

The bottom line: The pricing are the standard EC2, data out traffic, and storage + cost of Oracle licenses (or no cost of Oracle licenses if they have their own)

2. How do they manage BYOL or using unused licenses in practical terms? (afaik they have to BYOL for Oracle on EC2)

The bottom line : It is up to the customer to make sure they are not violating Oracle licensing policies.

ELB HTML file health check file

If you are running Apache, you will need to place the help check file for ELB in /var/www/html.  You can use the default web page (index.html). However, I normally use another file incase the index.html file gets removed for some reason.  Also, another file like ping.html indicates to a DevOps person that this is for the ELB health check.

EC2 Apache HTML file locations

My last blog discussed using user data and to install the Apache serve on your EC2 instance.  The next thing to do is place your web pages in the web server directory. The directory location is:  /var/www/html/.  You need to make sure you are the root user to add or modify files in this directory.  You can do this by using sudo su.

User data to install Apache and PHP

One of the most basic components on an EC2 instance is Apache and PHP.  Installing Apache and PHP is made easy with a short shell script, yum, and user data.  Here is all that needs to be placed in the user data section of the EC2 instance:

#!/bin/sh
yum -y install httpd php
chkconfig httpd on
/etc/init.d/httpd start

CloudFormation access denied error message

In the event you get this message when a CloudFormation script:

AccessDenied. User doesn't have permission to call ec2:RunInstances, this could mean that the AMI is not available or you don't have access to it.  The error is not intuitive.  Check to make sure the AMI exists or if you have access to it.