Migrating data from on premise to AWS

Here are some the approximate transfer rates for Amazon Storage Gateway and Riverbed Whitewater:

• Amazon Storage Gateway 5 TB a day
• Riverbed Whitewater  3-5 TB an hour

Keep in mind that using gateways to migrate a relational database is probably not the best option as these mechanism use snapshots.

Other blog entries are here:

Migrating data for migrating to RDS but useful for EC2 based databases as well: http://cloudconclave.blogspot.com/2013/10/loading-data-from-on-premise-oracle.html

This blog discusses the time it takes to transfer 1 TB of data over different lines with different transfer speeds: http://cloudconclave.blogspot.com/2013/02/oracle-db-data-transfer-to-aws.html

Mechanisms to bulk migrate data: http://cloudconclave.blogspot.com/2013/06/bulk-loading-data-to-aws-from-on-premise.html

Here is a blog post on different transfer mechanisms for applications but could be used for databases:http://cloudconclave.blogspot.com/2013/04/on-premise-application-replication-and.html

Attunity CloudBeam: http://cloudconclave.blogspot.com/2013/12/moving-and-migrating-data-from-on.html

EBS data transfer costs to snapshot Oracle DB to S3

Obviously, there is a cost to snapshot your EBS volumes of your Oracle database and store them in S3 for backup and recovery.  This is the standard S3 cost of $.095 GB a month.  The cost can go down to $.055 when you store more data.  There is no cost to transfer the data in and out of S3 to and from EC2. 

Traffic costs between EC2 instances, AZs, and VPCs

Often times when running on AWS you will have multiple AZs, accounts and VPCs.  This is a simple summary of EC2 data transfer costs:

1. Within the same AZ  : Free: It is within the same AZ and over private IP.
2. Between two AZs :   Cost. Will incur inter-az charges for both inbound and outbound

3. Between Accounts in different VPCs :  Cost. Inter-az charges are applicable for data transfer charges between instances in the different AZ's (same region) and in different accounts.
4. Between one EC2 classic instance and VPC EC2 instance in a same AZ : Cost
5. Instances to S3 in the same region: Free
6. Instance to S3 in different region: Cost. cross region charges

Bulk loading data to AWS from on premise

I spoke about bulk loading data to AWS in this blog post:

http://cloudconclave.blogspot.com/2013/04/on-premise-application-replication-and.html

A couple other options I did not mention are:

1. Apera - Asperasoft Company has developed a proprietary file transfer protocol based on UDP, which has shown to introduce very high-speed file transfer experience over the Internet.

2. http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_s3distcp.html   Apache DistCp is an open-source tool you can use to copy large amounts of data. DistCp uses MapReduce to copy in a distributed manner  You can also use S3DistCp to copy data between Amazon S3 buckets or from HDFS to Amazon S3.  Since it is based upon MapReduce, it is most applicable when moving HDFS and other map reduce files as you will obviously have MapReduce installed in your environment. 

AWS Storage options, cost, and SLA

A common question when moving from on premise to AWS is what are the storage options
 and how much they will cost.  A great place to calculate the cost is the Simple Monthly Calculator: http://calculator.s3.amazonaws.com/calc5.html. 

Before you can calculate the cost you need to be aware of the options, cost of each option, when to use it, and SLAs:

a.   EBS: http://aws.amazon.com/pricing/ec2/ : Data Transfer, EBS standard and EBS PIOPS. Sample pricing:

                                                             i.     Data Transfer, EBS Standard, EBS PIOPS : http://aws.amazon.com/pricing/ebs/

                                                            ii.     Price is for allocated storage

                                                          iii.     SLA : annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume

                                                          iv.     Common use cases: RDBMS (PIOPS), Application Server files (IOPS)

b.   S3 : http://aws.amazon.com/pricing/s3/

                                                             i.     First TB per month of storage: .095

                                                            ii.     Data transfer : Up to 10 TB .120 per GB + request prices (.005 per 1K requests)

                                                          iii.     Price is for used storage

                                                          iv.     SLA: 99.999999999% durability, 99.99% availability

                                                           v.     Common use cases: backups, multi media content, web logs, EMR jobs

c.    Glacier: http://aws.amazon.com/glacier/pricing/

                                                             i.     First TB per month of storage: .01

                                                            ii.     Price is for used storage

                                                          iii.     SLA: 99.999999999% durability

                                                          iv.     Common use cases: archiving, long term storage

d.   Instance storage : Included in price of instance

                                                             i.     Common use cases: web server files (instance disk storage), data warehouses (Redshift runs on instance storage), RDBMS (instance SSD..with redundancy), temporary files

AWS data transfer costs

Some times instance reside in different AZs and these instances need to share data. Instances also use S3 for things like backups. Here is breakdown of data transfer cost for some typical scenarios: 

1. Instance to Instance in same AZ 
Cost: Free
2. Instance to Instance in different AZ
Cost: Inter-AZ charge for both inbound and outbound
3. Instance to Instance in different account in same or different AZ
Cost: Inter-az charges are applicable for data transfer charges between instances in the different AZ's (same region) and in different accounts
4. Instance to S3 in the same region
Cost: Free

VPN costs for connections and data

VPC has no cost associated. However, if you want to extend your data center or provide a secure IPSec tunnel through a VPN client, you need to add in costs for the VPN connection and data transfer costs over the VPN tunnel.   This is the first place to look at to decipher VPN connection cost: http://aws.amazon.com/vpc/pricing/.  The cost is $0.05 per VPN Connection-hour + standard AWS data transfer charges for all data transferred via the VPN Connection. For example, if you have 2 VPN connections the cost for those connections would be:  .05x24x365.25x2(2 connections) / 12 = ~73.05 + data transfer.

Now for the data transfer out piece:  On data transfer out for VPN connection, the cost is .12 per GB for less then 10 TB a month per connection..  So, two connection each at 1 TB a month would be $245.76 a month.  

On premise application replication and snap shots to AWS for DR

When devising an disaster recovery approach that used AWS for DR of an on premise environment, there are literally hundreds of options for keeping the databases in sync. Each relational database vendor will have five of their own options and their partner ecosystems will offer many more.  For the application, it is different situation.  Their are not many features built into business application  and application servers.  Depending on what DR scenario you are using http://cloudconclave.blogspot.com/2012/11/aws-four-dr-scenarios.html you will be using either replication or snapshots to keep your AWS application in sync with your on premise environment.  
For warm standby and active-active you will need replication:

• RSync : Rsync is cheap (free) and easy.  Rsync can be chatty, does not have compression built in, and you have to code parallelism. 
• BitTorrent : BitTorrent is most often associated with moving large media files over the Internet in a fast matter. BitTorrent now has file syncing capabilities so can be used to sync application server files.  
• Amazon Storage Gateway " attach these volumes as iSCSI devices to your on-premises application servers.  These files will be stored in S3 where they can be instantiate as EBS volumes.

Riverbed Whitewater - http://www.riverbed.com/products-solutions/products/cloud-storage-whitewater/ https://aws.amazon.com/marketplace/pp/B007O0FXW4/ref=srh_res_product_title?ie=UTF8&sr=0-2&qid=1366938137611#product-detail

Attunity CloudBeam - http://www.attunitycloudbeam.com/solutions/disaster-recovery  https://aws.amazon.com/marketplace/pp/B00B5PB8IM/ref=srh_res_product_title?ie=UTF8&sr=0-3&qid=1366938015695

For backup and restore and pilot light you can use snapshots. 

• FTP or SFTP : FTP and SFTP are the two most common methods to transfer files over the internet.
• HTTP or HTTPs : HTTP and HTTPS can be used but they are slower than S/FTP.  However, if you don't want to require or can not have an FTP client in use than HTTP(s) can be used.  Some companies may not want to open up another port to the outside world have an FTP server running.
• AWS import/export : If files don't need to be synced that often, FedEx, UPS, or Post Office can be the least expensive method with the least amount of hassle (limited coding, no restarting needed)
• Attunity CloudBeam to move files into S3 and than instantiate as EBS volumes.

Oracle EBusiness Suite does offer rapid clone http://docs.oracle.com/cd/E18727_01/doc.121/e12841/T120505T120517.htm but this is not for moving production system data files but for setting test, dev, or moving a system to another machine. Weblogic also offers a cloning feature :http://docs.oracle.com/cd/E12839_01/core.1111/e10105/clone.htm

Oracle on AWS costs of Oracle on AWS and licensing management

Here are two common questions in regards to Oracle pricing and licensing on AWS:

1. How can they determine the pricing for an instance launched from an Oracle AMI e.g. https://aws.amazon.com/amis/oracle-database-11g-release-2-11-2-0-1-enterprise-edition-64-bit?

The bottom line: The pricing are the standard EC2, data out traffic, and storage + cost of Oracle licenses (or no cost of Oracle licenses if they have their own)

2. How do they manage BYOL or using unused licenses in practical terms? (afaik they have to BYOL for Oracle on EC2)

The bottom line : It is up to the customer to make sure they are not violating Oracle licensing policies.

Oracle DB data transfer to AWS

A typical Oracle database is 1 TB or more in size. This will mean that a T3 line or higher speed line will be needed to move this much data. Otherwise, the AWS import export service can be used. The information below is from the http://aws.amazon.com/importexport/ web page but repeated here for easy of finding.

Connection           Theoretical 80% Utilization    AWS Imp/Exp

T1 (1.544Mb)        82 days                                    100GB or more

10Mbps                  13 days                                    600GB or more

T3 (44.736Mbps)    3 days                                      2TB or more

100Mbps               1 to 2 days                                 5TB or more

1000Mbps             Less than 1 day                         60TB or more

Elapsed time is going to depend on many factors: speed of connection, compression used (if any), can load be done in parallel etc. The fastest load times will happen using AWS import/export (http://aws.amazon.com/importexport/ ). The next fastest will be using AWS Direct Connect (http://aws.amazon.com/directconnect). 

Direct Connect provides speeds of 1 Gbps to 10 Gbps. Using a 1 Gbps line, it would take about 3-4 hours (here is a good site: http://fasterdata.es.net/fasterdata-home/requirements-and-expectations/). 

AWS data movement and replication product

Attunity CloudBeam offers data movement and replication from on premise and AWS, and between AWS regions;
https://aws.amazon.com/marketplace/pp/B00B5PB8IM/ref=srh_res_product_title?ie=UTF8&sr=0-2&qid=1359126283717

AWS import export : When to use import export

This web page has some guidelines on when to use AWS import export and when to use a network connection:
http://aws.amazon.com/importexport/

Go down to this section: "When to Use Import/Export"

I often get asked this question when someone wants to move large Oracle database to the cloud.

Copying 1 TB of data from on premise RDBMS to AWS/EC2

I often get asked how long it will take to transmit a 1 TB database (assumes that the database is actually 1 TB in size and not just allocated at 1 TB). Elapsed time is going to depend on many factors: speed of connection, compression used (if any), can load be done in parallel etc.  The fastest load times will happen using AWS import/export (http://aws.amazon.com/importexport/ ).  The next fastest will be using AWS Direct Connect (http://aws.amazon.com/directconnect). Direct Connect provides speeds of 1 Gbps to 10Gbps.  Using a 1 Gbps line, it would take about 3-4 hours (here is a good site: http://fasterdata.es.net/fasterdata-home/requirements-and-expectations/).    Assuming an internet connection speed of 100 Mbps (100Base-T connection) it would take about a day to transfer using a traditional connection to AWS/EC2  Both DirectConnect and Import/export provide a secure method of transfer. 

AWS three top services that contribute to pricing

Typically the top three AWS services that contribute to the price of AWS services are:
1.  Compute (typically EC2)
2.  Storage (typically EBS or S3)
3.  Data transfer out