AWS CLI multiple profiles

When using the AWS CLI, you may be working with multiple AWS accounts.  Therefore, you will want to have multiple profiles.  The profile configuration file is contained in the ~/.aws/config on Linux, OS X, or Unix.  In this file, you will list all of your profiles. At the command prompt or in your .profile file you can have do the following to chose the profile you would like:

export AWS_DEFAULT_PROFILE=<profilename>

Restore S3 backup to MySQL

In this blog post, we backed up our MySQL database: http://cloudconclave.blogspot.com/2013/12/backup-of-mysql-on-aws-to-local.html. Now we will restore this backup.

1. Assumes this directory has been created: /backup/restore. If it does not exist on the EC2 instance, issue this command: mkdir -p /backup/restore

2. aws s3 cp s3://sysopsmysqlbackup/backups/<BACKUP-FILE> /backup/restore/restore.sql --region us-west-2

Note: BACKUP-FILE is the name of the backup file in S3

3. mysql -u root -ppassw-lab awslabrestore < /backup/restore/restore.sql

Data stores compatible with Amazon EMR

There are a number of different file systems that can be used

1. Hadoop Distributed File System (HDFS) : EC2 local/ephemeral disk is where HDFS  resides.  The obvious disadvantage is that it’s ephemeral storage which is reclaimed when the cluster ends. It can be used for caching the results produced by intermediate job-flow steps during a large EMR job.
2. Local (ephemeral) EC2 disk :  Each EMR node comes with local disk.  This disk works well for temporary storage of data that is continually changing, such as buffers, caches, scratch data, and other temporary content.
3. S3 native : Used for input (data set to be reduced) and output/results.
4. S3 block : Stay away from as not as performant as the other options.
5. HBase : HBase is an open source, non-relational, distributed database that runs on top of HDFS.  HBase works with Hadoop/EMR, sharing its file system and serving as a direct input and output to EMR jobs. HBase also integrates with Apache Hive, enabling SQL-like queries over HBase tables, joins with Hive-based tables, and support for Java Database Connectivity (JDBC).

More information here:
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-file-systems.html

Moving EBS volumes between Linux and Windows

EBS volumes are portable between instances running different operating systems but this does not mean the underlying file system format will be compatible.  EBS is a block level storage device.  The volume must be formatted with a file system which may or may not run across different EC2 instances based upon the OS of the instance.

Moving between Linux and Windows file formatted EBS formatted volumes were most of the compatible issues may arise. NTFS and EXT4 are some of the most common file system formats.  

An option is to used NTFS as your 'master' file system.  Then use a utility like ntfsprogs (http://en.wikipedia.org/wiki/Ntfsprogson) to use the EBS volume for both Windows and Linux instances. If you don't need writing on Linux, you can mount the NTFS drive and read it on Linux . Linux can natively read NTFS but it can not write to NTFS drives natively.  

The other option is to use EXT4 (or another Linux file system) as your 'master' file system. For example, EXT4 is not natively supported on Windows.  Good discussion here:

http://superuser.com/questions/465393/how-to-mount-read-write-an-ext4-parti

tion-on-windows-solution-back-to-ntfs

Also, ext2fsd  (http://www.ext2fsd.com)  a proven solution for reading EXT4 from Linux.  For those that don't need NTFS, sharing filesystems between Windows and Linux can be done using exFAT. Another potential solution that supports reading and writing to NTFS on Linux is: http://sourceforge.net/projects/ntfs-3g/

Bulk loading data to AWS from on premise

I spoke about bulk loading data to AWS in this blog post:

http://cloudconclave.blogspot.com/2013/04/on-premise-application-replication-and.html

A couple other options I did not mention are:

1. Apera - Asperasoft Company has developed a proprietary file transfer protocol based on UDP, which has shown to introduce very high-speed file transfer experience over the Internet.

2. http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_s3distcp.html   Apache DistCp is an open-source tool you can use to copy large amounts of data. DistCp uses MapReduce to copy in a distributed manner  You can also use S3DistCp to copy data between Amazon S3 buckets or from HDFS to Amazon S3.  Since it is based upon MapReduce, it is most applicable when moving HDFS and other map reduce files as you will obviously have MapReduce installed in your environment. 

On premise application replication and snap shots to AWS for DR

When devising an disaster recovery approach that used AWS for DR of an on premise environment, there are literally hundreds of options for keeping the databases in sync. Each relational database vendor will have five of their own options and their partner ecosystems will offer many more.  For the application, it is different situation.  Their are not many features built into business application  and application servers.  Depending on what DR scenario you are using http://cloudconclave.blogspot.com/2012/11/aws-four-dr-scenarios.html you will be using either replication or snapshots to keep your AWS application in sync with your on premise environment.  
For warm standby and active-active you will need replication:

• RSync : Rsync is cheap (free) and easy.  Rsync can be chatty, does not have compression built in, and you have to code parallelism. 
• BitTorrent : BitTorrent is most often associated with moving large media files over the Internet in a fast matter. BitTorrent now has file syncing capabilities so can be used to sync application server files.  
• Amazon Storage Gateway " attach these volumes as iSCSI devices to your on-premises application servers.  These files will be stored in S3 where they can be instantiate as EBS volumes.

Riverbed Whitewater - http://www.riverbed.com/products-solutions/products/cloud-storage-whitewater/ https://aws.amazon.com/marketplace/pp/B007O0FXW4/ref=srh_res_product_title?ie=UTF8&sr=0-2&qid=1366938137611#product-detail

Attunity CloudBeam - http://www.attunitycloudbeam.com/solutions/disaster-recovery  https://aws.amazon.com/marketplace/pp/B00B5PB8IM/ref=srh_res_product_title?ie=UTF8&sr=0-3&qid=1366938015695

For backup and restore and pilot light you can use snapshots. 

• FTP or SFTP : FTP and SFTP are the two most common methods to transfer files over the internet.
• HTTP or HTTPs : HTTP and HTTPS can be used but they are slower than S/FTP.  However, if you don't want to require or can not have an FTP client in use than HTTP(s) can be used.  Some companies may not want to open up another port to the outside world have an FTP server running.
• AWS import/export : If files don't need to be synced that often, FedEx, UPS, or Post Office can be the least expensive method with the least amount of hassle (limited coding, no restarting needed)
• Attunity CloudBeam to move files into S3 and than instantiate as EBS volumes.

Oracle EBusiness Suite does offer rapid clone http://docs.oracle.com/cd/E18727_01/doc.121/e12841/T120505T120517.htm but this is not for moving production system data files but for setting test, dev, or moving a system to another machine. Weblogic also offers a cloning feature :http://docs.oracle.com/cd/E12839_01/core.1111/e10105/clone.htm

Importing on premise Oracle Database into AWS RDS

There is an article here but does not get into specifics on the performance of each option. http://aws.amazon.com/articles/4173109646282306

Oracle data pump is one of the fastest ways to move data between Oracle databases.  However, the approach shown in the above article uses database links which can be very slow.  A more performant method would be to use data pump file mode. Unfortunately, AWS Oracle RDS does not off access to the OS.  There is a workaround to this by using a little known feature of the Oracle database called Oracle Managed File System

http://docs.oracle.com/cd/B13789_01/server.101/b10739/omf.htm#i1006122

Oracle RDS has access to this Oracle Managed File (OMF) system. Once the file exported from the on premise database and moved to the Oracle RDS OMF you can use the  UTL_FILE capability in the database and import a file into the RDS instance, and then do impdb (data pump import) from there.

Things to keep in mind:

1.       Since the file sits on the Oracle managed file system, it consumes the space allocated for the database. So you will need to make sure you allocate enough space for the instance

2.       Once you do the impdb operation, you can delete the file but till then it will consume space on the instance

3.       File sizes will need to be less than 400-500 GB since the largest instance currently is 1 TB, and you will need space for both the file as well as the database

4.       As you do the import, you will consume IOPS for reading the dump file and then writing to the database.

AWS shared disk options

Here the four options most often discussed when considering NAS/shared disk/storage on AWS:

1. S3 : Sometimes NAS isn't the right solution to the problem; it's just something that's relatively easy to implement.
2. GlusterFS, Lustre, openAFS : implementation of a distributed filesystem (GlusterFS, Lustre, openAFS, etc).  Write performance can be below writing to EBS.
3. S3-back 'filesystem' : Use a S3-backed "filesystem" (such as s3fs or Danilo's yas3fs), which is definitely easier to implement. However, write performance could become an issue.
4. NFS : You could just run NFS on another EC2 instances. However, this will not provide the fault tolerance and scalability that is built into a solution such as GlusterFS, or a solution such a Zadara. With Zadara you can have a central repository/shared file system in a NFS mount that will be accesible from EC2 machines.  You can mount Zadara from EC2 via NFS or iSCSI.
5. Of course, when you are running an Oracle database you will probably not use one of these options.  This would be like putting your on premise Oracle database storage on NFS. 

ELB HTML file health check file

If you are running Apache, you will need to place the help check file for ELB in /var/www/html.  You can use the default web page (index.html). However, I normally use another file incase the index.html file gets removed for some reason.  Also, another file like ping.html indicates to a DevOps person that this is for the ELB health check.

AWS data movement and replication product

Attunity CloudBeam offers data movement and replication from on premise and AWS, and between AWS regions;
https://aws.amazon.com/marketplace/pp/B00B5PB8IM/ref=srh_res_product_title?ie=UTF8&sr=0-2&qid=1359126283717

AWS EC2 public key from private key

In some cases, you may only have a private key (pem file) and you need to create a new instance from an AMI which requires a public key.  In this case, you can use the following SSH command:

ssh-keygen -y -f /path/to/private-key > /path/to/public-key

||

You can use PuttyGen to create the public key from the private key