HVM and PVM : Oracle AMIs

AWS AMIs are either Para Virtual Machine (PVM) or Hardware Virtual Machine (HVM) images. All of the AMIs from Oracle are currently only PVM-based. Therefore, they will not run on cluster compute or i2 instance types.

More information on HVM-based instances types:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html#concepts_cluster_compute_hvmAMI
http://aws.amazon.com/ec2/faqs/#Does_use_of_Cluster_Compute_Instances_differ_from_other_Amazon_EC2_instance_types

Currently, the new i2 instance types are HVM only:
http://aws.amazon.com/about-aws/whats-new/2013/12/19/announcing-the-next-generation-of-amazon-ec2-high-i/o-instance/

More on HVM and PVM can be found here:
http://serverfault.com/questions/222010/difference-between-xen-pv-xen-kvm-and-hvm

If you are looking to run Oracle solutions on Oracle Enterprise Linux and require more then 68 GB of memory, you would have to run on hs1.8xlarge (117 GiB) as all other instance types with more then 68 GB are HVM. See more here: http://aws.amazon.com/amazon-linux-ami/instance-type-matrix/

Peoplesoft on AWS

Here are some case studies and materials for running Oracle Peoplesoft on AWS:

1. Select Staffing presented along with AWS and DLZP. http://www.slideshare.net/tomlaszewski/dat202-using-amazon-rds-to-power-enterprise-applications-1-0

2. Data migration from on premise Oracle to Oracle AWS RDS (same technique can be used for EC2, in fact, with EC2 may be easier to just do an RMAN dump, FTP, SCP or Tsunami to transfer to AWS,) and restore using RMAN on EC2). http://www.slideshare.net/tomlaszewski/advanced-data-migration-techniques-for-amazon-rds

3. Best practices for Peoplesoft on premise to AWS done at Oracle OpenWorld last year by DLZP: http://www.slideshare.net/tomlaszewski/oracle-peoplesoft-on-aws

4. Test drive:http://aws.amazon.com/testdrive/oracle/

AWS Elastic Beanstalk basics

When first getting started with AWS Elastic Beanstalk here are some basic things to know

1. Logging : In Java, logging is done using the Apache Commons Logging framework. Logs can be captured with Apache Log4j or any other component that supports Apache Commons Logging.

2. Custom AMIs can be used : The process is documented here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.customenv.html

3. RDS support: Amazon RDS Oracle, MySQL and SQL Server databases can deployed as part of you Elastic Beanstalk application. 

4. Launch New Environment to get OS patches : Amazon periodically updates the AMIs that were used to build the server instances, but servers can’t be updated while their running. When you launch a new environment, you get the updates.
5. Load Balancing : The Elastic Beanstalk service creates the load balancer for you.
6. Auto Scaling : The service creates the auto scaling configuration and group for you.

7..Custom configuration : Example Uses for YAML Configuration 

• Define custom environment variables beyond PARAMx
• Identify files to be downloaded to hosts 
• Can automatically unpack downloaded archive files
• Specify software to install Specify which services should run on hosts 
• Create and run scripts 
• Create and configure AWS resources

More on Elastic Beanstalk here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/Welcome.html

AWS common command line interface

The AWS Common Command Line interface can be found here:

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

Redshift cluster sizes

100 nodes maximum for each configuration.

Dense Storage (DW1) nodes are available in two sizes.   These are HDD backed instances.

A. The Extra Large has three HDDs with a total of 2TB of magnetic storage. Maximum of 200 TB of storage.

B. The Eight Extra Large has 24 HDDs with a total of 16TB of magnetic storage. Maximum of 1.6 Pedabyte of storage. 

Dense Compute (DW2) nodes are also available in two sizes.  These are SSD back instances.

A. The Large has 160GB of SSD storage per EC2 instance with a maximum is 1.6 TB

B The Eight Extra Large is sixteen times bigger (then the dense compute large) with 2.56TB of SSD storage on the EC2 instance for a maximum of 256TB of SSD storage.

Oracle AWS RDS log mining

Oracle Log miner can be used with AWS Oracle RDS:

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.Oracle.html

If you are using Oracle Database 11.2.0.2.v7 or later, you can retain archived redo logs and use log miner (DBMS_LOGMNR) to retrieve log information.

SSO and Federation on AWS

Identity Federation refers to the ability to accept users that were not authenticated in your systems (such as AWS STS). SSO is the ability to login in once and then access many applications without needing to enter credentials again. It is possible to achieve SSO through federation (AWS SAML support : http://docs.aws.amazon.com/STS/latest/UsingSTS/CreatingSAML.html). But you can have SSO without federation. (e.g. an Active Directory domain and multiple apps in that domain)

AWS CLI multiple profiles

When using the AWS CLI, you may be working with multiple AWS accounts.  Therefore, you will want to have multiple profiles.  The profile configuration file is contained in the ~/.aws/config on Linux, OS X, or Unix.  In this file, you will list all of your profiles. At the command prompt or in your .profile file you can have do the following to chose the profile you would like:

export AWS_DEFAULT_PROFILE=<profilename>

AWS public and elastic IPs

If you launch an instance in EC2-Classic, it is assigned a public IP address by default. This is not an optional configuration.

If you launch an instance into a VPC, a public IP addressing feature is available for you to control whether your instance is assigned a public IP address. In the AWS console, there is a check-box called Assign Public IP to Auto-assign Public IP.The public IP address is assigned to the network interface with the device index of eth0. 

A public IP address is assigned to your instance from Amazon's pool of public IP addresses, and is not associated with your AWS account. When a public IP address is disassociated from your instance, it is released back into the public IP address pool, and you cannot reuse it.

Whether you assign a public IP address to your instance during launch or not, you can associate an Elastic IP address with your instance after it's launched. 

Backup of an Oracle DB on RDS and EC2

When you use RDS, the Oracle database is automatically backed up to S3 for you. You can also take a DB snapshot at anytime or schedule snap shots.  The cost of the backup is included in the price of the database.  You will be charged for the storage in S3 for snapshots.

Amazon RDS enables automated backups of your DB Instance with a 1 day retention period. Free backup storage is limited to the size of your provisioned database and only applies to active DB Instances. For example, if you have 10GB-months of provisioned database storage, AWS will provide at most 10GB-months of backup storage at no additional charge.

You can not use tools like Oracle RMAN or the Oracle Secure Backup for S3 to backup Oracle RDS.

AWS CLI filtering

The output from an AWS CLI command can be displayed in JSON, text or a table. More details here:
http://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html

AWS access key rotation

AWS supports multiple concurrent access keys and certificates. With this feature, customers can rotate keys and certificates into and out of operation on a regular basis without any downtime to their application.

IAM integration with on premise LDAP

You can manually create IAM users with passwords for access to AWS resources using the AWS Management Console, AWS CLI or API calls. This is a great way to start, but you may already have an identity store (such as Active Directory) that you wish to leverage for AWS resource access. Two methods are possible: Replication and Federation.

With replication, you copy account credentials from your identity store into AWS IAM. This is useful for smaller organizations who need a quick solution without building ongoing connectivity between the two identity stores. While fast to execute, this method has some drawbacks such as: Limited to the number of accounts supported by IAM (5,000 default), changes between identity stores are not automatically propagated, this includes password changes, and disabling or deleting of accounts.

Federation can be done using AWS STS or a third-party such as Okta.   

Remember IAM is not a substitute for an identity store such as Active Directory. 

Redshift sample schema from AWS web site

Below are the sample schemas that can be used when getting started with AWS Redshift. More information can be found here: http://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html

create table users(

     userid integer not null distkey sortkey,

     username char(8),

     firstname varchar(30),

     lastname varchar(30),

     city varchar(30),

     state char(2),

     email varchar(100),

     phone char(14),

     likesports boolean,

     liketheatre boolean,

     likeconcerts boolean,

     likejazz boolean,

     likeclassical boolean,

     likeopera boolean,

     likerock boolean,

     likevegas boolean,

     likebroadway boolean,

     likemusicals boolean);

create table venue(

     venueid smallint not null distkey sortkey,

     venuename varchar(100),

     venuecity varchar(30),

     venuestate char(2),

     venueseats integer);

create table category(

     catid smallint not null distkey sortkey,

     catgroup varchar(10),

     catname varchar(10),

     catdesc varchar(50));

create table date(

     dateid smallint not null distkey sortkey,

     caldate date not null,

     day character(3) not null,

     week smallint not null,

     month character(5) not null,

     qtr character(5) not null,

     year smallint not null,

     holiday boolean default('N'));

create table event(

     eventid integer not null distkey,

     venueid smallint not null,

     catid smallint not null,

     dateid smallint not null sortkey,

     eventname varchar(200),

     starttime timestamp);

create table listing(

     listid integer not null distkey,

     sellerid integer not null,

     eventid integer not null,

     dateid smallint not null  sortkey,

     numtickets smallint not null,

     priceperticket decimal(8,2),

     totalprice decimal(8,2),

     listtime timestamp);

create table sales(

     salesid integer not null,

     listid integer not null distkey,

     sellerid integer not null,

     buyerid integer not null,

     eventid integer not null,

     dateid smallint not null sortkey,

     qtysold smallint not null,

     pricepaid decimal(8,2),

     commission decimal(8,2),

     saletime timestamp);

EMR in a private subnet

When running EC2 instance or other AWS services in a private subnet, you need a NAT to access S3. 

You can not use a NAT when using EMR:

http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-vpc-subnet.html

Because access to and from the AWS cloud is a requirement of the cluster, you must connect an Internet gateway to the VPC subnet hosting the cluster. If your application has components you do not want connected to the Internet gateway you can launch those components in other subnets you create within your VPC. In addition, because of the need to access the AWS cloud, you cannot use Network Address Translation (NAT) when you are running Amazon EMR on a VPC.

AWS IAM services and features that play a role when moving Oracle workloads to AWS

Identity and access management (IAM) is a key component of AWS security.  IAM allows the creation of users, groups and roles. IAM services and features you will most likely consider when host Oracle on AWS are:

1. Security Token Service (STS) : STS generates temporary credentials for an IAM User or for users that you authenticate (federated users). Useful for improving security posture, mobile applications, and identity federation. Here are some basics of STS:

A. Generates a temporary Access Key, Secret Key, and Token
B. Expire automatically (15 minutes ~ 36 hours)
C. You write an “identity broker application” ( sample code here: http://docs.aws.amazon.com/STS/latest/UsingSTS/STSSampleApps.html)
D. Users authenticate to your identity broker
E. Your identity broker provisions temporary credentials via STS
F. SSO via the management console: Temporary credentials can be used to sign user directly into the AWS Management Console

More reading: http://docs.aws.amazon.com/STS/latest/UsingSTS/Welcome.html

  Get the temporary temporary token (access key ID, secret Key, and security token) by issuing the get_federation_token API call.  The thing to keep in mind with your identity broker application is that you will have to have all the policies defined some where (in code as below, relational database, XML file, JSON templates, some file, etc) as you probably don't want to hard code in the application and will have 1000's of different users with different policies. 

http://docs.aws.amazon.com/AWSSDKforPHP/latest/index.html#m=AmazonSTS/get_federation_token
The role needs to have a policy associated. Here is the way it is done it code:

// Generate a new IAM policy (https://awspolicygen.s3.amazonaws.com/policygen.html)

$policy = new CFPolicy($token, array(

    'Statement' => array(

        array(

            'Sid' => 'SID' . time(),

            'Action' => array('s3:GetObject', 's3:GetObjectVersion', 's3:ListBucket', 's3:ListBucketVersions'),

            'Effect' => 'Allow',

            'Resource' => 'arn:aws:s3:::my-bucket/*'

        )

    )

));

Here is a REST web services call to generate temporary credentials: http://docs.aws.amazon.com/STS/latest/UsingSTS/CreatingFedTokens.htm

or you can use the assume-role API or web services all in document above. Here you would not need to send in a policy as the role would have a policy:

http://docs.aws.amazon.com/AWSSDKforPHP/latest/index.html#m=AmazonSTS/assume_role

2. IAM resource level permissions - More information can be found here:  http://aws.amazon.com/about-aws/whats-new/2013/07/08/announcing-resource-permissions-for-amazon-ec2-and-amazon-rds/

3. IAM Roles: Roles can be used to securely allow EC2 instances access other AWS services, enables identity federation with STS, cross account access, and web identity federation. 

A. Allow your applications (e.g., Java) running on EC2 to securely access other services (e.g., S3, SQS, etc)

B. Identity federation (including STS)

C. Web Identity federation : Web Identity Federation allows authenticating via Login with Amazon, Facebook, or Google, getting temporary security credentials, and then using those credentials to make a request to AWS. 

D. Allow cross-account management/access

Jane in Account A may assume a Role in Account B, giving Jane an Access Key/Secret Key/Token that may be used to make API calls to Account B. 

More reading:http://docs.aws.amazon.com/IAM/latest/UserGuide/WorkingWithRoles.html