Wednesday, July 11, 2012

Blog Migrations using movable type

Middleware and Cloud Computing Book

One of only a couple books that really discusses AWS and Oracle in depth. This books is focused on AWS and Oracle middleware/fusion/weblogic:
AWS Middleware cloud computing book
It also covers Rackspace but in not near as much depth.

Oracle Enterprise Manager with AWS Database Instance

Oracle Enterprise Manager 11g Database Control support was just announced this May. It seems like it may be a long time in coming but Oracle Database support on the AWS cloud was first released just a few years ago in September 2008. Here is a great blog entry on using OEM with Oracle on AWS:

AWS OEM for Oracle DB

Remember that without https in front of the URL OEM will not respond.https://ec2-99-55-139-99.eu-west-1.compute.amazonaws.com:1158/em

AWS EC2 Overview book

A very good introduction book to AWS can be found here:
AWS Book

I would say the title is misleading as this book is more an introduction and overview book then a programming book. However, very good for those IT people new to AWS.

AWS EC2 Oracle Database - Storing and managing my data

When create an Oracle Database on the Amazon cloud you will need to store you database files somewhere on the EC2 cloud. There are basically three places where database files can be stored:
1. Local drive - This is the local drive that is part of the virtual server EC2 instance.
2. Elastic Block Storage (EBS) - Network attached storage that appears as a local drive.
3. Simple Storage Server (S3) - 'Storage for the Internet'.

S3 is not high speed and intended for store static document type files. S3 can also be used for storing static web page files. Local drives are ephemeral so not appropriate to be used as a database storage device. The leaves EBS which is the best place to store database files. EBS volumes appear as local disk drives. They are actually network-attached to an Amazon EC2 instance. In addition, EBS persists independently from the running life of a single Amazon EC2 instance. If you use an EBS backed instance for your database data, it will remain available after reboot but not after terminate. In many cases you would not need to terminate your instance but only stop it, which is equivalent of shutdown. In order to save your database data before you terminate an instance, you can snapshot the EBS to S3.

Using EBS as a data store you can move your Oracle data files from one instance to another. This allows you to move your database from one region or or zone to another. Unfortunately, to scale out your Oracle RDS on AWS you can not have read only replicas. This is only possible with the other Oracle relational database - MySQL. The free micro instances use EBS as its storage.

This is a very good white paper that has more details:
AWS Storage Options
This white paper also discusses: SQS, SimpleDB, and Amazon RDS in the context of storage devices. However, these are not storage devices you would use to store an Oracle database. This slide deck discusses a lot of information that is in the white paper:
AWS Storage Options slideshow

AWS CloudFormations, Oracle Assembly Builder, Chef and Puppet

I blogged about the difference and similarities between AWS CloudFormations and Oracle Assembler builder to package your software stack for deployment/provisioning to the cloud. However, these tools do not deal with software stack versioning and configuration management. This is where tools like Chef and Puppet come into play.
Puppet and Chef points of interest:
1. Can be used in any cloud environment (rackspace, private cloud etc).
2. There is a debate between which is better. I am not going to get into this debate other then to say Puppet is more mature.
3. AWS CloudFormations can integration with both Chef and Puppet.

A good blog on AWS CloudFormations and the need for something more:
AWS CloudFormation

IBM DB2 AIX RISC to Oracle Linux Sun x86

Mentioned RISC to x86 migrations a couple blogs ago and now there is a new white paper out by Oracle on just this topic. The white paper goes into detail on the Oracle Linux and Database installation. It also shows how to use the free Oracle SQL Developer tool to migrate the data:
Migrating from IBM DB2 AIX to Oracle Linux Sun x86

Amazon CloudFormations and Oracle Virtual Assembly Builder

Yesterday I blogged about AWS AMIs and Oracle VM templates. These are great mechanisms to stand up an initial cloud environment. However, they don't provide the capability to manage, provision and update an environment once it is up and running. This is where AWS Cloud Formations and Oracle Virtual Assembly Builder comes into play. In a way, these tools/frameworks pick up where AMIs and VM templates leave off.
Once again, there a similar offers from AWS and Oracle that compliant and also overlap with each other. Let's start by looking at the definitions:
AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.
AWS CloudFormations

Oracle Virtual Assembly Builder - Oracle Virtual Assembly Builder makes it possible for administrators to quickly configure and provision entire multi-tier enterprise applications onto virtualized and cloud environments.
Oracle VM Builder

As with the discussion around should you use AMI or VM Templates, there are pros and cons to each:
1. CloudFormation is JSON, Assembly Builder is GUI and CLI
2. VM Templates can be used in any private or public cloud environment. Of course, CloudFormations is tied to AWS public cloud

Amazon AMIs and Oracle VM templates

I have worked with Oracle VM templates and most recently with Amazon Machine Images (AMI). The similarities in the functionality and capabilities they provide are striking. Just take a look a the definitions:

An Amazon Machine Image (AMI) is a special type of pre-configured operating system and virtual application software which is used to create a virtual machine within the Amazon Elastic Compute Cloud (EC2). It serves as the basic unit of deployment for services delivered using EC2. AWS AMIs

Oracle VM Templates provide an innovative approach to deploying a fully configured software stack by offering pre-installed and pre-configured software images. Use of Oracle VM Templates eliminates the installation and configuration costs, and reduces the ongoing maintenance costs helping organizations achieve faster time to market and lower cost of operations. Oracle VM Templates

Other things they have in common:
1. Both have 35 Oracle images or templates:
AWS AMI pre-built images Oracle pre-built VM Templates
2. Both allow to build your own images or templates: A. OVM template builder - OVM Template Builder - Oracle VM Template Builder, an open source, graphical utility that makes it easy to use Oracle Enterprise Linux “Just enough OS” (JeOS)–based scripts for developing pre-packaged virtual machines for Oracle VM.
B. AMI 'builder' - AMI builder
However, AWS has the added feature/benefit of adding your own AMI to the AWS AMI catalog: AMI - Adding to the AWS AMI catalog

Another plus with AWS and AMI is there are hundreds of MySQL AMIs (AWS MySQL AMIs ). A benefit of Oracle VM templates is they can run on any public or private cloud environment, not just AWS EC2.
However, with Oracle VM templates they first need to be images as AMIs before they can run in the AWS cloud.

Migrating RISC to x86 - endianess 'issue'

Endianess always comes up when migrating applications and databases from RISC to x86. The issue is often time overblown as if you are running on a relational database the database vendor will provide tools or automated methods to convert the data properly. Oracle RMAN is often the first choice. Oracle imp/exp, data pump, and GoldenGate can also be used.

A bigger issue would be applications that access OS files. These OS files will need to be converted from big endian (RISC) to little endian (CISC) and then the application may be impacted because of the endianess differences.

Oracle Database on EC2 Cloud white paper link

Some very good information on running Oracle on AWS EC2:
AWS web site regarding Oracle

However, this link is broken: Getting started with Oracle on AWS link broken
Very good FAQ on OTN:
Oracle on AWS FAQ

Progress 4GL and DB to Oracle and cloud

Getting from client/server based 4GLs and databases where the 4GL is tightly linked to the database to Oracle and the cloud is not easy. The least risky and expensive option (in the short term) is to use the Progress OpenEdge DataServer for Oracle:
Progress OpenEdge DataServer
This eliminates the need to have to migrate the Progress 4GL to Java/J2EE.
The database can be migrated using SQLWays Ispirer:
Ispirer SQLWays ProgressDB migrations tool

The Progress 4GL can remain as is. In order to get the application on the cloud there are a few approaches:
1. VDI - Virtual Desktop is a way to put all of the users desktop in a centralized environment off the desktop. This is great in cases where it is just not one client/server application that the user needs access too. In many cases, users will utilize MS Access, MS Excel, Crystal Reports and other tools to get at the Progress DB and other centralized databases. Vmware's acquistion of Wanova shows how VDI is growing in usage. Citrix is the 800 pound gorilla in the VDI space with Citrix WinFrame (now called XenDesktop). Oracle offers a VDI solution that Oracle picked up when it acquired Sun.
2. Hypervisor Server Virtualization - Of course you can place applications written in client/server languages like Progress 4GL buy using server virtualization from Oracle, VMWare, Microsoft, Citrix and others.
3. Microsoft Remote Desktop Services (aka: Terminal Services Client)
The entire idea is to eliminate all the client/server desktop devices and connections which require desktop software and database drivers. A solution to removing database drivers from the desktop is to use DataDirect SQLLink


CloudCruiser Chargeback in the cloud

Another company that does chargeback has just been pointed out to me:
CloudCruiser
There is interesting quote on this company's web site:
"Accurate and transparent chargeback is a key requirement in this age of cloud computing. By 2015, we forecast more than 50% of the Global 2000 will charge back most IT costs using service-based pricing, up from less than 10% today. New integrated tools will be needed to implement IT service-based chargeback."

- Jay Pultz, Vice President and Distinguished Analyst, Gartner

Mainframe modernization - Getting it to work

Interesting read on a very old (1950s) Medicaid system that the State of NC is struggling to modernize. The new language for the modern system is COBOL. This is causing some 'heads to turn':
State of NC modernization to COBOL

Chargeback and billing across public and private clouds

Had a great conversation today regarding the need for metering, chargeback, and billing of cloud computing resources. The person I spoken with at a Fortune 1000 company increased the scope and magnitude of the issue of billing for cloud computing resources beyond what I had previously considered. I believed that doing any type of chargeback and billing for one public, private or hybrid installation was difficult. This person pointed out that the problem is even bigger in scope. The reality is many companies are using multiple public cloud vendors and have many different private cloud data centers. A customer may use AWS for some smaller public cloud applications, Salesforce.com (SaaS), Rackspace for IaaS, Savvis for colocation and a variety of Iaas and PaaS implementations for the private cloud. How does a company get a consolidated bill for all these different cloud environments? I am not sure their is an answer right now.

Migrating C/C++ embedded SQL code

When moving from Sybase C Embedded SQL to Oracle C Embedded SQL, there are no tools are process. Since Embedded SQL is a standard it should be pretty straight forward. The biggest challenge is going to be SQL that is Sybase specific. I would map the process out as follows:
1. Change include files from Sybase Embedded SQL libraries to Oracle Pro*C in each module.
2. Make sure the C code does not test for Sybase specific error messages. Fix where needed.
3. Run all modules through the Pro*C pre-compiler. Fix any embedded static SQL that does not make it through the pre-compiler using the SQL Developer scratch pad editor.
4. Compile using C compiler.
5. Test run the application fixing any dynamic SQL that does not work against Oracle using SQL Developer scratch pad editor.
SQL Developer can be downloaded here:

SaaS usage billing software

Just found out about this SaaS billing company that can be used for software usage billing in the cloud:
Metratech Metanga

Unidata and RDB migrations to Oracle

Have a couple of unique migrations that don't come along to often. They are Unidata and RDB migrations. The top three things that make these migration more challenging are:
1. No automated data migration tools - Because these migration don't happen that often, there are no tools in the market place to automated the data migration.
2. Application is tied to database - The application needs to be re-architected/re-engineered. Unidata Basic and COBOL for RDB. TSRI can migrate Basic to Java and PL/SQL. Transoft can migrate DEC COBOL to Java.
3. New client hardware potentially involved - Many Unidata and RDB based systems use 'green screens' as the front end. These are character based screens that will run on very old dumb terminals such as: Wyse and DEC 5250 terminals. The user interface can be replicated in a web browser but many times these old terminals do not support web browsers.

AWS EC2 Oracle RDB connection to Oracle Database Instance

Provisioning my Oracle database instance to AWS EC2 RDB was easy. Just a few clicks! However, getting my connection to my Oracle cloud database was not as easy. A couple things that are not obvious (using Oracle SQL Developer):
1. Need to set up a database security group.
2. Need to use end point for the host name.
This video is the best one on the internet to explain both points:
Youtube video on Oracle AWS RDB

Oracle Cloud offering - What makes it unique ?

Three things that make Oracle cloud offering unique:
1. SaaS, PaaS and IaaS offerings
2. Private and Public cloud offerings
3. Ability to seamlessly move between private and public cloud

Advantages of relational databases over VSAM, ISAM and hierarchical data stores

When migrating companies from legacy environments to the cloud, invariably you run into older hierarchical, flat file, VSAM, ISAM and other legacy data stores. There are many advantages to moving these databases into a relational database structure. The most important which is that most cloud providers run on relational database models. AWS, for example, supports Oracle, SQL Server, and MySQL. The top three 'other reasons' for moving to a relational database are:
1. Data Access – Thousands of database access tools from query creation to business intelligence.
2. Management and monitoring – Hundreds of tools for management and monitoring of the database.
3. Leverage all the free tools from relational database vendors. Free Oracle database tools include:
-Application Express – WYSIWIG browse based application development and deployment.
-SQL Developer – SQL and PL/SQL development. Database object maintenance.
Adding number four based upon feedback:
4. Data quality, integrity, referential integrity and even business logic is embedded in the database.
What is interesting is that Big Data NoSQL databases and XML databases are taking us back to the days of VSAM (key value databases) with NoSQL and IMS (hierarchical) with XML databases?

AWS EC2 WordPress Blog sample application

Deployed the WordPress Blog to my EC2 and the performance is so slow. Actually it is unusable. This is a well known issue when using the free AWS EC2 free micro instance. If you can get to it, here is my wordpress implementation:
Wordpress Blog

AWS web site up and running

Not to much effort and have the cloud migrations web site up and running on AWS EC2 public cloud:
Cloud Migrations .ORG
Cloud Migrations .NET

Public Cloud, co-location and managed services ... what is the cloud?

Recently I have had conversation with a number of people that are selling and implementing 'cloud' solutions. I put cloud in quotes as implementations like co-location (aka co-lo) and managed services (sometimes referred to as 'your mess for less') have become popular options for companies moving to the cloud. These are obviously not pure public cloud offerings and probably more of hybrid cloud implementations as the infrastructure (PasS and IaaS)is dedicated to a specific customer. This eliminates the security, multi-tenancy, performance and other concerns that companies have regarding public cloud.

Are co-location and managed services cloud to you? Are they something your company is considering when you think about cloud ?

AWS secure FTP

I was looking for this information a couple we back. Should have just went to the Oracle web site:
AWS SFTP to upload files

Hana Vs Oracle in memory databases

The debate continues:
Hana Vs Oracle

AWS web site creation

After much struggle to write my own HTML and get for formatting correct etc, I found this tool on the web. It is incredible for creating web sites / web pages. Free and there is not even a place to donate...I would donate in a heart beat.
BlueGriffon HTML Editor

AWS EC2 amazon cloud domain name and ip fowarding

Once I have the web pages loaded I need to get the web site up and running. This requires a couple simple steps:
1. Add the IP address of your EC2 cloud instance to the your domain name provider. In this case, it is go daddy. More details on how to do here:
Go Daddy IP domain name add
2. For other domain names you have, you can simply use domain name forwarding. More here:
Go Daddy domain name forwarding

AWS EC2 amazon sftp only for web site upload pem pkk ftp ssh

The most difficult part of creating my AWS EC2 web site (other then the 'coding' part ... more on this tomorrow) was getting the files loaded.
1. SFTP can only be used. FTP will not work! Discussion is here: SFTP with EC2 (or can use scp)
2. PEM file to a .PPK file - I used Filezilla as described here: Converting PEM to PPK file. Seems as though sftp does not use pem file like SSH does.
3. login in to EC2 instance using SSH and change user to root ( sudo su ) and change the directory with location of you web pages (/var/www/html) to '755' (chmod
-R 755 /var/www/html). This is because you will be using your ec2-user name to upload using SFTP.
4. login into SFTP using ec2-user and your instance name: sftp ec2-user@
Note: These file go into the directory /var/www/html.

Amazon CloudFormations and Oracle Virtual Assembly Builder

Yesterday I blogged about AWS AMIs and Oracle VM templates. These are great mechanisms to stand up an initial cloud environment. However, they don't provide the capability to manage, provision and update an environment once it is up and running. This is where AWS Cloud Formations and Oracle Virtual Assembly Builder comes into play. In a way, these tools/frameworks pick up where AMIs and VM templates leave off.


Once again, there a similar offers from AWS and Oracle that compliant and also overlap with each other. Let's start by looking at the definitions:


AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.


AWS CloudFormations




Oracle Virtual Assembly Builder - Oracle Virtual Assembly Builder makes it possible for administrators to quickly configure and provision entire multi-tier enterprise applications onto virtualized and cloud environments.


Oracle VM Builder




As with the discussion around should you use AMI or VM Templates, there are pros and cons to each:


1. CloudFormation is JSON, Assembly Builder is GUI and CLI


2. VM Templates can be used in any private or public cloud environment. Of course, CloudFormations is tied to AWS public cloud

Amazon AMIs and Oracle VM templates

I have worked with Oracle VM templates and most recently with Amazon Machine Images (AMI). The similarities in the functionality and capabilities they provide are striking. Just take a look a the definitions:




An Amazon Machine Image (AMI) is a special type of pre-configured operating system and virtual application software which is used to create a virtual machine within the Amazon Elastic Compute Cloud (EC2). It serves as the basic unit of deployment for services delivered using EC2. AWS AMIs




Oracle VM Templates provide an innovative approach to deploying a fully configured software stack by offering pre-installed and pre-configured software images. Use of Oracle VM Templates eliminates the installation and configuration costs, and reduces the ongoing maintenance costs helping organizations achieve faster time to market and lower cost of operations.
Oracle VM Templates




Other things they have in common:


1. Both have 35 Oracle images or templates:


AWS AMI pre-built images
Oracle pre-built VM Templates


2. Both allow to build your own images or templates:
A. OVM template builder - OVM Template Builder - Oracle VM Template Builder, an open source, graphical utility that makes it easy to use Oracle Enterprise Linux “Just enough OS” (JeOS)–based scripts for developing pre-packaged virtual machines for Oracle VM.


B. AMI 'builder' - AMI builder

However, AWS has the added feature/benefit of adding your own AMI to the AWS AMI catalog: AMI - Adding to the AWS AMI catalog




Another plus with AWS and AMI is there are hundreds of MySQL AMIs (AWS MySQL AMIs ). A benefit of Oracle VM templates is they can run on any public or private cloud environment, not just AWS EC2.

However, with Oracle VM templates they first need to be images as AMIs before they can run in the AWS cloud.