Amazon EMR fact and information

Here are some information about EMR:

1. Job Flow : A Job Flow is an Amazon term for an end-to-end processing of data that occurs through a number of compute steps.  A Job Flow is defined by the MapReduce application and input and output parameters.  An EMR cluster does not need a Job Flow as data processing can be done interactively using Hive, Pig, Impala, or some other language.

2. Task group : The task group is optional. The task group instances do not have HDFS storage so data needs to be transferred to these nodes by the master mode. The task group can off load heavy computational work from the core group instances.

3. S3 : Amazon s3 is used for the input and output storage of data sets to processed and analyzed.

4. AMIs : The EMR cluster nodes are maintained by Amazon.  Amazon regularly updates the EC2 AMIs with newer releases of Hadoop, security patches, and more.

5. Map and reduce : The map procedure takes data as input and filters and sorts the data down to a set of key/value pairs that will be processed by the reduce procedure.  The reduce procedure performs a summary procedure of grouping, sorting, or counting the key/value pairs. For example, the map procedure parses out the date and time and treats this data element as a key. Then, a reduce procedure can determine a count of each day (date and time). 

6. s3cmd : s3cmd is used at the OS command to load data into S3.

7. Job Flow Scheduling : To schedule a Job Flow to run every hour you can configure cron to execute the script.

8. EMR technologies supported as steps : EMR supports six technologies to be used in steps in the EMR cluster:

a. Hive : Open source data warehouse package. The Hive Query Language (HQL) is a lot like RDBMS SQL. It is best for organizations with strong SQL skills.  Also has extensions to support direct access to DynamoDB to directly load EMR from DynamoDB.

b. Custom Jar : Core Hadoop Jave libraries preloaded into the EMR cluster.

c. Streaming : Allows you to write Amazon EMR Job Flows in Ruby, Perl, Python, PHP, R, Bash, or C++.   Convert an existing ELT job to run in EMR using streaming.

d. Pig : Pig is a data flow engine that is preloaded in the EMR cluster.  Good fit for organization with strong SQL skills.

e. Impala : Impala is similar to Hive but works faster in certain use cases. More here: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-impala.html

g. HBase : HBase is efficient way of storing large quantities of sparse data using column-based compression and storage. In addition, HBase provides fast lookup of data because data is stored in-memory instead of on disk. 

9. Filter statement : A Map custom JAR application uses the filter statement which is like a WHERE clause on a SQL statement.

10. GROUP: A Reduce custom JAR application uses the GROUP statement which is like a GROUP clause in a SQL statement.

11. Limited structure data and late binding : Unlike data warehousing solutions based upon OLAP or RDBMS, Amazon EMR clusters work with unstructured data and perform late binding of the schema.

12. Performance on small data sets : When running Hive queries against EMR, the run time will appear shocking slow against small data sets when compared to running against a traditional RDBMS. The structure nature of the data sets and indexing capabilities make the RDBMS faster.  EMR (MapReduce) is made for large , unstructured data sets.

13. Mahout : Mahout is supported in EMR.

Redshift SQL query tools

      Here are two tools that can be used with Amazon Redshift to issue query commands:      

1    1. SQL Workbench :

http://www.sql-workbench.net/downloads.html

The information required regarding your Redshift cluster are:

A. JDBC or ODBC connection string: example JBDC connection string looks like this: jdbc:postgresql://redshiftexamplecluster.cvhv40lk8tel.us-east-1.redshift.amazonaws.com:5439/redexampletl?tcpKeepAlive=true

B. User ID : master

C. password : password

More details here: http://docs.aws.amazon.com/redshift/latest/mgmt/connecting-using-workbench.html

2. Aginity : http://www.aginity.com                       A. Server (endpoint): example looks like this: lab.cfmvmxhkrtel.us-west-        2.redshift.amazonaws.com

B. User ID : master

C. password : password

D. Database: databasename

E. Port: 5439

The nice thing about Aginity is that there is no JDBC driver to install. With SQL Workbench, you will need to download and install 

More on where to find the  JDBC driver and configure with SQL Workbench

1. Location of driver: http://jdbc.postgresql.org/download/postgresql-8.4-703.jdbc4.jar

2.     Add the downloaded driver JAR to the driver to the PostGres Driver in SQL Workbench

OpenWorld Oracle on AWS sessions

Here are the two sessions focused on Oracle on AWS at Oracle OpenWorld:

https://oracleus.activeevents.com/2013/connect/search.ww?eventRef=openworld#loadSearch-event=null&searchPhrase=Amazon&searchType=session&tc=0&sortBy=titleSort&p=&i(11180)=20800

I will be co-presenting this session: Best Practices for Running Oracle Database Instances on Amazon Web Services.

DBA and developer access to Oracle hosted on AWS

Here are three common methods used to limit access to the AWS environment for DBAs and developers:

1. Bastion host : A bastion can be used as a jump box' / proxy server. Developers and DBAs would be given access using SSH and than use other credentials to log into the web, application, and database servers. More on bastion host security can be found here: http://cloudconclave.blogspot.com/2013/05/aws-bastion-host-as-single-point-of.html.  There is the cost of the EC2 instance that is the bastion host and data transfer out costs.
2. VPN with customer gateway and virtual private gateway.  In the case, you create a VPN tunnel.  The costs here are the VPN hardware on your side (customer gateway), cost of virtual private gateway (VPG), and costs of VPN connections and data transfer out of AWS. More on VPN costs here (this assumes this option): http://cloudconclave.blogspot.com/2013/05/vpn-costs-for-connections-and-data.html
3. OpenVPN : You do not incur the cost of hardware on your side and the VPG on the AWS side.  You still have the cost of the data transfer out.  You would also incur the cost of the EC2 instances that is running an open source VPN software stack (in this case OpenVPN).

I am sure there are other methods as well.

Could also use these constructs to provide secure integration from your on premise or third party applications (SFTP for flat file integration, VPN for web services).