Showing posts with label replication. Show all posts
Showing posts with label replication. Show all posts

Monday, April 7, 2014

Oracle RDS GoldenGate support

Oracle GoldenGate can be used with Amazon RDS for Oracle for active-active database replication, zero-downtime migration and upgrades, disaster recovery and data protection, and cross region replication. 

Sunday, March 30, 2014

IAM integration with on premise LDAP

You can manually create IAM users with passwords for access to AWS resources using the AWS Management Console, AWS CLI or API calls. This is a great way to start, but you may already have an identity store (such as Active Directory) that you wish to leverage for AWS resource access. Two methods are possible: Replication and Federation.
With replication, you copy account credentials from your identity store into AWS IAM. This is useful for smaller organizations who need a quick solution without building ongoing connectivity between the two identity stores. While fast to execute, this method has some drawbacks such as: Limited to the number of accounts supported by IAM (5,000 default), changes between identity stores are not automatically propagated, this includes password changes, and disabling or deleting of accounts.

Federation can be done using AWS STS or a third-party such as Okta.   

Remember IAM is not a substitute for an identity store such as Active Directory. 

Thursday, October 31, 2013

Replication of on premise Oracle to Oracle RDS

When Oracle databases are moved from on premise environments to AWS Oracle RDS, there may still be existing on premise Oracle (and non-Oracle) databases that need to replicate data to Oracle RDS. Options such as Oracle GoldenGate and Oracle Streams can be used.  There are also some other options: 

SymmetricDS is open source software : http://www.symmetricds.org
DBVisit : http://www.dbvisit.com/solutions/data_distribution/
Shareplex : http://www.quest.com/shareplex-for-oracle/
Pervasive : http://integration.pervasive.com/Integration-Scenarios/Data-Migration-ETL.aspx

Friday, June 7, 2013

DynamoDB cross region support


DynamoDB does currently not support cross-region replication. Therefore, were you to store your data in two (or more) regions, you would have to roll out your own implementation to replicate.

Wednesday, June 5, 2013

How to support on premise and EBS volume back up to S3 with one tool

There are many solutions to move data (backup, replicate, and synchronize) between your on premise environment and AWS S3. However, when you are looking for a solution that can backup your AWS EBS volumes to S3 there are not as many.  There are some vendor specific products (i.e. Oracle Secure Backup) but nothing that natively (at a file system or raw partition level) backups up EBS to S3.  Such a solution, would also have the added value of using the same tool / product to back up on premise disk and EBS volumes using the same method.  One company that offers such a solution is Ctera (http://www.ctera.com).  

Wednesday, May 1, 2013

AWS disaster recovery for applications, database and storage


When coming up with an approach to perform disaster recovery on AWS from your on premise environment,  you need to look at these layers of your system as individual entities.  This is because for each of them their are different tools and approaches.
1. Storage Layer: The storage layer includes multi media files, batch files, log files and any other files that not directly owned by the application or database server.  These files are typically on SAN or NAS devices on premise.
More information here:

2. Database Layer:  This would be data files associated with your relational or NoSQL database.  There are many database vendor specific tools. So the best approach is typically to use the tool or facility provided by the DB vendor.  
More information in these blog entries:
http://cloudconclave.blogspot.com/2013/01/disaster-recovery-in-cloud-options.html
http://cloudconclave.blogspot.com/2013/03/aws-ebs-snapshot-volumes-for-mysql.html – More on snapshots as apply to MySQL


http://cloudconclave.blogspot.com/2013/03/aws-ec2-oracle-database-backup.html - OSB or EBS snapshots
http://cloudconclave.blogspot.com/2012/10/oracle-helps-amazon-save-money-on.html – More on OSB
http://cloudconclave.blogspot.com/2013/01/oracle-exadata-dr-to-cloud.html - Exadata
http://cloudconclave.blogspot.com/2013/02/oracle-secure-backup-restore.html – Recovery speed for OSB



3. Application Layer:  The application layer includes all source code (Java, PHP, Ruby, HTML etc), log files and anything else associated with your application server.
More entries in these blog entries:







Tuesday, April 30, 2013

On premise application replication and snap shots to AWS for DR

When devising an disaster recovery approach that used AWS for DR of an on premise environment, there are literally hundreds of options for keeping the databases in sync. Each relational database vendor will have five of their own options and their partner ecosystems will offer many more.  For the application, it is different situation.  Their are not many features built into business application  and application servers.  Depending on what DR scenario you are using http://cloudconclave.blogspot.com/2012/11/aws-four-dr-scenarios.html you will be using either replication or snapshots to keep your AWS application in sync with your on premise environment.  
For warm standby and active-active you will need replication:

  • RSync : Rsync is cheap (free) and easy.  Rsync can be chatty, does not have compression built in, and you have to code parallelism. 
  • BitTorrent : BitTorrent is most often associated with moving large media files over the Internet in a fast matter. BitTorrent now has file syncing capabilities so can be used to sync application server files.  
  • Amazon Storage Gateway " attach these volumes as iSCSI devices to your on-premises application servers.  These files will be stored in S3 where they can be instantiate as EBS volumes.

  • Riverbed Whitewater - http://www.riverbed.com/products-solutions/products/cloud-storage-whitewater/ https://aws.amazon.com/marketplace/pp/B007O0FXW4/ref=srh_res_product_title?ie=UTF8&sr=0-2&qid=1366938137611#product-detail
  • Attunity CloudBeam - http://www.attunitycloudbeam.com/solutions/disaster-recovery  https://aws.amazon.com/marketplace/pp/B00B5PB8IM/ref=srh_res_product_title?ie=UTF8&sr=0-3&qid=1366938015695


  • For backup and restore and pilot light you can use snapshots. 

    • FTP or SFTP : FTP and SFTP are the two most common methods to transfer files over the internet.
    • HTTP or HTTPs : HTTP and HTTPS can be used but they are slower than S/FTP.  However, if you don't want to require or can not have an FTP client in use than HTTP(s) can be used.  Some companies may not want to open up another port to the outside world have an FTP server running.
    • AWS import/export : If files don't need to be synced that often, FedEx, UPS, or Post Office can be the least expensive method with the least amount of hassle (limited coding, no restarting needed)
    • Attunity CloudBeam to move files into S3 and than instantiate as EBS volumes.

    Oracle EBusiness Suite does offer rapid clone http://docs.oracle.com/cd/E18727_01/doc.121/e12841/T120505T120517.htm but this is not for moving production system data files but for setting test, dev, or moving a system to another machine. Weblogic also offers a cloning feature :http://docs.oracle.com/cd/E12839_01/core.1111/e10105/clone.htm

    Tuesday, April 23, 2013

    AWS RDS cross region replication


    RDS offers a simple way to replicate across AZs (Multi-AZ feature) for fault tolerance and high availability.  RDS cross region synchronization / replication provides additional fault tolerance.  This is an interesting situation because RDS is a 'black box'/PaaS/DaaS environment. Therefore, no access to the OS level is provided. This means this replication solutions like MySQL Replication or Oracle Goldengate can not be used.  Until RDS offers snapshots like the new EBS region snapshotting, you will have to use tools like BitTorrent or CloudBeam to move database dump files across regions.You could use DB links with materialized views as this article mentions: http://aws.amazon.com/articles/4173109646282306. DB Links are slow but if you don't have a low RTO and RPO you could use this...Or come up with your own creative solution.

    Monday, April 22, 2013

    Oracle WebLogic with AWS Auto Scaling

    The question of using Oracle WebLogic with AWS Auto Scaling and propagation of session state is often asked.  Before getting into the details, let's look at the two ways of handling session state at the server layer (using cookies in the browser can be used as well):
    1. Session stickiness : session data issue is to send all requests in a user session consistently to the same backend server. 
    2. Session in database : Another solution is to keep the per-session data in a database.  Of course, AWS ElastiCache, SimpleDB or DynamoDB, or RDS.

    If the session is stored in a database, nothing needs to be done when using Auto Scaling with WebLogic; for obvious reasons.


    When sticky sessions is used, nothing needs to be done, but the reason is not so obvious so let's discuss it.

    When a request comes into a WebLogic cluster via a load balancer (AWS ELB for example) or through Apache (mod_proxy_balancer plug in) the first time,  WebLogic creates an HTTP session on the primary node and also puts session state on a backup node. (You can provide guidance/control where to put the failure/backup).  If the server goes down that contains the primary WebLogic node, the new primary node will know where the backup session is stored and the session state will automatically get replicated to the new node.  So when AWS Auto Scaling is used there is nothing that needs to be done from an AWS perspective. 

    Oracle Data Guard on EC2


    Oracle Data Guard can be used to set up one or several slave databases which will be the foundation of a highly available environment. It maintains the standby databases as transaction- consistent copies of the primary database.  These instances can be placed in several availability zones.Then, if the production database becomes unavailable because of a planned or an unplanned outage of this instance or of the full availability zone, Data Guard can switch any standby database to the production role, minimizing the downtime associated with the outage.   The Active Data Guard module enables read-only access to the standby databases, thereby allowing customers to run read queries and reports on the standby instances, and to perform the backups from a standby instance.
    You can have zero data loss by using the appropriate data protection mode:http://docs.oracle.com/cd/B28359_01/server.111/b28294/protection.htm#CHDEDGIF