Geolocation information for IP addresses

MaxMind (https://www.maxmind.com/en/geolocation_landing) can be used to provide  you the ability to identify the location, organization, connection speed, and user type of your Internet visitors. 

AWS EBS PIOPS : block size and IOPS

Having spent more time in the database world than in the web development world, I am accustomed to measuring (database) performance/through put in terms of IOPS or TPS.  The web/video/image world like to use MB/sec.  Why I am saying this? Because it relates to the conversation about getting a certain level of PIOPS (based upon a 16 KB block) on AWS EBS and how this effects MB/sec.  MB/sec, I am beginning to understand, and maybe move to the 'dark side', is the ultimate measure of disk 'performance'.  

Example: A 2000 Provisioned IOPS volume can handle:

•2000 16KB read/write per second, or 1000 32KB read/write per second, or 500 64KB read/write per second 

•You will get consistent 32 MB/sec throughput (with 16KB or higher IOs)

•Perform an index creation action and sends I/O of 32K, IOPS becomes 1000, you still get 32MB/sec throughput

•On best effort, you may get up to 40 MB/sec throughput 

So, you may be better off using a 64 KB block size but your PIOPS will show up as lower but your MB/sec could be better.

Data warehousing load times

Some times when people are moving to AWS for their databases they are looking for load times in the range of 2 TB per hour (this is 650+ MB/second).  

1. Green Plum : http://www.emc.com/collateral/hardware/data-sheet/h7419-greenplum-dca-ds.pdf : 10 TB to 60 TB per hour claimed in this document.

2. Exadata : http://www.oracle.com/ocom/groups/public/@otn/documents/webcontent/173705.pdf  Data Load Rate: Up to 12 TB/hour

Numbers listed in this book:  http://books.google.com/books?id=Whx9nldv8UEC&pg=PA102&lpg=PA102&dq=oracle+database+loads+terabytes+per+hour&source=bl&ots=VEXelUIhd2&sig=9LW4T0PgottDF2kpgvUobd2FonQ&hl=en&sa=X&ei=PHjyUcPKMuKbygHuyIDgBg&ved=0CHkQ6AEwBw#v=onepage&q=oracle%20database%20loads%20terabytes%20per%20hour&f=false

Of course,  Greenplum and Exadata are significant monetary investments.

Paraccel claims to achieve 2 TB an hour: http://www.paraccel.com/resources/Analyst-Reports/TDWI_Checklist-Report_Analytic-Databases-for-Big-Data.pdf

Network speeds and Moore's Law

The speed of the Ethernet has increased from 1 Gbps in 1997 to 100 Gbps in 2010; this increase is slightly slower than the Moore law for traffic [251] which would require 1 Tbps Ethernet by 2013.

AWS import export : When to use import export

This web page has some guidelines on when to use AWS import export and when to use a network connection:
http://aws.amazon.com/importexport/

Go down to this section: "When to Use Import/Export"

I often get asked this question when someone wants to move large Oracle database to the cloud.

Copying 1 TB of data from on premise RDBMS to AWS/EC2

I often get asked how long it will take to transmit a 1 TB database (assumes that the database is actually 1 TB in size and not just allocated at 1 TB). Elapsed time is going to depend on many factors: speed of connection, compression used (if any), can load be done in parallel etc.  The fastest load times will happen using AWS import/export (http://aws.amazon.com/importexport/ ).  The next fastest will be using AWS Direct Connect (http://aws.amazon.com/directconnect). Direct Connect provides speeds of 1 Gbps to 10Gbps.  Using a 1 Gbps line, it would take about 3-4 hours (here is a good site: http://fasterdata.es.net/fasterdata-home/requirements-and-expectations/).    Assuming an internet connection speed of 100 Mbps (100Base-T connection) it would take about a day to transfer using a traditional connection to AWS/EC2  Both DirectConnect and Import/export provide a secure method of transfer.