Mike Ault's thoughts on various topics, Oracle related and not. Note: I reserve the right to delete comments that are not contributing to the overall theme of the BLOG or are insulting or demeaning to anyone. The posts on this blog are provided “as is” with no warranties and confer no rights. The opinions expressed on this site are mine and mine alone, and do not necessarily represent those of my employer.

Monday, September 29, 2008

Oracle's Data(Warehouse)base Machine

I just got back from OOW08, my wife and I tagged a couple of days in Napa, California onto the conference for our 35st anniversary so I am still getting back into the swing of things.

The biggest news at the conference was Larry Ellison’s announcement of the Exadata storage concept and the Oracle Database Machine both developed jointly with HP. These new storage and database devices offer up to 168 terabytes of raw storage with 368 gigabytes of caching and 64 main CPUs in 8 stacked DL 360 G5 servers and each Exadata unit has a HP Proliant DL 180 G5 with dual quadcore CPUs, 8 gigabytes of memory and 12 SAS 300 GB or SATA 1 terabyte drives. The entire HP Oracle Database Machine contains 14 Exadata blocks and 8 – dual quadcore servers in a full configuration. The Exadata blocks can be purchased separately. There are 4-24 port Infiniband switches provided in the Database Machine. The entire device provides a throughput of 10.5 (SATA) to 14 (SAS) GB/second.

Now, each Exadata block can only provide 1 terabyte if the 300 GB drives are utilized and 3.3 terabytes if the 1 terabyte drives are used unless Oracle compression is also used. This space calculation (from Oracle documentation) is based on mirroring of all the drives and subtracting space for logs, undo and temp space. The usual “your mileage may vary” warning applies to this available space. ASM with what appears to be high redundancy storage is being used to manage the drives. So while raw storage appears to be 3.3 TB to 12 TB the actual space that ends up being usable is only 1/3 of those amounts. Each Exadata has 2 – 20 Gigabit Infiniband interfaces. However, the blocks can only support 1 GB per second of output with the SAS configuration and 750 MB per second in the SATA configuration.

The Oracle Database Machine was actually designed for large data warehouses but Larry assured us we could use it for OLTP applications as well. Performance improvements of 10X to 50X if you move your application to the Database Machine are promised. This dramatic improvement over existing data warehouse systems is provided through placing an Oracle provided parallel processing engine on each Exadata building block so instead of passing data blocks, results are returned. How the latency of the drives is being defeated wasn’t fully explained.

The HP Oracle Database Machine must run Oracle11g,, RAC and Linux and each Exadata block must have the new Oracle 11g parallel query engine installed. So in a full configuration you are on the tab for a 64 CPU Oracle and RAC license and 112 Oracle parallel query licenses (assuming it is per CPU, if it is per Exadata block then it will be 14) as well as any Grid control licenses you may need. The base cost of the full Database Machine is around $650K which seems quite a bargain for 14-46 terabytes of usable storage and a 64 processor stack, however, you will also need over a million dollars in licenses even with aggressive reductions from your sales representative.

The HP-Oracle Database Machine only works with Oracle databases (just thought I should throw that in.)

Whew! The HP-Oracle database Machine offers quite an impressive array of facts, figures, promises and price tags. It will be interesting to see how this all sorts out over the coming months. Will there be enough profit in the HP-Oracle Database Machine to keep HP interested? Or is this another network computer? For those too young to remember Larry’s last hardware foray was the Network Computer, a device that would replace all the desktops and centralize application and data storage, it failed. [Note: don’t forget about Pillar Data another Ellision investment that would seem to be hurt by this announcement]. I believe this system is designed to help Oracle protect turf from Netezza and promote growth in the analytics market. Targeting the product to OLTP environments is just sloppy marketing as the system will not offer the latency needed in real OLTP transaction intensive shops. These applications do not need parallel queries, they need low latency database writes and reads.

So for nearly 2 million dollars (licenses plus hardware) you get a dedicated Oracle server in a rack with 64 CPUs of central processing and 46 terabytes of usable storage managed by 112 block resident CPUs and a wee bit less than 224 gigabytes of cache area (168 gigabytes of cache were promised after processing overhead was subtracted). However, you must throw away your existing infrastructure, upgrade to Oracle11g and marry your future to the HP Oracle Database Machine to do so.

What might be an alternative? Well, how about keeping your existing hardware, keep your existing licenses, and just purchase solid state disks to supplement your existing technology stack? For that same amount of money you will shortly be able to get the same usable capacity of Texas Memory Systems RamSan devices. By my estimates that will give you 600,000 IOPS, 9 GB/sec bandwidth (using fibre Fibre Channel , more withor Infiniband), 48 terabytes of non-volatile flash storage[S1] , 384 GB of DDR cache and a speed up of 10-50X depending on the query (based on tests against the TPCH data set using disks and the equivalent Ram-San SSD configuration). More importantly, this performance can be delivered with sub-millisecond response time. At Oracle World I was presenting on the importance of latency to Oracle databases. The Exadata is massive and offers great bandwidth but will have nearly awful disk access tedencies due to the massive and slow disk drives included in the system.

Of course, if you don’t need 48 terabytes you can purchase Ram-San SSD technology from 32 gigabytes up to whatever you actually need. Ram-San SSD technology works with all Oracle versions and requires no special licensing or changes to your system. Ram-San uses fibre channel or Infiniband for connection to your infrastructure and looks identical to a disk drive once configured (it takes about 10 minutes.)

Oh, and the Ram-San doesn’t care if you are on Oracle, SQL Server, MySQL, or cousin Joe’s Ozark Mountain special database.

So let’s recap:

Purchase the complex, tied to Oracle with a golden chain, HP-Oracle Database Machine and end up throwing away your existing technology stack, and spend up to 2 million dollars for the privilege, for a speed up of 10x to 50x on data warehouse type queries.


Purchase just as much Ram-San SSD technology as you need ($43K base price for 32 GB mirrored), keep your existing hardware and license structure (or possibly reduce it) and get a 10x to 50x speed up on data warehouse type queries, with the freedom to change databases as you need to.

Call me simple, but I think I see the proper choice.

[S1]Note that the 8TB systems are projected to have 1.5GB/second of bandwidth sustained reads or writes (to Flash). This can be accomplished with four 4Gbit FC ports or two 4x IB ports per unit.


Brent Ozar said...

Nice writeup, and there's even another RAMSAN advantage you didn't mention: you don't have to put all of your data on the fast storage. You can keep your archival data (or rarely accessed, inexpensive data) on your existing storage, and put "hot" data on the fast storage. You probably don't need everything on the blazing fast, expensive storage.

Mike said...


Quite right, SSD (Ram-San) acts as tier zero storage for the items that require the fastest access while tier 1 and lower can be disk based although we recommend that tier 0 be DDR based storage, tier 1 be flash based and then tied 2 fast disk and so on.

Ben Prusinski said...

Hi Mike,

You have some valid points. It was great to finally meet you at the Oracle conference last week.

Best regards,
Ben Prusinski

Mike said...


Thanks! It was good to meet you "In the flesh" as it were..


Richard said...

Interesting stuff!

Who in their right mind would take a $2M punt on this Frankenstein's monster creation of Oracle & HP? I seriously expect many potential purchasers will say "No", simply because $2M is a gigantic price tag and one would have to be VERY sure that Oracle & HP stayed pals for the VERY long time one would expect one's pricey kit to last.

The fact that it's Oracle or the highway is hardly encouraging, too.

I am surprised that this Oracle wonderment ever got off the ground.

Alex said...


If you would know the prices of the existing(in production) data warehouse hardware ,2 mil $ isn;t much(and they all have performance problems)...on the other hand,this is list price,if you order you'll probably receive serious discount
Also,a key advantage of exadata storage server is the fact the the queries are decomposed in small pieces and executed on nodes where the storage resides.No matter how much ssd you'll use,you won;t achieve this .

Mike said...


The decomp of the query is to speed them up, for all but the most complex queries this would not be needed with high speed storage such as SSD.

The same algorithms that are used to speed the query on disks would also speed it on SSD.


Alex said...


I'm quite sure that in the near future,SSD disks will replace SAS disks inside the exadata storage server.After all,the price for Oracle licenses is much bigger than the actual hardware price.
It will be clear some performance improvement.
But,I believe that the serious performance improvement comes from the exadata storage server concept:the ability to decompose the queries in smaller and parallel quries which are executed on the node where the date resides and provide smaller result sets who are sent to the master node to be finally processed.
I haven't worked with this product,but,read some articles about the results of this new technology:ten times faster than the previous hardware.