Mike Ault's Blog

Mike Ault's thoughts on various topics, Oracle related and not. Note: I reserve the right to delete comments that are not contributing to the overall theme of the BLOG or are insulting or demeaning to anyone. The posts on this blog are provided “as is” with no warranties and confer no rights. The opinions expressed on this site are mine and mine alone, and do not necessarily represent those of my employer.

Monday, September 08, 2014

Status of Home Lab

Well, all the equipment is stacked onto a heavy duty plastic storage shelf. I will probably retrieve my wire rack from Atlanta next time I am through there. Of course if anyone wants to donate a computer rack (even a half-rack would do) I would be willing to consider it…J

When I unpacked the FC switch I realized they hadn’t included GBICs so I had to order some off Ebay. In addition the FC optical cables I had were the wrong type so I had to order more. Looking at the small SAN I purchased I discovered it is using a serial connection rather than a Ethernet so I ordered the appropriate adapter…from Hong Kong. Unfortunately it came with a 230 v power adapter (not even dual voltage) so I had to order a new power supply. The serial-to-Ethernet adapter also had a male plug as does the SAN, so back to Ebay for a female-to-female adapter. Hopefully by tomorrow all these various cables, adapters and power supplies will have arrived and I can finally start configuring the system.

At least all the KVA cables worked (Keyboard-Video Adapter interface) and I can talk to both servers from the same touchpad/ keyboard and Monitor. The servers come with a minimal install of Windows Server 8, unfortunately it seems someone has set a password. Luckily I am blowing the whole OS setup away on both servers and installing Oracle Linux, so the lack of a Windows password really doesn’t bother me that much.

Once everything is up I get to set up the network (2-1GB Enet switches, one for general networking between the operator, servers, FC switch and SAN and the interconnect, set of the 4GB SAN switch for proper zoning, format and configure the SAN and then configure the OS on the servers and install Oracle.

I’ll keep you posted.

I am also doing an All-Flash-Array (AFA) testing setup at work for the IDC based testing. We are looking at setting up VDBench through Windows to test the IBM FlashSystem 840 in accordance with the IDC guidelines and then having the setup and generated setup documentation available so any client can use the guidelines for testing to satisfy them we aren’t using smoke and mirrors in our claims for performance of the FS840. I will post any lessons learned for that as I experience them.

This weekend I split and stacked about a cord of wood here at Sunset Cove for the basement woodstove in case the projections that this will be another record snow year for Tennessee, North Carolina and points North comes true. Luckily I have a hydraulic splitter that made quick work of the actual splitting, but the loading, unloading and stacking still had to be done by hand.  Also had to adjust the brakes and clutch on the tractor and buy a new battery for it. The clutch still needs a tweak as the Power-Takeoff (PTO) isn’t disengaging properly when the clutch is fully depressed.

Looks like next weekend I get to pick more pears. Since the apple crop this year was a bit disappointing I guess instead of hard cider I will make what they call Peary which is Pear Cider. I made five gallons of hard apple cider last year but it only lasted until half-way through the winter, so to make enough for it to last until the next crop comes in I guess I will have to do 10-15 gallons which will be around three bushels of pears. If I can get it setup next weekend then it can ferment for the three weeks I am going to be gone to OOW, Edge and vacation. Then I will have just enough time to filter it and bottle it before the relatives show up for the Punkin Chunkin in late October. If you have never seen a Punkin Chunkin, check it out online, there is probably one near you. Essentially you build the best punkin throwing device you can envision then compete for distance, last year 900 yards was the winning distance.

The gardens need to be tilled under as we are getting the last of the tomatoes and cucumbers now. Luckily I have a yard service that will do that for me. I’ll have them weed wack the gardens to grade then till the plants under and then place some of that weed-block fabric over the top to kill off any weeds that want to try a comeback. In a few weeks we’ll plant the winter garden items.

The hydroponics setups are still languishing in the greenhouse, afraid I haven’t had a chance to get the lower well re-lined and I need a good supply of water to do them properly and about 6 weeks of non-travel so I can monitor the ammonia and nitrate levels daily to get the biologics established on the growth media, then you put in the Tilapia and get the nitrate cycle going and then start growing some leafy vegetables like Lettice, Kale, Boc choy or even good old spinach. Maybe later I will look at vertical gardening techniques to do something more dense.

Of course deer season is just around the corner and I have that new platform stand to put together down overlooking the Orchard. I spent several weeks trying to figure out where the deer where last year only to realize they were busy raiding the orchard for low-hanging and windfall fruit. Since it is near the road I will need to use archery only. That reminds me, have to tune the new compound bow and adjust the draw length to allow for my d-loop and trigger as well as sight in the black powder and deer rifles and check that the crossbow is still sighted in as well.

Then there is also the ongoing work with my half ownership in the Wherehouse of Murphy, an art gallery and hand-made item, antique and general merchandise store. We also do free concerts, Yoga and art classes. Luckily I am mostly a silent partner and Joy Stein handles most of that for us.

Did I mention I am also writing the third book in the Rex Franklin series? Yep, Rex is installing a Tsunami warning system in the Turks and Caicos when he gets roped into helping out his least favorite ex-boss fetch a fortune in emeralds from a Spanish wreck….

Also toying with writing a book on Oracle AWR Analysis, in my spare time.

Add into that the travel and full-time work load of IBM and it rounds out my schedule nicely.

No rest for the wicked I guess. And people wonder why I laugh when they ask “But Mike if you retired what would you do with yourself?”

Friday, August 29, 2014

A Friday at Bradley International

Well, sitting here in Poughkeepsie waiting on my flight back to Atlanta. Asfar as the proof of concept I was here for, let me give you one take-away, when Oracle says put the RAT clients on a different server, they mean it! Of course it didn't help that the client was testing 24 databases simultaneously on the same Power8 box...
Anyway, John with the IBM lab and I convinced them to move RAT to its own server and things are now going great. 
I decided I better put down on paper what I need to do for the home lab:
Action Plan for Lab Set up
  1. Convert files to disk images on DVD
  2. Cable up systems
    1. Interconnet
                                                               i.      2-1 GB Ethernet cables and switch

    1. Network
                                                               i.      2-1 GB ethernet cables for servers

                                                             ii.      1 cable for SAN

                                                            iii.      1 Cable for Storage Switch

                                                           iv.      1 GB Ethernet switch

                                                             v.      1 wifi switch for remote access (2nd floor office to basement lab)

    1. SAN
                                                               i.      Install 4-4 GB HBAs (2 per server installed)

                                                             ii.      Install 6-4 GB FC optical cables between components and Switch

1.       2 for SAN

2.       2 for each server

                                                            iii.      4 GB switch setup

1.       Zone SAN and servers

    1. Install OS
                                                               i.      Install Oracle enterprise Linux on each server

                                                             ii.      Locate and install any needed drivers for HBA, monitor, etc.

                                                            iii.      Implement multipathing to SAN

    1. Install Oracle12c
                                                               i.      Install grid software

                                                             ii.      Install RAC software

                                                            iii.      Create database

    1. Download SLOB and/or Swingbench and install
    2. Create and load databases
Trying to keep it simple to allow for some flexibility.

Wednesday, August 27, 2014

New Lab status

Well, I have all the equipment, power cords and Ethernet cords. I am waiting on some Fibre channel cables that should arrive by Tuesday at the latest. I purchased a WiFi Ethernet switch so I can connect to the system form my loft office without having to go all the way downstars. I salvaged my KVM switch and cables along with the power cords, Ethernet switches and touchpad keyboard and monitor from my old lab. Should get the system mostly wired and cables on Friday night and be able to start loading the OS on Monday with any luck, then begins the joy of loading Oracle12c! I have a line on a couple of FusionIO 190 gb cards I made an offer on them, we will see if they will give them up. I offered $175 apiece for them since I have no idea how many PE cycles they have on them. I figure to use the FusionIO cards as flash cache for the RAC instances since it won’t be shared storage.
I am at the Atlantsa airport waiting on a flight to Poughkeepsie where I will be helping on a proof of concept tomorrow, it is about boarding time so I will keep this entry short.

Thursday, August 21, 2014

Upgrading the Lab

I am afraid I have been a bit negligent in my blog posts of late. I will endeavor to make them more frequently and of course try to make them interesting and enlightening! I am embarking on a new project, I am rebuilding my computer lab with newer, better equipment. To that end I just completed the purchase of the following gear:

  1. 2 – Dell PowerEdge 2950 III servers
    1. 2x2.83 GHz E5440 Quad core
    2. 16 GB memory
    3. 4x500 GB internal drives
    4. PERC6i 2PS
  2. 4 – Dell Qlogic QLE2460 single port 4 GB FC PCIe HBAs
  3. 1 – A7394a HP Storageworks SAN 4 GB/s Switch with 32 active ports
  4. 1- CIPRIO MediaVault RX 4 GB dual FC Storage 5 TB array (capable of RAID0, RAID1 or RAID3)

Now, you should realize I have access to a full IBM Power8 server with an IBM FlashSystem 840 attached via remote access to the STG lab, but when I am testing and learning I prefer a system where if I crash it or trash it, it will effect only my work. Once I learn what not to do I will take advantage of the advanced equipment that IBM has made available.

It is interesting to note that when I started writing Oracle books I purchased a new server (back in the late 1990’s) at that time I got a hyper-threaded, dual core system with 2 GB of memory for about $5K. When RAC was released I needed to upgrade so I purchased 2 – single core hyper-threaded servers with 4 GB or memory, a 1 GB switch and 2 disk arrays at a total cost of around $3K. For all of the equipment listed above I paid $1.6K on Ebay. It just goes to show the incredible decrease in costs and increase in capabilities that have occurred over the last 20 years.

Next I will be acquiring Oracle OEL for the operating system and Oracle12c for testing and learning on. I will post my lessons learned as I go along. I hope to test many of the new features of 12c such as container databases, in-memory database options, use of Flash Cache in container, regular and RAC 12c databases and of course use of the in-memory option with a 12c RAC database.

Next I have to decide whether to install the equipment in the house or out in the shed, if I do it in the shed I have to build a computer room out there and put in a small AC unit. I an existing AC unit that will work, just have to cut the needed ports in the side of the building and install the intake/outlet lines for the air handler section. Of course instead of building a room it may be more cost effective to buy a small 8x10 foot shed and place it in the larger 30x50 foot equipment shed. I’ll post pictures along with the blog entries as I go along.

Friday, June 13, 2014


Excerpt from a meeting yesterday:

"IBM now has the VSC that wraps around multiple clustered SVCs providing virtualization of the XIV, V7K and FS840. The FS840 is completely built on FPGA technology utilizing ASIC to provide a purpose built architecture driving latency to microseconds and proving the highest IOPS. We support FC, IB, FCoE, and iSCSI via the SVC and IB and FC directly."

I realized that I completely understood what had just been said, guess I am full Geek now. Over half of understanding a technology is understanding its verbiage, in this case its acronyms. I realized that if I was talking to someone on my cell phone and said this, 95% of the people around me would think I was channeling Robin Williams from the movie "Good Morning Vietnam!" with his famous acronym speak when talking about the visit of the vice president. If you haven't heard it, I am sure it is on YouTube or available somewhere on line. Here it is written but it just isn't as funny:

:Adrian Cronauer: Excuse me, sir. Seeing as how the V.P. is such a V.I.P., shouldn't we keep the P.C. on the Q.T.? 'Cause if it leaks to the V.C. he could end up M.I.A., and then we'd all be put out in K.P.

Anyway, hope you can all LOL and don't have a SHTF moment anytime soon.

Wednesday, September 25, 2013

How is publishing a successful book like Nuclear Physics?

You are probably looking at this title and wondering what the heck I am talking about. Let me explain. When I was in the Navy I was a Nuke, that meant we attended 2 years of training where a $20K dollar (in 1973 dollars) education was shoved where the sun don’t shine one nickel at a time. In that training we learned nuclear physics, chemistry, electrical theory, fluid dynamics and heat transfer, in short, anything they thought we would need to run a nuclear power plant on a Navy ship.

One of the classes was nuclear physics in this class we learned about the nuclear multiplication formula. Now I don’t have fancy formula rendering software but here is what this formula looked like:



K is the multiplication factor
N is the thermal neutron production from fission
F is the probability that an absorbed neutron is absorbed in fuel
P is the fraction of neutrons that slow down to thermal energies without getting absorbed
e is the fast fission factor (ratio of total neutrons produced to total neutrons from thermal fission
Pth is the thermal neutron non-leakage factor
Pf is the fast neutron non-leakage factor

If K is less than 1 then the reactor is sub critical and if it equals 1 then the reactor is steady state and critical and if it is greater than 1 then the reactor is super-critical and power is increasing.

So…what the heck does this have to do with selling a book successfully? Well, let’s re name the terms of the equation a bit:

K = Chance that book will succeed
N is the number of books sent out/given away/available to the public
F is the probability a book will reach someone who will read it
P is the percentage of books that never get read
e is the total number of books read in the first week they are purchased over the total number of books read within the first month after they are sold
Pth is the ratio of books read within the last month that result in an additional sale
Pf is the ratio of books read within the last week that result in an additional sale

So, if a lot of books are given away/sent out/sold initially but none of them result in new sales (Pth and Pf are near zero) then the book won’t make it. As long as each book put into someone’s hands generates more than 1 additional sale each the book should be successful. If K=1 then the book will have steady sales, but not spectacular, if it is less than 1 the book is dead and greater than 1 then the sales are accelerating and the book is successful.

Ok, so it has been a slow day, see what happens when you have too many things stuffed in your head and your idle brain starts making connections….

Wednesday, August 07, 2013


As you can tell from the title I have little use of the DISPATCHERS parameter. In the default init.ora provided by Oracle DBCA this value is set, so. Many people leave it set thinking Oracle is saying it is a good idea or best practice to have it set. From the Oracle11g manual:

“Configure the DISPATCHERS parameter if either of the following conditions apply:
You need to configure another protocol other than TCP/IP
Configure a protocol address with one of the following attributes:
You want to configure one or more of the optional dispatcher attributes
  • POOL (POO)
  • TICKS (TIC or TICK)”

Obviously, the second option is only when you have the DISPATCHERS already set. Most of the time I see DISPATCHERS set to configure the XDB protocol, for example:


Now, the question is, when do you need the XDB protocol? Going again to the Oracle11g

If you create your Oracle database with Database Configuration Assistant (DBCA), DBCA configures a dispatcher for Oracle XML DB (XDB). This is because XDB protocols like HTTP and FTP require shared server. This results in a SHARED_SERVER value of 1. Although shared server is enabled, this configuration permits only sessions that connect to the XDB service to use shared server. To enable shared server for regular database sessions (for submitting SQL statements), you must add an additional dispatcher configuration, or replace the existing configuration with one that is not specific to XDB. See "Configuring Dispatchers" for instructions.”

So, whether you need it or not, DBCA sets it up, even if you tell it you don’t want shared servers initialized. You will only use XDB if you are doing XML, FTP or HTTP transfers in and out of your database or if you are using APEX you will need the XDB protocol. Now, if you use XDB protocols and need DISPATCHERS set that way, then go no further, you are set correctly (however, you may want to see what this does to your system, so read on!)

So you are probably thinking: “Big deal, so DISPATCHERS is set, so what?” I mean, compared to the myriad other things Oracle creates and runs in the background on our behalf this seems a bit mild, right? Actually, no, let me explain.

Setting DISPATCHERS or setting SHARED_SERVERS turns on shared servers in Oracle. Shared servers are a means to do connection pooling in Oracle with SHARED_SERVERS configuring how many shared processes to have and DISPATCHERS setting up the master DISPATCHER processes which then serves out the connections to the connection processes setup with SHARED_SERVERS. So, if no one connects to the XDB process, shared servers are never used, so what is the big deal?

Well, shared server expects connections to be shared; certain types of shared connections must share memory areas. One example is that some types of memory sharing in PL/SQL will move form the shared pool to the large pool, this can improve PL/SQL performance form some operations, in version 9 of Oracle it was a way to prevent certain out of memory operation from happening in the SQL pool as a result of PL/.SQL. Shared servers also may require processes to share temporary segments.

This is were the rubber meets the road, when the processes must share memory areas (for example temporary segments) then those areas cannot automatically be controlled by automatic memory management and won’t fall under PGA_AGGREGATE_TARGET, MEMORY_MAX_TARGET or SGA_MAX_SIZE  or their related parameters. Processes such as parallel query slaves and RMAN backup processes fall under this set of processes as do some parallel back ground and DDL statements. In addition, if you have STAR_TRANSFORMATION set to TRUE and DISPATCHERS set, then CREATE_BITMAP_AREA_SIZE and BITMAP_MERGE_AREA_SIZE also fall out from under automatic controls.

So, what did I just say? Essentially DISPATCHERS turns off PGA_AGGREGATE_TARGET control for temporary segments for certain parallel and background processes and removes bitmap temporary sizing parameters from AMM control. Under this situation, you must use manual sort/hash area control using the SORT_AREA_SIZE parameter. In the Oracle11g 11.2 manual about SORT_AREA_SIZE, CREATE_BITMAP_AREA_SIZE, BITMAP_MERGE_AREA_SIZE:

Oracle does not recommend using the SORT_AREA_SIZE parameter unless the instance is configured with the shared server option. Oracle recommends that you enable automatic sizing of SQL working areas by setting PGA_AGGREGATE_TARGET instead. SORT_AREA_SIZE is retained for backward compatibility.
Oracle does not recommend using the CREATE_BITMAP_AREA_SIZE parameter unless the instance is configured with the shared server option. Oracle recommends that you enable automatic sizing of SQL working areas by setting PGA_AGGREGATE_TARGET instead. CREATE_BITMAP_AREA_SIZE is retained for backward compatibility.

Oracle does not recommend using the BITMAP_MERGE_AREA_SIZE parameter unless the instance is configured with the shared server option. Oracle recommends that you enable automatic sizing of SQL working areas by setting PGA_AGGREGATE_TARGET instead. BITMAP_MERGE_AREA_SIZE is retained for backward compatibility.”

And, so far, these are the only places this is mentioned, other than HASH_AREA_SIZE, but since HASH_AREA_SIZE is derived from the sessions SORT_AREA_SIZE I am assuming that if we set SORT_AREA_SIZE, HASH_AREA_SIZE will follow, but that could be a bad assumption.

So how can you tell if this is happening in your system? First, you will get a load of sorts and temporary activities between 64K and 512 mb on your system. Check out the AWR PGA Aggregate Target histogram to see this. Why do I say that? Well, according to the released algorithm details of how PGA_AGGREGATE_TARGET works it uses the undocumented parameter ” _PGA_MAX_SIZE” to determine how much temporary area to give to specific user up to 5% of the total PGA_AGGREGATE_TARGET setting. Now on some systems this defaults to 256 megabytes and on others to 512 megabytes. So in the most current releases the 512 megabyte setting is the default so PGA_AGGREGATE_TARGET should handle most sorts and temporary actions at less than 512 megabytes without resorting to going to disk. The reason for 64k being the minimum is that is the setting for SORT_AREA_SIZE if you haven’t reset it so up to 64K the default SORT_AREA_SIZE should handle any out-of-band sort requests. Here is an example PGA Aggregate Target Histogram showing out-of-band sorts.

PGA Aggr Target Histogram

  • Optimal Executions are purely in-memory operations
Low Optimal
High Optimal
Total Execs
Optimal Execs
1-Pass Execs
M-Pass Execs

As you can see from the above chart only 24 out of 253 of the temporary executions-to-disk were greater than 512 MB and none were lower than 64 KB. This could indicate out-of-band sorts if the PGA_AGGREGATE_TARGET is set to a sufficient size and DISPATCHERS or SHARED_SERVERS is turned on.

Temporary space requests between 64K and 512MB that generate actual disk IO activity are called (at least by me) out-of-band sorts. The best way to clean these up is to turn off the DISPATCHERS parameter if you aren’t using XML DB and XDB related protocols. Also make sure SHARED_SERVERS isn’t set. If you can’t turn off DISPATCHERS because you are using the XDB service it creates, then set SORT_AREA_SIZE to 512MB.

When should you consider turning off DISPATCHERS?
  1. You are running a 3rd party application which makes no use of HTTP, FTP or APEX
  2. You are running an in house application that makes no use of HTTP, FTP or APEX
  3. You aren’t using XML, HTTP, FTP or APEX in your application.

One thing to remember, if you are only using XDB services for batch processing or only for intermittent connections you can actually set DISPATCHERS dynamically and turn it on and off at will. Leaving DISPATCHERS set 24x7 when you only need it for a 10 minute window at night for a batch process can cause many issues as shown above. Consider only turning on DISPATCHERS or SHARED_SERVERS when you need it.