Book Review – Oracle Information Integration Migration And Consolidation

I recently had the chance to look over the book “Oracle Information Integration Migration and Consolidation” by Jason Williamson, published by PackT Publishing.

The book deals with all aspects of migration to Oracle from various Oracle and non-Oracle sources. In a soup-to-nuts manner, the book covers the major tools to move data into and out of Oracle including OWB, SQLPLUS, EXPDP, IMPDP, RMAN, SQL*DEVELOPER, GoldenGate, and various ETL type tools including some form 3rd party companies.

The book includes many examples and use cases and numerous charts showing what tools should be used for which tasks. The book is written in a down-to-earth style and was easy to read.

I would recommend this book for anyone facing a data migration, platform change or complete system re-write into Oracle from another source.

The book is available here:

http://www.packtpub.com/oracle-information-integration-migration-and-consolidation/book
http://www.amazon.com/Oracle-Information-Integration-Migration-Consolidation/dp/1849682208/ref=sr_1_fkmr0_1?ie=UTF8&qid=1321543957&sr=8-1-fkmr0

Minimizing Re-Rebuild Time with Benchmark Factory from Quest

In my job as Oracle Guru for Teas memory Systems I perform a multitude of bechmarks to showcase the performance of our SSD products. One tool I have found invaluable while doing these benchmark tests is the Benchmark Factory from Quest Software, Inc. as it allows TPC-C, TPC-E and TPC-H as well as custom and other recognized benchmarks to be easily performed and provides decent reporting of the results. However, there is one fly in the ointment.

Benchmark factory does great when the benchmark size is small, however, its time to reload the database grows exponentially as the size of the database increases, for example, the creation of a 3000 warehouse TPC-C takes over 4 hours using a 16-CPU server with plenty of memory and SSD for storage, I can only imagine how long it would take with smaller servers using normal hard drive based SAN or JBOD storage. So, how can this obstacle be overcome?

First, I had Quest provide some parallel load scripts which allow me to load all 9 tables independently. Using the standard load utility it loaded them serially and any errors resulted in having to restart the load process from the beginning. With the independent load scripts you can load any combination of the tables, restarting any that may fail due to space or undo problems as needed. These are the load scripts that reduced the time to 4 hours mentioned above. They are available on the Quest website. However, spending 4 hours to refresh a database to perform a 45 minute test seems rather much, is there anything else that can be done to reduce this time further?

The http://www.tpc.org website has the dbgen utility that can be used to create load tables, then you create a set of SQL Loader files to map them into the database and you can load in parallel, this reduces the load time to about an hour, you can also map the external files as external tables. Of course, rather than deal with learning a second utility (which you have to download and link as a C routine) I just loaded the tables using BMF then before messing with them, created a set of external table using simple CTAS and the external table clauses as required. Now my load time is down to about an hour. I could further reduce the load time by parallelizing specific table loads. Note that to use the ORACLE_DATAPUMP as a source to insert from you must set the FILESYSTEMIO_OPTIONS parameter to NONE. Of course what about statistics? Statistics on multi-million row tables can take a couple of hours to generate as well.

Command to create a set of external table build scripts:
(you must use the “create directory” command to create the ext_tab directory entry first)

select 'create table ext_'||table_name||'
organization external (
type oracle_datapump
default directory ext_tab
location ('||chr(39)||'ext_'||table_name||'.ext'||chr(39)||')) AS
select * from '||table_name||';' from user_tables
/

Example result script:

create table ext_C_HISTORY
organization external (
type oracle_datapump
default directory ext_tab
location ('ext_C_HISTORY.ext')) AS
select * from C_HISTORY;

Following the data load and index build I did a statistics collection using DBMS_STATS.GATHER_SCHEMA_STATS then I stored the statistics into a statistics export table. Since I have the statistics stored it is simply a matter of importing the statistics at the conclusion of the reload.

Commands to export statistics:

EXEC DBMS_STATS.CREATE_STAT_TABLE('TPCC', 'STATS_TABLE');
EXEC DBMS_STATS.export_schema_stats('TPCC','STATS_TABLE',NULL,'TPCC');

Command to import statistics:

EXEC DBMS_STATS.import_schema_stats('TPCC','STATS_TABLE',NULL,'TPCC');

So now my data reload simply becomes a set of truncate table commands followed by an insert with the APPEND hint using a SELECT * from the external tables followed by a statistics import. This cuts the turn around time from around 6-8 hours to less than 2 for a 3000 warehouse TPC-C benchmark.

Example INSERT command:

insert /*+append*/ into c_customer select * from ext_c_customer;

Fun with External Tables

During recent testing, I came upon an interesting error. This is on Oracle Linux in an 11.2.0.2 enterprise database. I am doing TPCC testing and wanted a faster way to reload the tables between the tests rather than using the Benchmark Factory loader, which takes forever! I decided to use a datapump format external table using a CTAS to create it for each table after reloading the data then simply do an insert using an APPEND hint to reload the tables following a test and subsequent table recreation (since some are single table clusters and hash clusters truncate doesn’t work so they must be dropped and rebuilt.)

I used the following to create an external table:

create table ext_C_STOCK
organization external (
type oracle_datapump
default directory ext_tab
location ('ext_C_STOCK.ext')) AS
select * from C_STOCK;

The C_STOCK table is part of a single table cluster.

After a test run I dropped and recreated the C_STOCK cluster and table as an empty table then I tried to reload it from the external table:

insert /*+append*/ into C_STOCK select * from ext_C_STOCK;

To which I received:

insert /*+append*/ into C_STOCK select * from ext_C_STOCK
*
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-31619: invalid dump file "/mnt/exttables/ext_C_STOCK.ext"
ORA-27072: File I/O error
Linux-x86_64 Error: 22: Invalid argument
Additional information: 4
Additional information: 1
Additional information: -1

The problem turned out to be that the initialization parameter filesystemio_options was set to setall, changing it’s value to NONE fixed the issue.

What Are We Waiting For?

It seems everything but the temperatures are in a slump. The economy, hiring, employment, purchasing, proper government action, all seem to be missing. My question is what are we waiting for? We aren’t the ineffective government, we aren’t the ones making poor decisions, at least for the most part.

Perhaps we need to step forward and start movement. We need to do something positive, each of us. Perhaps it would just be cleaning up our yard, giving a bit more at church or working just a bit harder at our jobs. If everyone ignored Washington and just did a little more for each other I think we could positively effect not only our lives, but everyone else’s as well. If all of us treat each other honestly and fairly maybe it will trickle up.

Maybe instead of charging as much as we can, we just need to charge as much as we need to for goods and services. I am tired of seeing ten cents worth of plastic being sold for several dollars. How many employers could let their employees work from home? Do you really need to watch your employees like hawks to make sure they work? Can’t you tell by results if they are working? I submit that if you can’t trust your employees to do their work if you let them work form home, you can’t trust them to do their work at work either and perhaps you should find new employees!

I guess we need to lead from the bottom, if the entire base of the pyramid moves in the correct direction, the rest has to follow.

The Times, they are a Changing

For my presentations at VirtaThon least! Here is the latest schedule of my presentations:

Best of Both Worlds: Using SSd and Disks via ASM
Presenter(s):
Michael Ault
Time slot: 17 July 10:00 - 10:55

Room:
Virtual Room # 100
Infrastructure Technologies - Oracle
Target Audience:
All




VirtaThon GoldStar Session!!! 4-Hour Deep-Dive: Got Performance? Statspack and AWR Analysis
Presenter(s):
Michael Ault
Time slot: 18 July 11:00 - 15:25

Room:
Virtual Room # 100
Database - Oracle
Target Audience:
All




The Need for Speed: Using SSDs with Oracle
Presenter(s):
Michael Ault
Time slot: 19 July 13:30 - 14:25

Room:
Virtual Room # 200
Infrastructure Technologies - Oracle
Target Audience:
All





VirtaThon Keynote: Tuning on the Cheap: Oracle tuning Utilities Everyone Has
Presenter(s):
Michael Ault
Time slot: 20 July 12:30 - 13:25

Room:
Virtual Room # 100
Database - Oracle
Target Audience:
All

Looking Forward

Well, looks like I am booked out for the month of July! Starting Saturday (16 July, 2011) I am doing a 4 hour presentation on AWR analysis for VirtaThon http://www.brainsurface.com/virtathon here is the rest of my VirtaThon schedule (all times EDT):

Sunday 17 July, 2011 10:00-11:00 am "Best of Both Worlds: Using SSd and Disks With ASM"
Tuesday 19 July, 2011 4:30-5:30 pm "The Future is Now: A Fully SSD Database System"
Wednesday 20 July, 2011 12:30-1:30 pm " Tuning on the Cheap: Oracle Tuning Utilities Everyone Has"

On Thursday I present at the Twin Cities Oracle User Group in Minnesota: "Data Warehouse Development Techniques and New PL/SQL Features"

On Tuesday July 26 ,2011 12:00-01:00pm "Oracle Exadata X2-8: A Critical Review" (webcast)Check out http://www.ramsan.com/company/events for more information and a link.

On Wednesday, July 27, 2011 I will be doing a joint presentation with Tariq Farooq on "360 Degrees: Everything to know about Virtualization for Oracle DBAs"

Should be a quick end of the month! Hope you can all make it to at least one of the virtual or live events!

To see any of my past Webinars look at http://www.ramsan.com/company/events and http://www.ramsan.com/resources/videos

A Bitter Nostalgia

I remember in grade school and middle school when I was growing up they would roll TVs into the classroom and we would all watch in wonder as the Mercury and then the Saturn V launch vehicles would, with a rumble and roar lift our astronauts into space. With this technology we put the first men on the moon, sent probes into deep space and around the planets in this solar system. They also lifted up young hearts and minds to believe that maybe they could someday walk on the surface of a different world and stare back at the star that gave them birth with wonder.

Today, the last space shuttle mission launched. After the shuttle returns it will be mothballed and with that action, the USA will no longer has the capability to put people into space. Now, with the demise of the space shuttle we will become a beggar nation when it comes to space, just like the third world countries we will have to go begging with hat (and money) in hand to the Russians and Chinese to use their rockets, vehicles and launch facilities. John F. Kennedy will be spinning in his grave.

We have record unemployment, the lowest new job creation numbers and the dollar is at one of its lowest points in relation to other currencies that it has ever been. I don’t know what the future holds but I am worried. I worry that my grandchildren will be burdened with an immense load of debt that they can’t possibly repay. I worry that they will have no jobs and become dependent on the government for their sustenance, not because of lack of desire or ability, but because of the incompetence of our leaders. For the first time the future looks worse than the past.

Perhaps my memories just aren’t correct. I recall affordable housing, cars, gas and food. I recall a time when one income would feed a family and mothers weren’t expected to work outside of the home. I recall when a majority of time children were respectful to their elders, God was allowed in courts, in schools in lives. Many intellectuals wanted change, wanted God out of society, because after all, God was an outmoded concept that modern, thinking, scientific man didn’t need. How is a Godless world working out for you folks? Personally I find it a disturbing and quite a bit a scary place and getting worse as we get further from the basic rules, handed down by God that this country was based on.

Disgusting, violent video games that desensitize our youth to violence and death are authorized by the highest court in the land. Movies you are afraid to attend because you never know what they will contain fill the movie theaters. The basic rights that the country was founded on are perverted because our highest courts are filled with people with no common sense, no moral courage and nothing but empty academic theories instead of wisdom.

I am sorry if I am a bit morose today. The 10 year old with dreams of being an astronaut just died in my soul.

JavOraThon Conference

Imagine a conference with top Oracle, Java and related topic presenters that you can attend in your PJs, from home, from your office, or even on your commute! A conference where you don't need to put up with TSA touching things that even your spouse won't touch and possibly having your image as the next TSA pinup of the week. Not to mention saving all those travel expense dollars in this tight economy! Well, imagine no more! The JavOraThon conference is a virtual conference with industry recognized speakers and as long as you have internet connectivity you can attend!

Here is the information for being a presenter if you are interested, I know I will be giving several presentations (yet to be announced) and know of several other experts who will be doing the same. The Conference dates are July 16-21 2011, coming to a computer near you!

Call-for-Papers URLS:

Speaker Registration: http://www.brainsurface.com/javorathon/node.add.speakers/
Propose a Session: http://www.brainsurface.com/javorathon/node.add.session/
View your proposed Sessions: http://www.brainsurface.com/javorathon/program.mysessions.proposed/

Hope to (virtually) see all you fellow Oracle users and Gurus there!