1. First I got messages (during installation of the software) that files could not be found in the target oracle folder
2. Second, after finding the solution for the first annoyance, I spent two hours finding out why everything installed correctly, I could create a database, but could not start a Listener. I am an Administrator on my laptop. The errors I got at starting a listener were ORA-12560 and ORA-12541.

Regarding 1:

It appeared to be very simple: when you download Oracle 11 R2, there are two zip files. Make sure to extract both zip files into the same folder and not into two separate folders (like a Disk1 and Disk2 folder). After extracting both zip files into the same folder, the installation went smooth without errors.

Regarding 2:

After a fresh installation and creation of a database, I am used to open a Command prompt, start LSNRCTL, and execute a START to start the default listener. If a Windows Service does not exist yet, LSNRCTL throws some errors around that but as part of this it creates the Windows Service. But not this time. I kept on getting the result:

C:\>lsnrctl start
LSNRCTL for 64-bit Windows: Version 11.2.0.1.0 - Production on 18-MRT-2011 21:48:56
Copyright (c) 1991, 2010, Oracle.  All rights reserved.
Starten van tnslsnr: Even geduld a.u.b.
TNS-12560: TNS: fout bij protocoladapter.
 TNS-00530: Fout in protocoladapter.

and a LSNRCTL status gave me:

C:\>lsnrctl status
LSNRCTL for 64-bit Windows: Version 11.2.0.1.0 - Production on 18-MRT-2011 21:52:02
Copyright (c) 1991, 2010, Oracle.  All rights reserved.
Verbinding wordt gemaakt met (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=1.1.1.1)(PORT=1521))).
TNS-12541: TNS: geen listener.
 TNS-12560: TNS: fout bij protocoladapter.
  TNS-00511: Geen listener
   64-bit Windows Error: 61: Unknown error
C:\>

But no listener got created. The Oracle support site, nor any forum or blog pointed me in the correct direction. The only thought I had was: I know Windows requires the Listener to be present as a Service, so as long as that is not the case, it will not start. So here is what I tried: I created the service myself using the Windows “sc” command. Now a first attempt gave an Access Denied, but after storing the command in a Batch file and executing the batch file to “Run as Administrator” (right-click the command file and choose that option), the service got created!

The SC command used was: sc create OracleOraDb11g_home1TNSListener binPath= “C:\app\toine\product\11.2.0\dbhome_1\BIN\TNSLSNR.exe” (take care of the space after binPath= !)

And after starting it using the Service Control Panel, LSNRCTL gave me a good status:

C:\>lsnrctl status
LSNRCTL for 64-bit Windows: Version 11.2.0.1.0 - Production on 18-MRT-2011 21:57:57
Copyright (c) 1991, 2010, Oracle.  All rights reserved.
Verbinding wordt gemaakt met (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=1.1.1.1)(PORT=1521))).
STATUS van LISTENER
------------------------
Alias                            LISTENER
Versie                           TNSLSNR for 64-bit Windows: Version 11.2.0.1.0 - Production
Startdatum                       18-MRT-2011 21:57:48
Ingeschakeld                     0 dagen 0 uren 0 minuten 9 seconden
Traceerniveau                    support
Beveiliging                      ON: Local OS Authentication
SNMP                             OFF
Parameterbestand van listener   C:\app\toine\product\11.2.0\dbhome_1\network\admin\listener.ora.
Log van listener            c:\app\toine\product\11.2.0\dbhome_1\log\diag\tnslsnr\mypc\listener\alert\log.xml
Traceerbestand van listener c:\app\toine\product\11.2.0\dbhome_1\log\diag\tnslsnr\mypc\listener\trace\ora_6728_7012.trc
Overzicht beluisteren eindpunten...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=1.1.1.1)(PORT=1521)))
Overzicht van services...
Service "ONTW112. MYPC.MYDOMAIN.NL" heeft 1 instance(s).
  Instance "ONTW112", toestand UNKNOWN, heeft 1 handler(s) voor deze service...
Uitvoeren van opdracht is geslaagd.
 

and I could login using Oracle Net
The problem appeared to be Administrative privileges. Which was not obvious to me at first. I wondered why the installer could create all necessary services, whereas LSNRCTL was not able to create the Listener service. I stopped and deleted the service again (using “sc delete”) and started LSNRCTL.EXE as Administrator (go to the Oracle\bin folder and right click LSNRCTL.EXE. That lead to the same result: a START within that LSNRCTL session created the Windows Service as I am used to from other Oracle installations on Windows I did many times.

After all it is probably a logical error on Windows 7.  What bugs me is that I did not read it somewhere and that the errors did not point me to the problem. They pointed me towards some network problem. I almost wonder whether I missed some guideline. Still, it may help others, who have the same trouble.

These last weeks I am trying to get the VPD option (Virtual Private Database) enabled for my application. This time as a developer with no DBA or SYS privileges. I am used to being a DBA for the databases I work on and am used to being able to get the fullest out of the Oracle database. For one, because I think Oracle provides us with a great deal of built-in features that do things more efficiently than we can do ourselves (more efficient, built-in kernel code) and also especially because it seems useless and time-consuming to write code that is already supplied by built-in packages. Now these provided features may not all be as intuitive as we want to or lack some important features, but that is where we can spend some time extending them or making them more user-friendly by creating “wrapper” packages: packages you write that hide the complexity or unfriendliness of a built-in package or limit the features you can use. Because there are quite some packages out there that contain functions or procedures or even parameters that you do not want to give to your developers or end users.

An example of such a package that is too powerful to grant to your end-users as a whole is DBMS_RLS. This package enables you to put Row Level Security (and even Column Level Security) on your schema objects. When you take a look at most tutorials available about the VPD option, all of them simply state that you should grant EXECUTE on DBMS_RLS to the schema owner so that it can add, drop, enable, disable policies on its objects to limit the data an end-user of the object can see or manipulate.

First, I love the feature. It is a way of declaratively filtering data based on business criteria dependent on the logged in user/location/terminal/anything that can be set or queried as a context variable. It is so much cleaner than having the same kind of predicates sitting around in all parts of the application, almost knowing for sure that:

• something will have been missed somewhere and people see data they should not
• or will change in the future and turns in to a maintenance nightmare
• or can be circumvented

Why is DBMS_RLS too powerful in “my” opinion? At least I made it my opinion also after consulting with a DBA at the company where I need the VPD feature, who knows what it is like to manage several hundreds of databases and knows the challenge of maintaining some level of standardization without compromising security but on the other hand providing application developers with a toolset they can solve their problems with. I did not think of it at first when asking for EXECUTE privileges on DBMS_RLS for my application, but he mentioned that DBMS_RLS is not preventing me from hurting someone else’s policies:

If a database contains schema owners SCOTT and TOINE and both need VPD for their schema objects…the tutorials state that both should get execute privileges on DBMS_RLS. But that does not prevent SCOTT from dropping policies of user TOINE. Me as TOINE would hate the idea that someone else can touch my ‘rules’. As it turns out there is more about securing DBMS_RLS that goes beyond this simple article (SQL Injection and XMLDB or some other built-in packages can dig a hole into your beautiful VPD security, see for such an example http://www.oracle-dox.net/Wiley-The.Oracle.Hacker.s.Hand/final/BBL0043.html or Pete Finnigan’s presentation http://www.petefinnigan.com/Oracle_Security_VPD_2009_6slides.pdf). Also there is a lot of useful information to be found for a hacker in a number of data dictionary views.

So how would a DBA that needs to standardize hundreds of databases enable a developer to use DBMS_RLS? Several options, one better than the other:

• He or she could execute any policy manipulation (enable/disable/add/drop) on behalf of the developer. This requires tickets to be raised and will frustrate the developer who has to wait for the DBA to find time to execute that task out of his growing queue of such tasks
• I like the wrapper package idea: create a wrapper package on top of DBMS_RLS, it can be as simple as providing all subprograms, but then WITHOUT the object_schema parameter. Instead of having the developer provide the object_schema, YOU take control and allow that developer to only manipulate policies on objects of its own. Instead of a grant EXECUTE on DBMS_RLS, a grant EXECUTE on the wrapper package is provided.

So DBMS_RLS looks like (In Oracle 10R2):

CREATE OR REPLACE PACKAGE SYS.dbms_rls AS
  STATIC                     CONSTANT   BINARY_INTEGER := 1;
  SHARED_STATIC              CONSTANT   BINARY_INTEGER := 2;
  CONTEXT_SENSITIVE          CONSTANT   BINARY_INTEGER := 3;
  SHARED_CONTEXT_SENSITIVE   CONSTANT   BINARY_INTEGER := 4;
  DYNAMIC                    CONSTANT   BINARY_INTEGER := 5;
  DV_INTERNAL                CONSTANT   BINARY_INTEGER := 10;
  ALL_ROWS                   CONSTANT   BINARY_INTEGER := 1;

  PROCEDURE add_policy(object_schema   IN VARCHAR2 := NULL,
                       object_name     IN VARCHAR2,
                       policy_name     IN VARCHAR2,
                       function_schema IN VARCHAR2 := NULL,
                       policy_function IN VARCHAR2,
                       statement_types IN VARCHAR2 := NULL,
                       update_check    IN BOOLEAN  := FALSE,
                       enable          IN BOOLEAN  := TRUE,
                       static_policy   IN BOOLEAN  := FALSE,
                       policy_type     IN BINARY_INTEGER := NULL,
                       long_predicate BOOLEAN  := FALSE,
                       sec_relevant_cols IN VARCHAR2  := NULL,
                       sec_relevant_cols_opt IN BINARY_INTEGER := NULL); 

  PROCEDURE drop_policy(object_schema IN VARCHAR2 := NULL,
                        object_name   IN VARCHAR2,
                        policy_name   IN VARCHAR2);

[intentionally left out the rest of subprograms)
END dbms_rls;
/

A wrapper package, which can be created in any DBA managed schema (it might be a dedicated VPD admin schema) could look like:

CREATE OR REPLACE PACKAGE VPDADMIN.secure_dbms_rls AS
  STATIC                     CONSTANT   BINARY_INTEGER := 1;
  SHARED_STATIC              CONSTANT   BINARY_INTEGER := 2;
  CONTEXT_SENSITIVE          CONSTANT   BINARY_INTEGER := 3;
  SHARED_CONTEXT_SENSITIVE   CONSTANT   BINARY_INTEGER := 4;
  DYNAMIC                    CONSTANT   BINARY_INTEGER := 5;
  DV_INTERNAL                CONSTANT   BINARY_INTEGER := 10; 
  ALL_ROWS                   CONSTANT   BINARY_INTEGER := 1;

  PROCEDURE add_policy(object_name   IN VARCHAR2,
                       policy_name     IN VARCHAR2,
                       function_schema IN VARCHAR2 := NULL,
                       policy_function IN VARCHAR2,
                       statement_types IN VARCHAR2 := NULL,
                       update_check    IN BOOLEAN  := FALSE,
                       enable          IN BOOLEAN  := TRUE,
                       static_policy   IN BOOLEAN  := FALSE,
                       policy_type     IN BINARY_INTEGER := NULL,
                       long_predicate BOOLEAN  := FALSE,
                       sec_relevant_cols IN VARCHAR2  := NULL,
                       sec_relevant_cols_opt IN BINARY_INTEGER := NULL);

  PROCEDURE drop_policy(object_name   IN VARCHAR2,
                        policy_name   IN VARCHAR2);

  [intentionally left out the rest of subprograms)
END dbms_rls;
/ 

In the body of this package the subprograms can simply call DBMS_RLS. Add_policy and drop_policy like this:

CREATE OR REPLACE PACKAGE BODY VPDADMIN.secure_dbms_rls AS
     PROCEDURE add_policy(object_name   IN VARCHAR2,
                          policy_name     IN VARCHAR2,
                          function_schema IN VARCHAR2 := NULL,
                          policy_function IN VARCHAR2,
                          statement_types IN VARCHAR2 := NULL,
                          update_check    IN BOOLEAN  := FALSE,
                          enable          IN BOOLEAN  := TRUE,
                          static_policy   IN BOOLEAN  := FALSE,
                          policy_type     IN BINARY_INTEGER := NULL,
                          long_predicate BOOLEAN  := FALSE,
                          sec_relevant_cols IN VARCHAR2  := NULL,
                          sec_relevant_cols_opt IN BINARY_INTEGER := NULL)
  IS
   BEGIN
     sys.dbms_rls.add_policy(USER,
                             object_name,
                             policy_name,
                             function_schema,
                             policy_function,
                             statement_types,
                             update_check,
                             enable,
                             static_policy,
                             policy_type,
                             long_predicate,
                             sec_relevant_cols,
                             sec_relevant_cols_opt);
   END; 
  [intentionally left out the rest of subprograms)
END dbms_rls;
/ 

The only disadvantage I can think of is portability of the code: if the target database is not following the same principle, you would have to alter your package name. But if you call your add_policy program using named parameters instead of positional…then the package name may be the only thing you would have to change…since DBMS_RLS takes the current user as the default if none is provided:

BEGIN
   VPDADMIN.secure_dbms_rls.add_policy
                             (object_name          => 'mytable',
                              policy_name          => 'mypolicy',
                              function_schema      => 'myschema',
                              policy_function      => 'mypackage.myfunction',
                              statement_types      => 'select, update, delete'
                             );
END;

I am currently in the middle of discussing this wrapper option…I hope to be able to use VPD as wanted soon. Anyone feeling differently on grants on DBMS_RLS?

I am probably in worse shape this year because I spend so much time doing things like the PL/SQL Challenge  and doing a presentation for the Dutch User Group on Performance Tuning and Troubleshooting. Ofcourse the first one is mostly fun and it does not take so much time: the statistics on the site even show clearly how much time I needed to answer all those questions so far (it started the 8th of April). It is a superb idea worked out by Finn Elebaek Nielsen and  Steven Feuerstein, whom I got to know rather well in the last decade since I used to work intensively with his PL/Generator and invited him a few times for the Dutch User Group. Great guy and very enthusiastic about spreading the word of PL/SQL. It encourages me to play at my best in this Challenge and so far I am doing much better than expected (currently in second place and have been in first place for several days). And we are reaching Quarter’s end, meaning the top players will do a play-off to decide who will be 1st, 2nd and 3rd. And these will win some serious cash prices. It is serious stuff and I hope to be able to be in that top 10 to give it a shot.

The second thing, the presentation for the Dutch User Group, was at May 26 where I was able to talk to almost 100 Oracle minded people. An unexpected high number of people subscribed for my presentation. Not often a waiting list is needed for this monthly event, where the User Group invites a speaker on a certain topic. This time it was about Database Performance Tuning and Troubleshooting and they found me on LinkedIn. I can only say: it seems to work and to pay off to have a profile on such a social network. Even though I am not fond of exposing on all kinds of social networks, I find LinkedIn an almost “must have” for professional people. But that is very personal and I know people who stay away from it as far as possible. But for me it works. So I was invited to speak about Database Tuning. Since I do not do presentations that often, I did not have a ready-made set of slides yet and all you people who present once and a while know how much energy this takes. I must say it was in my head almost 24*7. Oh yes, I was thinking of ways to present this with this joke and to present that with that joke and I was having a lot of fun before I fell asleep each night. I know, it sounds sad, especially since I seemed to forget about all those jokes as soon as I thought of a new one. Is that the age-thing…? As said earlier, I have reached my 40′s. It was so bad that I almost wanted to start with the sentence: “If I remembered all the jokes I had in mind for this evening, you people would have had a splendid and joyful evening!” Now, reality was that it became much more serious than that. I kept the jokes to an acceptable minimum . Still I think I kept almost all attendees awake for almost 2,5 hours and got some nice feedback afterwards. To me it was a success  for a first “big” presentation and I had to battle against a beamer that did not want to recognize my Dell laptop. Which was a bit of a shame, since the prepared demo’s could only run on that laptop and not on the one I borrowed from a former colleague who was in the room.

I talked about several things that are important in my work as a Performance Troubleshooter. I included some highlevel thoughts for which I am mostly inspired by Cary Millsap and his great book “Oracle Performance Tuning”, but also from his Masterclass. By that I mean things like doing diagnosis in the proper scope and not being blinded by averages and totals that make real performance problems invisable. But also that I feel a modern DBA and most certainly one that is involved in solving performance problems needs to communicate. Not only talk to other technical people (like developers), but also communicate with people in the business. They determine your priorities, since your priorities need to allign with the business priorities. Example: 1,5 years ago someone told me that we needed to focus on performance of some part of the system, since some dashboard showed a decreasing performance line. So we did some extra analysis and did not really find a performance problem. Users were also not complaining. Then I said I wanted to be able to talk to key users who could tell me their worst problems. It was a great thing: their impression of that “bad” part of the system was in fact the same as ours after that analysis: that part had no real performance problems according to all those users but most of them asked if we could have a similar conversation soon about a totally other part of the system which had some real bad problems. After all it turned out that “dashboard” that showed the so-called “performance problem” was not accurate: it was running on a laptop with performance problems of its own. Our communication with the users revealed no problem, but lead us to a part that we did need to look at and focus on. One for which we did not have an accurate performance monitor for. Without those users showing us their real problems, we would have focussed on the wrong part of the system. It can be that simple, sometimes.

For the rest I presented some more technical things:

• I told about some mysteries and fallacies of the optimizer (in which part I referred to Wolfgang Breitling’s “Tuning by cardinality”) and showed some fixes you can do when the optimizer cannot get it right
• I talked about indexing strategy in an OLTP and OLAP system and showed some thoughts of Tapio Lahdenmäki (3-star indexes and the difference between single block reads and multi block reads)
• And finally I had some slides on one of my personal favorites: clustering of data

One of the backgrounds for the presentation was that I wanted to show things you can do when a system has problems, without immediately buying faster or more hardware. There is often so much you can achieve with simple solutions. Only well tuned systems are hard to tune, but in my experience and opinion, a great deal of systems out there are not even near being well-tuned. And there you can save a lot of money by having someone with Oracle knowledge and the ability to understand what the business wants and how the application works, take a look at your application. It becomes hard when the application itself is the biggest problem, but even a bad application can be helped with proper indexes, data clustering or optimizer related improvements, without touching the application itself.

At one customer site we are migrating to Oracle 11 and we have an Oracle 9i database with a mod_plsql application to serve a dynamic webpage. Now if you are used to the HTTP server in Oracle8i or 9i installed by default together with the database software, you are wondering where it is after you install Oracle 10 and above. Now I knew that already so I knew I had to look at the Embedded PL/SQL Gateway or download and install the standalone Oracle HTTP server 10g from OTN. I started with the Embedded PL/SQL Gateway. It was not very hard to get it to work, but soon I realized it was not a substitute for the Apache web server in our case because we do not only have PL/SQL pages, but also static pages and the PL/SQL dynamic pages are just a part of the entire site.

So I downloaded the 10g standalone HTTP server from OTN. Great. I installed it and got it to work very fast (I knew where to look for configuration files and changes to make from my experience in 9i). I only wondered where my Windows Service was (Oh yes, it is a Windows environment), but I did not really take notice of it because I just thought the way the HTTP server was managed in 10g was different from 9i. This in the sense that you must use Oracle Process manager (OPMNctl) to start and stop the web server. And that worked indeed.

So everything was working, I had successfully upgraded the (test)web server to an Oracle 11.1.0.7 database and a 10.1 HTTP server and logged off from the server, but not before I sent an e-mail to the testers that the test site had moved and was accessible using the URL I gave them. I logged of and the next day I logged in to the network, tried the URL and it did not work. Huh? It worked yesterday. So I logged in to the server and used opmnctl to Start the HTTP server again. Now it worked again. I did not have time to check what the reason might be so I notified the testers that the webserver must have crashed or something and we would look at it tomorrow. I also showed that if it might happen again, she could start ‘opmctl startall’ to at least be able to test the site (she is an administrator herself).

The next day the tester informed me that it did not work again and that even the opmnctl startall did not work. So the day after I started looking for reasons why the server was not started. I saw some SSL error, but that made no difference. And suddenly it appeared to me (stupid, I should have known): when I log off from the server, the HTTP processes were stopped because I started them under my control. The missing Windows Service seemed to be a key point here: the httpserver was not started as a service and I could not start it as a service, because the service was not there. And I wondered: how do I create that service? I could not find anything on the support site and at first nothing with a Google search. Mostly I stop looking after 2 or 3 pages of search results but this time I looked a bit further and there it was…my saving web page: http://www.dbaportal.eu/?q=node/131. I thought it would probably benefit more people so here is the clue:

• The standalone HTTP server on OTN does not create a Windows Service and this can be considered a bug
• You can create the service yourself with the Windows SC command:

sc create OracleHTTPProcessManager binPath= "D:\ORACLE\ORAHTTP\opmn\bin\opmn.exe -S" DisplayName= OracleHTTPProcessManager start= auto

You can choose a name for the Service and DisplayName yourself and the binPath must match your Oracle HTTP server installation’s opmn\bin folder.

And this hint was not in the article itself, but in a comment by the same author. Thank you! It is important that people show their problems and especially the answer. It “saved” my day…

1. An issue with the buffer parameter in Import
2. An issue with Resumable=Y and FROMUSER/TOUSER where the FROMUSER and TOUSER have different default tablespace and quotas

Buffer

• I did a DIRECT export yesterday of a table that contains a LONG column
• I did an import (which is conventional) of that table with BUFFER=<some value>
• I got embarrassed because my laptop started beeping around and my screen started screaming weird characters

I went through that embarrassment a few times, because I tried twice with that same import file (only the foolish try the same thing twice and expect a different result, yes I am guilty ) and once with another import file of which I knew it contained that same table and I imported successfully in an 11g database earlier. (I almost thought I had a corrupt importfile and lost the contents of that table). But when that same import also initiated the terrible beeping (I have to turn of system beeps) and everybody in the large room was staring at me, I knew it had to be something with my import parameters. It turned out to be the buffer parameter. Even though that still surprises me. Oracle documentation says about buffer in IMPORT:

Default: operating system-dependent

The integer specified for BUFFER is the size, in bytes, of the buffer through which data rows are transferred.BUFFER determines the number of rows in the array inserted by Import.

The following formula gives an approximation of the buffer size that inserts a given array of rows: buffer_size = rows_in_array * maximum_row_size

For tables containing LOBs or LONG, BFILE, REF, ROWID, UROWID, or DATE columns, rows are inserted individually. The size of the buffer must be large enough to contain the entire row, except for LOB and LONG columns. If the buffer cannot hold the longest row in a table, Import attempts to allocate a larger buffer.

So I would have expected that Oracle would adjust the size of the buffer automatically when it turns out not to be big enough…

Resumable

I did an import for an entire schema into another schema (VIND into VINS). User VIND’s tables are in tablespace VIN_D. User VINS’s tables need to be created in tablespace VINS_D. USER VINS has no quota on the tablespaces of user VIND. So the meaning is that IMPORT will create the tables (which are defined to be in tablespace VIN_D) into tablespace VINS_D. I have seen that working all the time in the past. Until this day when it did not work as expected. My import hung…it was due to the RESUMABLE=Y parameter I added to be able to handle out of space issues if they would occur.  But apparently RESUMABLE=Y and the need to import into a different tablespace without the hassle of pre-creating the tables first does not seem to work. It was not after I removed my RESUMABLE=Y that Import went ahead and created the tables in the default tablespace of user VINS.

In this post I want to share good experience I had in the use of DBMS_REDEFINITION to speed up the Siebel OLTP application I am responsible for (at least, for the performance part of it). It is about clustering of data and rebuilding tables as ONLINE as can be and how I used a trick with a stored outline to overcome a performance problem during rebuild of one table.

In every application there are certain tables that are queried heavily. Some applications have tables that are even used in almost every query. A lot of times the access path into these tables is concentrated around one or two columns or at least a few “important” columns. In Siebel for example there is a table called S_POSTN_CON, which stores links between Contacts (Professionals) and Positions. A Sales Representative for a certain product can be a “position” and a District manager is another “position”. The Contacts they are allowed to ‘see’ (visibility) is defined by assignment rules and these assignment rules are the driving force for the contents of the S_POSTN_CON table and its counterpart for Organizations (Accounts), S_ACCNT_POSTN – Don’t you love such “standardization”, why not S_POSTN_ACCNT?). An extreme lot of queries join S_POSTN_CON or S_ACCNT_POSTN with other tables, to control the Contacts and Accounts the logged in user is allowed to see. These tables are always accessed using the POSTN_ID, which is the Siebel ID for the position the logged in user has.

A lot has been written already on the topic of data clustering in Oracle and why it is beneficial from a performance perspective (for example Jonathan Lewis: http://jonathanlewis.wordpress.com/2007/05/17/data-cluster/). It touches things like the clustering factor for Indexes and the db_file_sequential_read statistic. To be short: it really helps if the data you are after is clustered in a few Oracle blocks instead of spread over many blocks, especially when an index is used to get the data from the table. That is because the most expensive part of using an index is not the index access itself, but visiting the table to get the data blocks containing the rows identified by the index. Provided that all the data needed cannot be retrieved from the index alone (another good thing to look at). The more different blocks that need to be retrieved (using single block reads), the longer the execution time.

So with that in mind, we started looking at the ‘key’ tables (the heavily used tables) in the Siebel OLTP application and looked at their most common access paths. As said for S_POSTN_CON and S_ACCNT_POST it was definitely the POSTN_ID column. And for example the S_CONTACT_XM table was accessed mostly via the PAR_ROW_ID (the link between the S_CONTACT_XM records and the S_CONTACT table, which stores the Contacts/Professionals) and the TYPE column.

Now what we did was a rebuild of these tables and we pre-sorted them on the access path columns. In that way, Oracle stores the table data sorted (well initially, it does not maintain it afterwards when new records enter the table) which means related data is stored tightly in as few Oracle blocks as possible. Which means: when S_POSTN_CON is queried for all the records for a given POSTN_ID, Oracle uses the index on S_POSTN_CON to get the rowid’s for the records with that POSTN_ID and only needs to get one or at most 2 or 3 Table blocks for all the records, instead of 20 or more blocks it needed to get before the rebuild.

The results were significant. A lot of queries became 50% or even a lot faster after the rebuild of in total 7 key tables. It sometimes lead to different execution plans, because the clustering factor of certain index became much more interesting. It was mainly after this action that the entire system constantly stayed at a performance level of more than 99% screens within 3 seconds. Before that we hardly reached that 99%! And even though the ordering is not maintained, a lot of data is stable but nevertheless queried over and over again: these parts of the table more or less stay ‘ordered’. For some tables we will be repeating the ordering process periodically. There are other ways to maintain clustering within  table like partitioning, IOT’s, sorted hash clusters etc, but all of these do not apply to use in a Siebel OLTP application. For partitioning you need a lot of money for licenses when you have a system with 16 CPU’s as well as you have to involve Siebel support to enable partitioning in OLTP.

But how do you rebuild a table and pre-sort its data in an always up and running application? This is where DBMS_REDEFINITION shows up as very useful. This package, available since 9i but much more powerful in 10g, makes it possible to completely redefine tables online with a minor lock moment at the end. I have become to love this built-in package.

With DBMS_REDEFINITION, you can:

• rebuild a table because you want to reorganize (and do not want to use ALTER TABLE MOVE, which is not entirely online, especially regarding indexes that are unusable after the MOVE and need a rebuild)
• alter the structure of a table: add columns, drop columns, change data types
• alter its physical structure: convert a heap organized table into an IOT, convert to a partitioned table, etc
• Change the order of the records in the table

Here is an example of the code I executed to rebuild the Siebel S_ADDR_PER table

-- Check if the table can be rebuilt at all:
BEGIN
  DBMS_REDEFINITION.CAN_REDEF_TABLE('SIEBEL', 'S_ADDR_PER',
                     DBMS_REDEFINITION.CONS_USE_ROWID);
END;
/

-- Create an intermediate table with the same structure or
-- (if required, with structure changes)

CREATE TABLE "SIEBEL"."BCK_S_S_ADDR_PER"
( "ROW_ID" VARCHAR2(15 CHAR) NOT NULL ENABLE
,"CREATED" DATE DEFAULT sysdate NOT NULL ENABLE
,"CREATED_BY" VARCHAR2(15 CHAR) NOT NULL ENABLE
,"LAST_UPD" DATE DEFAULT sysdate NOT NULL ENABLE
,"LAST_UPD_BY" VARCHAR2(15 CHAR) NOT NULL ENABLE
,"DCKING_NUM" NUMBER(22,7) DEFAULT 0
,"MODIFICATION_NUM" NUMBER(10,0) DEFAULT 0 NOT NULL ENABLE
,"CONFLICT_ID" VARCHAR2(15 CHAR) DEFAULT '0' NOT NULL ENABLE
,"ADDR_NAME" VARCHAR2(100 CHAR) DEFAULT 'x' NOT NULL ENABLE,
[...]
all remaining columns exactly matching the original S_ADDR_PER table
[...]
)
PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255 NOCOMPRESS LOGGING
TABLESPACE "DATA_TS";

-- The actual work: start the online redefinition

DECLARE
  l_num_errors NUMBER;
BEGIN
  -- This will build a materialized view log on the table to be rebuild,
  -- to capture changes during the rebuild process
  -- it will also copy all table data initially, ordered by "orderby_cols"
  SYS.DBMS_REDEFINITION.START_REDEF_TABLE('SIEBEL', 'S_ADDR_PER', 'BCK_S_S_ADDR_PER'
  , col_mapping => null
  , options_flag => SYS.DBMS_REDEFINITION.CONS_USE_ROWID
  , orderby_cols => 'x_bu_id, country, x_brick_id, zipcode, city');
  -- Copy all triggers/indexes/privileges/optionally constraints and statistics
  SYS.DBMS_REDEFINITION.COPY_TABLE_DEPENDENTS('SIEBEL', 'S_ADDR_PER'
  , 'BCK_S_S_ADDR_PER'
  , copy_triggers=>TRUE
  , copy_constraints=>FALSE
  , copy_privileges=>TRUE
  , copy_indexes=>dbms_redefinition.cons_orig_params
  , copy_statistics=>FALSE
  , num_errors=>l_num_errors);
  SYS.DBMS_REDEFINITION.SYNC_INTERIM_TABLE('SIEBEL', 'S_ADDR_PER'
                                                                  , 'BCK_S_S_ADDR_PER');
EXCEPTION
  WHEN OTHERS
  THEN
    DBMS_OUTPUT.put_line ( '# of errors while copying dependent objects: '
                                      ||l_num_errors);
  RAISE;
END;
/

BEGIN
  DBMS_STATS.gather_table_stats (ownname => 'SIEBEL',tabname => 'BCK_S_S_ADDR_PER',
  estimate_percent => NULL,degree => 5,method_opt => 'FOR ALL COLUMNS SIZE 1',
  CASCADE => TRUE );
END;
/

-- Finish the redefinition: switch the two tables, drop the materialized view log
BEGIN
  SYS.DBMS_REDEFINITION.FINISH_REDEF_TABLE('SIEBEL', 'S_ADDR_PER', 'BCK_S_S_ADDR_PER');
END;
/

-- For some reason, after redefinition, the indexes were set to NOLOGGING

ALTER INDEX SIEBEL.S_ADDR_PER_EI LOGGING;
[...]
--set all remaining indexes to LOGGING again
[...]
ALTER INDEX SIEBEL.S_ADDR_PER_F1_X LOGGING;

spool off

The end result is an exact copy of S_ADDR_PER, but with ‘clustered’ data around the columns x_bu_id, country, x_brick_id, zipcode, city. The original table has now been renamed to the intermediate table’s name, as well as its indexes, triggers and other dependent objects.

Internal SQL

During the execution of DBMS_REDEFINITION.START_REDEF_TABLE, an INSERT statement is executed that inserts the original table’s data into the intermediate table (with an ORDER BY in our case). In one case (the rebuild of table S_CONTACT_XM) it happened that the execution plan Oracle used to execute this INSERT AS SELECT statement used an existing index as its driving index because this index was defined on the columns to order on. Now I could not remove that index or set it to UNUSABLE, because that would be violating the “100% Online” goal of the exercise. But the difference between a FULL table scan and the use of the index as driving index was 1 versus 7 hours. So I wanted to force a FULL table scan. Now that is worth tuning, but how could I do that? I tried optimizer parameters at the session level, but could not find one that eliminated the index. Now regardless of it being my lack of knowledge or there really is no magic optimizer parameter to set here, I used a trick with an outline to force the FULL table scan.

A few challenges there:

1. The internal SQL adds a column to the intermediate table, called m_row$$ of type VARCHAR2(15) which is needed to store the Oracle ROWID value of the original table. It uses it during the SYNC_INTERIM_TABLE calls of DBMS_REDEFINITION to quickly find the records to update or delete. The internal INSERT statement inserts the ROWID into the intermediate table’s m_row$$ column. And that is where the creation of an outline on that internal statement would fail: it cannot parse that statement because the m_row$$ column does not exist yet!

2. It is a challenge to create outlines and adding your own hints to it. I use a technique learned from colleagues to do that easily, even though it does not seem to be documented and I am in doubt if it is supported. It works perfectly however.

So here are the steps:

I temporarily add the m_row$$ column to the intermediate table myself:

ALTER TABLE "SIEBEL".bck_s_contact_xm ADD m_row$$ VARCHAR2(15);

And then I can create an outline:

DECLARE
  bq VARCHAR2 (32767); --sql statement bad query
  gq VARCHAR2 (32767); --sql statement bad query
  outln_name_bq VARCHAR2 (30); -- Final outline name (for corrected bad query)
  outln_name_gq VARCHAR2 (30); -- Temporary outline name
BEGIN
  outln_name_bq := 'REDEF_CONTACT_XM_ORDERED'; -- Final name
  outln_name_gq := 'REDEF_CONTACT_XM_ORDERED_GD'; -- Temporary name
  bq :=
    'INSERT /*+ BYPASS_RECURSIVE_CHECK APPEND */
    INTO "SIEBEL"."BCK_S_CONTACT_XM"(M_ROW$$,"ROW_ID","CREATED",
    "CREATED_BY","LAST_UPD","LAST_UPD_BY", [...]
    SELECT "S_CONTACT_XM"."ROWID","S_CONTACT_XM"."ROW_ID",
    "S_CONTACT_XM"."CREATED",
    "S_CONTACT_XM"."CREATED_BY","S_CONTACT_XM"."LAST_UPD", [...]
    FROM "SIEBEL"."S_CONTACT_XM" "S_CONTACT_XM" ORDER BY par_row_id, type';

  gq :=
    'INSERT /*+ BYPASS_RECURSIVE_CHECK APPEND */
    INTO "SIEBEL"."BCK_S_CONTACT_XM"(M_ROW$$,"ROW_ID","CREATED",
    "CREATED_BY","LAST_UPD","LAST_UPD_BY", [...]
    SELECT /*+ FULL(S_CONTACT_XM) */ "S_CONTACT_XM"."ROWID",
   "S_CONTACT_XM"."ROW_ID", "S_CONTACT_XM"."CREATED",
    "S_CONTACT_XM"."CREATED_BY","S_CONTACT_XM"."LAST_UPD", [...]
    FROM "SIEBEL"."S_CONTACT_XM" "S_CONTACT_XM" ORDER BY par_row_id, type';

  EXECUTE IMMEDIATE 'create outline "' || outln_name_bq || '" on ' || bq;
  EXECUTE IMMEDIATE 'create outline "' || outln_name_gq || '" on ' || gq;

  -- delete the hints from the bad outline
  DELETE FROM outln.ol$hints
  WHERE ol_name = outln_name_bq;

  -- rename the good hints so they apply to the bad outline
  UPDATE outln.ol$hints
     SET ol_name = outln_name_bq
   WHERE ol_name = outln_name_gq;

  -- update count of hints on the bad outline to that of the good one
  UPDATE outln.ol$
  SET hintcount = (SELECT hintcount
                     FROM outln.ol$
                    WHERE ol_name = outln_name_gq)
  WHERE ol_name = outln_name_bq;

  COMMIT;

  EXECUTE IMMEDIATE 'drop outline "' || outln_name_gq || '"';
END;
/
 

Now the outline is created, because the query can be parsed. And the ‘lucky’ thing is: I can drop that m_row$$ column again without the outline becoming invalid:

ALTER TABLE bck_s_contact_xm DROP COLUMN m_row$$;

Now the intermediate table is in the state it should be and I can continue with the DBMS_REDEFINITION. START_REDEF_TABLE, which will add the m_row$$ column to the intermediate table again and the outline will be used during the internal INSERT statement.

If I can, I always try to get a feeling for the cardinality estimates the optimizer makes, inspired at first by Wolfgang Breitling. The optimizer is strongly driven by expected cardinality and miscalculations, especially in the early stages of an execution plan, lead to suboptimal or even bad plans. There are several reasons for miscalculations (among those data skew and correlated predicates), for which histograms, dynamic sampling and if possible query redesigns are some solutions. Too many times I see predicates in a query that are heavily or even completely correlated. If these latter can be avoided, the better.

The query mentioned was also a case of predicate correlation and I want to show you that the existence of an index, even though not really used for query execution, may still give the optimizer better information.

This is the query:

SELECT   t70910.lvl1anc_divn AS c1, SUBSTR (t70910.lvl1anc_postn, 6, 4) AS c2,
         t70910.lvl1anc_postn AS c3,
         CONCAT (CONCAT (t153459.emp_last_name, ', '),
                 t153459.emp_fst_name
                ) AS c4,
         COUNT
            (DISTINCT CASE
                WHEN t427832.accnt_con_flg = 'C'
                   THEN t427832.contact_wid
             END
            ) AS c5
    FROM wc_prod_rank_tc_a t426428 /* Contact Rank OA (WC_PROD_RANK_TC_A) */,
         w_source_d t163300 /* Sales Cycle (W_SOURCE_D) */,
         w_position_d t69295 /* Position (W_POSITION_D) */,
         w_position_dh t70910 /* SalesForce_Area */,
         w_position_d t153459 /* Lvl1Position_Area (W_POSITION_D) */,
         wc_per_rank_tc_a t427832
   WHERE (    t163300.row_wid = t427832.cycles_wid
          AND t70910.lvl1anc_postn_id = t153459.integration_id
          AND t69295.row_wid = t70910.row_wid
          AND t69295.row_wid = t427832.owner_postn_wid
          AND t70910.lvl1anc_divn = 'SomeValue'
          AND t163300.mktpln_name = 'SomeValue 2008'
          AND t426428.row_wid = t427832.contact_rnk_wid
          AND t427832.datasource_num_id = 10008.0
          AND t427832.accnt_con_flg = 'C'
          AND CAST (t426428.ranking AS INTEGER) <> 1
          AND t70910.lvl1anc_postn LIKE 'UK-1-%'
          AND (t69295.datasource_num_id IN (-1.0, 10008.0))
          AND (t163300.datasource_num_id IN (-1.0, 10008.0))
          AND (t426428.tc_cat IN ('Professional Overall', 'Unspecified'))
          AND (t426428.datasource_num_id IN (-1.0, 10008.0))
          AND t70910.lvl1anc_postn NOT LIKE '%OLD%'
         )
GROUP BY t70910.lvl1anc_postn,
         t70910.lvl1anc_divn,
         CONCAT (CONCAT (t153459.emp_last_name, ', '), t153459.emp_fst_name)
  HAVING 0 <
            COUNT
               (DISTINCT CASE
                   WHEN t427832.accnt_con_flg = 'C'
                      THEN t427832.contact_wid
                END
               )
     AND 0 <
            COUNT
               (DISTINCT CASE
                   WHEN t427832.accnt_con_flg = 'C'
                      THEN t427832.contact_wid
                END
               )
ORDER BY c3, c4, c1;

And by examining the execution plan with the “gather_plan_statistics” hint, I noticed quite a difference between the number of records estimated (186) for the WC_PROD_RANK_TC_A materialized view access and the actual records (912).  I started with dynamic sampling levels 3 and 4 and they already made the query run in 47 seconds. And it was immediately visible that the cardinality estimate for that WC_PROD_RANK_TC_A part was now completely accurate: 912.

However,  for that moment, dynamic sampling was not an option: we cannot add hints, we cannot alter sessions and we cannot just set it at the system level without testing (a representative set of) all other reports. It is certainly planned to test dynamic sampling level 4 soon.

Here is the explain plan for this part of the query without dynamic sampling:

SELECT *   FROM wc_prod_rank_tc_a t426428  
WHERE CAST (t426428.ranking AS INTEGER) <> 1   
  AND (t426428.tc_cat IN ('Professional Overall', 'Unspecified'))   
  AND (t426428.datasource_num_id IN (-1.0, 10008.0));



Plan


ALL_ROWS SELECT STATEMENT Cost: 4 Bytes: 24,924 Cardinality: 186 Time: 1 


 
2 PARTITION LIST INLIST Cost: 4 Bytes: 24,924 Cardinality: 186 Time: 1 Partition #: 1 Partitions accessed #KEY(INLIST)


 
 
1 SIEBELP.WC_PROD_RANK_TC_A Filter Predicates: ("T426428"."DATASOURCE_NUM_ID"=(-1) OR "T426428"."DATASOURCE_NUM_ID"=10008.0) AND ("T426428"."TC_CAT"='Professional Overall' OR "T426428"."TC_CAT"='Unspecified') AND CAST("T426428"."RANKING" AS INTEGER)<>1 MAT_VIEW ACCESS FULL MAT_VIEW Cost: 4 Bytes: 24,924 Cardinality: 186 Time: 1 Partition #: 1 Partitions accessed #KEY(INLIST)

And with dynamic sampling level 4 (level 3 in this case was also enough):

Plan


ALL_ROWS SELECT STATEMENT Cost: 4 Bytes: 122,208 Cardinality: 912 Time: 1 


 
2 PARTITION LIST INLIST Cost: 4 Bytes: 122,208 Cardinality: 912 Time: 1 Partition #: 1 Partitions accessed #KEY(INLIST)


 
 
1 SIEBELP.WC_PROD_RANK_TC_A Filter Predicates: ("T426428"."DATASOURCE_NUM_ID"=(-1) OR "T426428"."DATASOURCE_NUM_ID"=10008.0) AND ("T426428"."TC_CAT"='Professional Overall' OR "T426428"."TC_CAT"='Unspecified') AND CAST("T426428"."RANKING" AS INTEGER)<>1 MAT_VIEW ACCESS FULL MAT_VIEW Cost: 4 Bytes: 122,208 Cardinality: 912 Time: 1 Partition #: 1 Partitions accessed #KEY(INLIST)

So with dynamic sampling as “not an option for now”, my next plan was to create a (function based)  index on the predicates for this table:

CREATE INDEX SIEBELP.TVB_WC_PROD_RANK_TC_A
ON SIEBELP.WC_PROD_RANK_TC_A (TC_CAT, CAST("RANKING" AS INTEGER), ROW_WID)  
LOCAL;

And now look at that same statement’s explain plan:

It is not entirely accurate, but estimating 3 times more rows than 5 times less was much better in this case. With the index, even though it was not used to execute the query, only to estimate  cardinality, the query returned in about 40 seconds instead of an hour.

Today was the first day of the second Miracle Masterclass this year: Tanel Põder has come to The Netherlands, saw a room filled with a lot of people (he called it the best seminar even before he started, just because of the amount of attendants) and made me feel I have an awful lot to explore yet. However, that is in fact a good thing and it made me get through the “trains nightmare” today with a smile: I had trouble in the morning since I left home far too late, had to cycle as fast as I could and jumped completely exhausted with sour legs into an already late train packed with people, so I had to stand. Good start of the day. And on the way back tonight all trains around Utrecht had trouble because of some “logistic problems”. Eventually I was not unlucky: only a half hour delay, although standing again (great thing a First class ticket to Utrecht, I could really use it …NOT!). But hey, I said I went through it with a smile. Because this first seminar day was already worth the money. The more new things you hear may make you feel exhausted and even a little frustrated, but at the same time satisfied because the net pay-off is awesome, right?

Cary Millsap told me (and others) Tanel is a very clever guy who knows extremely much about Oracle internals. So I checked out his blog and I figured I should be able to get a lot out of this course so I subscribed, even though my customers have limited Unix/Linux Oracle installations. To my own amazement I was able to really understand most of it (I guess my Unix/Linux skills are not so bad). It only goes so fast. As I told Anjo (Kolk, from Miracle) at the end of the day: when I was not paying attention for a few moments (dreaming about white sandy beaches or my well going 400 metres hurdles training of yesterday evening, which may very well have contributed to some problems of staying focused all time ), I looked at a screen showing the results of yet another demo and wondered: “What the heck caused that result?” Many, many demo’s and Tanel switching screens faster than my split times during the 400 metres hurdles training! And I was running very well…

What I like in particular is the drive Tanel has to create scripts that need as little privileges and change as possible. A lot of sites just have a strong policy that you cannot create packages, types, get access to X$-views etc. I am working in a place like that at this moment where I am facing the issue that they are going to revoke Unix access from everyone not in the official IT department. And I am hired by the application team as a performance troubleshooter. It is in such places that you need tools and scripts that work with the minimum amount of effort and do not need to go through days and weeks of change approvals. I was amazed by the way Tanel emulates ‘loops’ within a SQL statement, taking advantage of the fact that V$-views are not Read consistent. Check out his blog, you will find a lot of valuable scripts and tools there. And as a last comment before I hit the bed: today I really saw why most Oracle guys like Unix/Linux so much. Windows has a number of drawbacks when it comes to troubleshooting problems (and then there is that paradox: in a Operating system known as  having frequent and strange problems, would you not like to have the best tools to troubleshoot them?) 

Audience from behind

03-04-2009, update to the post:  day two ended as well. No train problems today and the advantage of that is that I can use the travel time to write an update yesterday’s post.  

It was a great day again. Lots of interesting stuff, luckily also a lot of recognition for me now, otherwise I would have been exhausted again . The first day we talked about the different layers in the stack and the Oracle kernel layers. We dealt with the way to approach a problem and what to do if the usual ways do not show any data (V$SESSION_WAIT, SQL*Trace, ASH). We talked about latches and mutexes (a kind of cheaper latch available in the newer versions). We have seen the usage of ps, top/prstat, pmap, ipcs, pstack, strace (all to show Oracle process activity from an OS point of view) and Tanel’s snapper script. The word “script” might have been the top contributor when it comes to words spoken these two days. There was hardly a single thing where Tanel could not say the magic words: “I hate to do things by hand, so I have a script for that”.

 The second day was mainly about execution plans, row sources, application instrumentation and tracing beyond the SID level (at the module or client_identifier level, if your connection pool applications call some instrumentation functions to set the client identifier, the module, action and client_info). Oracle has efficient ways to set these kinds of information along with the SQL statement in one network round trip by using OCI calls. In two applications I support I am really in need with more information about the actual user issuing the database calls or using the session. In both applications (Siebel and FileNet) this is not instrumented out of the box and I need to see if we can code it in somewhere. It would be so helpful if these packaged applications would at least provide this minimum amount of instrumentation. 

Tanel in discussion

 Also interesting was the topic on the life cycle of a cursor and why SQL cannot always be shared, even though the SQL text is similar. Since this topic touched the SQL Hash value and SQL_ID I raised the question if you can generate the SQL_ID or hash value when you know the SQL statement. In other words: is the function that generates the hash value and the SQL_ID available to us? Tanel and Anjo discussed some things about this, but we came to the conclusion that this hashing algorithm (and SQL_ID algorithm) is not publicly available. Anyone knows whether it probably is? I would love it, since we use a great tool written by one smart guy at our customer to display performance characteristics of Siebel applications (parsing Siebel log files) and the SQL stored in there I would love to match with SQL I see in V$SQL…but to search on the SQL_TEXT is both slow and cumbersome (especially with the long Siebel queries which are a lot of the time very similar, but slightly different).

The seminar was a big success: a very technical one presented by indeed (Cary was right) very smart guy. I like Oracle internals and if you like it too, go and see Tanel Põder’s seminar and read his blog. You have to have a solid background in Oracle otherwise it might be very frustrating. And some Unix/Linux knowledge is important!

Thanks Tanel, for two great days (and thanks Anjo and Annette as well for giving the opportunity)! See you next time!

First discovery

We soon found out the ETL and reports problems all had to do with this new materialized view, which was an ‘exact’ replacement of a manually refreshed aggregate table with the exact same indexes and columns. The statistics had also been gathered on the MV. I looked at the execution plans of the badly behaving ETL first and saw that Oracle was using a GLOBAL index defined on the new MV. The cardinality estimates in the execution plan were 1, however, the number of records to be retrieved were in the millions. So it had to be  a statistic issue. I checked and saw that all the indexes were last analyzed somewhere during the normal ETL and to be exact: they were analyzed as part of the COMPLETE refresh. At first I wondered: who did an analyze of that table’s indexes during the ETL, is it defined in the ETL somewhere? It took me a few moments to realize that the way we do the COMPLETE refresh is a so called ‘non-atomic’ one:

begin
  dbms_mview.REFRESH('SOME_JOIN_MV','COMPLETE',atomic_refresh=>FALSE);
end;
/

This effectively does a COMPLETE refresh by doing a TRUNCATE, followed by a direct path insert and an efficient rebuild of all the indexes. And it was this rebuild of the indexes that lead to the analyze of those indexes: Oracle 10g and above automatically (re-)analyzes the new or rebuilt indexes on a table when this table already has statistics gathered. So OK, I found out where the analyze came from. But why was the cardinality so wrong? I checked and to my surprise the num_rows statistic was set to zero (not NULL) for the GLOBAL indexes only. The partitioned indexes however where analyzed fine: their statistics were accurate. So I went testing this behavior (after first correcting the statistics on those indexes in Production to have performance return to normal) and I saw what Oracle did (among others by taking a SQLtrace during the COMPLETE refresh). And this was the situation on HP-UX Oracle 10.2.0.4 64bit:

1. Oracle did a TRUNCATE of the MV
2. Oracle set the indexes to unusable AND set the statistics of the GLOBAL indexes to zero!
3. Oracle did the Direct Path Insert
4. Oracle rebuilt the indexes –> automatic analyze of the LOCAL partitioned indexes, but not of the GLOBAL ones

So after the refresh the statistics on the GLOBAL indexes were zero and the LAST ANALYZE was from the moment after the TRUNCATE, but the LOCAL indexes statistics were fine: they seemed to have gathered normally during the rebuild. To me that sounds like a bug. I cannot explain why this behavior would be like that. And that was the cause of the performance problems: with index statistics set to zero, Oracle considered these GLOBAL indexes as extremely cheap and used them in about every query. But that was completely misleading information. I adjusted the materialized view refresh stored procedure by adding an explicit dbms_stats.gather_index_stats on those GLOBAL indexes and everything went back to normal.

But then…

 …I wanted to wrote this post and started retesting this on my Windows XP 64bit 10.2.0.4. The same version (I may be off by a minor patchlevel, I need to check) and the same kind of MV (not the same one, but also a join MV). To my surprise during test Oracle did a good job at the COMPLETE refresh for the GLOBAL indexes: now these were gathered also and to be most specific: as one of the last steps of the refresh. Seems like the thing that was forgotten on the HP-UX version. But now another thing seems wrong: the TABLE level statistics of the LOCAL partitioned indexes are not gathered. The example I prepared was:

DROP TABLE some_fact;
DROP TABLE some_domain;

CREATE TABLE some_fact AS SELECT * FROM all_objects;
CREATE TABLE some_domain AS SELECT * FROM all_tables;

CREATE UNIQUE INDEX pk_some_domain ON some_domain (owner,table_name);

CREATE MATERIALIZED VIEW some_join_mv
PARTITION BY LIST (object_type)
(   PARTITION part_table VALUES ('TABLE'),  
    PARTITION part_index VALUES ('INDEX'),  
    PARTITION part_proc VALUES ('PROCEDURE','PACKAGE','FUNCTION'),  
    PARTITION part_remaining VALUES (DEFAULT))
BUILD IMMEDIATE REFRESH
COMPLETE ON DEMAND AS
SELECT f.object_name
     , f.object_type
     , f.owner
     , f.status
     , f.last_ddl_time
     , d.avg_row_len
     , d.blocks
     , d.empty_blocks
     , d.partitioned
     , d.tablespace_name
  FROM some_fact f, some_domain d
 WHERE f.object_name = d.table_name(+) AND f.owner = d.owner(+);

BEGIN
   DBMS_STATS.gather_table_stats (ownname               => USER
                                , tabname               => 'SOME_JOIN_MV'
                                , estimate_percent      => 100
                                , granularity           => 'ALL'
                                , method_opt            => 'FOR ALL COLUMNS SIZE 1'
                                , CASCADE               => TRUE
                                 );
END;
/

CREATE  INDEX idx_some_join_mv_p1 ON some_join_mv (object_name, object_type, owner) GLOBAL;
CREATE  INDEX idx_some_join_mv_type_local ON some_join_mv (object_type) LOCAL;

SELECT   index_name
       , table_name
       , partition_name
       , leaf_blocks
       , distinct_keys
       , clustering_factor
       , num_rows
       , last_analyzed
    FROM all_ind_statistics
   WHERE table_name = 'SOME_JOIN_MV'
ORDER BY index_name, partition_name NULLS FIRST;

Results were (the TABLE level statistics of the LOCAL partitioned index in Red):

Then I deleted almost all records in the Fact table, did a COMPLETE refresh and see what the index statistics were after that:

DELETE FROM some_fact
      WHERE ROWNUM <= 77000;
COMMIT ;

BEGIN
   dbms_mview.REFRESH ('SOME_JOIN_MV', 'COMPLETE', atomic_refresh => FALSE);
END;
/

See the inaccuracy:

All indexes have accurate statistics, but the TABLE level statistics on IDX_SOME_JOIN_MV_TYPE_LOCAL are still the old ones.

So I witnessed two strange things in INDEX statistics with COMPLETE refreshed materialized views. Can anyone tell me the reasons…? At least I learned to pay carefull attention to the statistics when dealing with materialized views refreshes, that is for sure!

At one customer, we are upgrading from Oracle 8.1.7.4 to Oracle 11g. It was about time. So far, there are not many challenges. The few problems so far have to do with migrating from a Single Byte character set to a Multibyte character set (on purpose) and some PL/SQL mismatches. Today I was looking into some package bodies that did not compile in 11g. The error was a PLS-00593: default value of parameter “x” must match that of spec.  So we seemed to have a few packages with one or more functions in them in which the parameter default values did not match. So I had to straighten that and I wondered: what is the one it must be if I do not want the risk of changing the application logic that may rely on the default values. The one in the specification, or the one in the body? The best way to answer that question is to test it:

CREATE OR REPLACE PACKAGE par_test
IS
  PROCEDURE test_par (a IN VARCHAR2, b IN VARCHAR2 DEFAULT 'TRUE');
END;
/ 

CREATE OR REPLACE PACKAGE BODY par_test
IS
  PROCEDURE test_par (a IN VARCHAR2, b IN VARCHAR2 DEFAULT 'FALSE')
  IS
  BEGIN
   DBMS_OUTPUT.put_line ('b=' || b);
  END;
END;
/

BEGIN
  par_test.test_par ('And the answer is...', 'UNKNOWN'); -- specify a value for parameter b
  par_test.test_par ('And the answer is...'); -- rely on default value
END;
/

Result:
b=UNKNOWN
b=TRUE

So it seems to be the default value of the package specification that rules…

PLS-00801: internal error [phd_get_defn: unexpected overloaded D_S_ED]

One other error that came along was in one other package body. I got the scary looking PLS-00801: internal error [phd_get_defn: unexpected overloaded D_S_ED]. There was not much to be found on the Internet, except that you need to contact Oracle support if you get it. I like to check if I can isolate the cause for it first and then probably solve it.

The package body consists of a lot of generated code, generated by PL/Generator (a PL/SQL generator I have used with lots of joy. It is created by Steven Feuerstein and has an even much more powerful successor called the Quest CodeGen Utility now). Since almost all the packages in the schema contained similar generated code, I had a feeling the error must be caused by my own customized code. So I commented out the custom procedures/functions in the specification and tried to compile both spec and body. That already compiled fine. So I started to un-comment the custom modules one by one and very soon I found that the error was caused by calls to

PRAGMA RESTRICT_REFERENCES(code_convert_to,WNDS,WNPS);

 These packages were created in the years of Oracle 7 and back then it was a lot of times necessary to “tell” Oracle that a function adhered to certain rules, like that it did not changed package state or database state, in order to use it in regular SQL statements. As years went by, the compiler got more intelligent and Oracle could find out a lot of these things itself. I did not expect that I would get a nasty PL/SQL error on the remains of this “Oracle 7″ code once… Well, it may be of use to anyone who gets this same error, since Google and MetaLink did not help me here.