Is user equivelance really required at the OS level after RAC installation?

Posted on December 18, 2008. Filed under: Database, DBA/Admin Tips, Operating System, RAC, TroubleShooting, Uncategorized | Tags: , , , |

A wild though came to my head late this evening about the user equivelance in RAC. What will happen if i remove the user equivelance for the oracle home owner between the RAC nodes. What do you think – will this mess up the normal operation of the database ???

BTW, User Equivelance = Ability to ssh between rac nodes without password.


DB Name : opitstop
DB Owner : orapstop


1. srvctl stop database -d OPITSTOP
2. cd $HOME/.ssh && mv authorized_keys authorized_keys_old [ authorized_keys file holds the rsa/dsa keys, which enables ssh to other boxes without password]

At this stage, opitstop DB is down and RAC_NODE_2 cannot ssh to RAC_NODE_1 since RAC_NODE_1 authorized_keys file had been moved.


cd $HOME/.ssh && mv authorized_keys authorized_keys_old

Now both the nodes cannot do ssh between them without password.


Lets start the database, using srvctl to see if it errors out

srvctl start database -d OPITSTOP

NOPE, it doesnt … it works as before, started the instances on both RAC_NODE_1 and RAC_NODE_2.

At this stage in the alert log file, the database was able to get information about the other node and a select on gv$instance shows both instances too. I let the db to run for sometime … still there was no error.

Now lets revert back the user equivelance on both nodes.

orapstop@RAC_NODE_1: cd $HOME/.ssh && mv authorized_keys_old authorized_keys
orapstop@RAC_NODE_2: cd $HOME/.ssh && mv authorized_keys_old authorized_keys

So based on this excercise my conclusion is, user equivelance is required only for RAC installation’s / RAC binary upgrades viz., to or 4 / patching using opatch option napply and not for normal operations of the database.

Oh yeah, its a good feeling to mess up with the setup and to get it back working.

Happy Messing !!!


Read Full Post | Make a Comment ( None so far )

dbca – Creating RAC Database using dbca in silent mode

Posted on December 16, 2008. Filed under: Database, DBA/Admin Tips, Linux, RAC, TroubleShooting | Tags: , , |

During the course of the DBA life, we all know how irritating it is to create a DB or to install a Oracle Software on a UNIX Flavour box from a remote machine using a X – Server. Each click would take minutes to react (My favourite is the tab button on the keyboard) but still at-times we have to make use of the mouse. Anyways, recently we installed 10203 on a HP server which already has cluster setup. Everything went good in terms of 10201 installation, 10203 upgrade (both binaries) and even dbca 14 screens. But nothing happened when clicked on the finish button in dbca (14th screen). No useful information in the log files. I pressed cancel and re-launched it again and did the dbca exercise all the way till the end, still no luck. By this time my patience level was minus 25.  It is at this stage i thought how about creating the database in silent mode and you know what, creating the DB in silent mode is 200 times faster than the GUI and the db creation completed in less than 5 minutes.

Syntax to create a General Purpose RAC Database on a Cluster File System.

dbca -silent -createDatabase -templateName General_Purpose.dbc -gdbName opitstop -sid opitstop -sysPassword opitstop123 -systemPassword opitstop123 -emConfiguration NONE -datafileDestination /opt/oracle/oradata/opitstop -storageType CFS -nodelist RAC_NODE_1,RAC_NODE_2 -characterSet WE8IS08859P15 -listeners opitstop -memoryPercentage 30 -continueOnNonFatalErrors true

Place above command in a .sh file (shell script) say and just run it. Before running the script make sure the cluster services (crs) are running and also you have user equivelance between the rac nodes for the oracle owner ie., eg. if the oracle owner username is oraracdb in RAC_NODE_1 and RAC_NODE_2 then when you do a ssh from RAC_NODE_1 to RAC_NODE_2 it should go without asking for the password.

There are more options available with dbca. Some of them are -createDatabase, -createTemplateFromDB, -createCloneTemplate, -generateScripts, -deleteDatabase, -configureASM, -deleteInstance. For a complete list of the usage execute dbca -help.

DBCA Log Location: $ORACLE_HOME/cfgtoollogs/dbca/[SID]

Happy DB creation(s) !!!


Read Full Post | Make a Comment ( 3 so far )

Unable to generate forwarding URL. Exception: oracle.apps.fnd.cache.CacheException

Posted on November 18, 2008. Filed under: Apps Technology Stack, DBA/Admin Tips, Diagnostics, R12, TroubleShooting | Tags: , , |

While trying to access the login page below error is reported.

“Unable to generate forwarding URL. Exception: oracle.apps.fnd.cache.CacheException”

This simple message doesnot give much information. Digging into logs yielded below detailed error message.


html: Servlet error
        at oracle.apps.fnd.cache.AppsCache.get(
        at oracle.apps.fnd.profiles.Profiles.getProfileOption(
        at oracle.apps.fnd.profiles.Profiles.getProfile(
        at oracle.apps.fnd.profiles.ExtendedProfileStore.getSpecificProfileFromDB(
        at oracle.apps.fnd.profiles.ExtendedProfileStore.getSpecificProfile(
        at oracle.apps.fnd.profiles.ExtendedProfileStore.getProfile(
        at oracle.apps.fnd.common.logging.DebugEventManager.configureUsingDatabaseValues(
        at oracle.apps.fnd.common.logging.DebugEventManager.configureLogging(
        at oracle.apps.fnd.common.logging.DebugEventManager.internalReinit(
        at oracle.apps.fnd.common.logging.DebugEventManager.reInitialize(
        at oracle.apps.fnd.common.logging.DebugEventManager.reInitialize(
        at oracle.apps.fnd.common.AppsLog.reInitialize(
        at oracle.apps.fnd.common.AppsContext.initLog(
        at oracle.apps.fnd.common.AppsContext.initializeContext(
        at oracle.apps.fnd.common.AppsContext.initializeContext(
        at oracle.apps.fnd.common.AppsContext.<init>(
        at oracle.apps.fnd.common.WebAppsContext.<init>(
        at oracle.apps.fnd.sso.Utils.getAppsContext(
        at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(
        at oracle.apps.fnd.sso.AppsLoginRedirect.init(
        at com.evermind[Oracle Containers for J2EE 10g ( ].server.http.HttpApplication.loadServlet(
        at com.evermind[Oracle Containers for J2EE 10g ( ].server.http.HttpApplication.findServlet(
        at com.evermind[Oracle Containers for J2EE 10g ( ].server.http.HttpApplication.findServlet(
        at com.evermind[Oracle Containers for J2EE 10g ( ].server.http.HttpApplication.getRequestDispatcher(HttpApplicati
        at com.evermind[Oracle Containers for J2EE 10g ( ].server.http.HttpRequestHandler.doProcessRequest(HttpRequestHan
        at com.evermind[Oracle Containers for J2EE 10g ( ].server.http.HttpRequestHandler.processRequest(HttpRequestHandl
        at com.evermind[Oracle Containers for J2EE 10g ( ]
        at com.evermind[Oracle Containers for J2EE 10g ( ]
        at com.evermind[Oracle Containers for J2EE 10g ( ].util.ReleasableResourcePooledExecutor$
Caused by: oracle.apps.jtf.base.resources.FrameworkException: Error in IAS Cache: java.lang.NullPointerException: null Connection
        at oracle.apps.jtf.cache.IASCacheProvider.get(
        at oracle.apps.jtf.cache.CacheManager.getInternal(
        at oracle.apps.jtf.cache.CacheManager.get(
        at oracle.apps.fnd.cache.AppsCache.get(
        … 30 more
Caused by: oracle.apps.jtf.base.resources.FrameworkException:
An exception occurred in the method CacheAccess.get
The base exception is:
null Connection
        at oracle.apps.jtf.base.resources.FrameworkException.convertException(
        at oracle.apps.jtf.base.resources.FrameworkException.addException(
        at oracle.apps.jtf.base.resources.FrameworkException.<init>(
        at oracle.apps.jtf.base.resources.FrameworkException.<init>(
        at oracle.apps.jtf.base.resources.FrameworkException.<init>(
        at oracle.apps.jtf.base.resources.FrameworkException.<init>(
        at oracle.apps.jtf.base.resources.FrameworkException.<init>(
        … 34 more

There could be many root causes for this error. If you look closely into the errorstack, it complains about null connection.

Caused by:
Error in IAS Cache:
null Connection

Every access to the login page ( (in R12) establishes a connection to the database as GUEST user using the DBC file. (In 11i, the url page is a static html which doesnt establish a connection)

If the mid-tier gets a connection, it displays the login page else it throws “oracle.apps.fnd.cache.CacheException”.

Now lets look at some possible reasons for not getting a connection.

1. Database is down
2. Database is up but the listener is down.
3. Archive Location is full, which prevents new connections till freed-up.
4. The instance was recently cloned and the GUEST user password is not in sync.

When the instance is cloned from source, there is a remote chance that the guest user password is not in sync. By sync, i mean the profile option value and the dbc file value for guest user password.

Steps to verify the values (perform below steps in the mid-tier)

4.1 DBC File value



4.2 Profile Option Value

sqlplus apps/[apps_pwd]
select fnd_profile.value(‘GUEST_USER_PWD’) from dual;


Values from 4.1 and 4.2 should be in sync and in our case it is. Now lets connect to the database using the guest password.

4.3 GUEST User Connectivity Check

sqlplus apps/[apps_pwd]


5. Ok, i didnt do a clone and the GUEST user password is in sync. What next.

The next possible reason, the database is not allowing any new sessions. May be the maximum number of sessions defined at the database level had already reached.

If you had followed all the steps sequentially as defined above, you would have realised that database does accept new connections, since you were able to verify the GUEST user password profile option value. If SESSIONS parameter value had been reached you would have got the error ORA-00018: maximum number of sessions exceeded.

6. Now what, my database is accepting new sessions.

Next option is to look for stale database connections.

ps -fu [oracle_user] | grep -i ‘LOCAL=NO’

This command will give you a list of stale connections. Use below command to kill them and to free up resources.


ps -fu [oracle_user] | grep -i ‘LOCAL=NO’ | grep -v grep | awk ‘{print “kill -9 “$2}’ | ksh

Hope one of the above mentioned steps resolves your issue when you encounter “Unable to generate forwarding URL. Exception: oracle.apps.fnd.cache.CacheException”


Read Full Post | Make a Comment ( 6 so far )

How to enable trace with BIND variables and WAITs for a concurrent program?

Posted on November 12, 2008. Filed under: DBA/Admin Tips, Diagnostics, R12, TroubleShooting | Tags: , , |

In this link, we had mentioned about tracing a concurrent program, but the drawback of this method is it doesnt yield information about the BIND variable values. So in this post we will talk about it.


X submitted a concurrent request and its running for a long time and never completes. X approached you to see what is the problem with the concurrent request.


1. Inform X to logout from his ERP session.
2. Login as sysadmin user
3. System Administrator -> Profile (System)
4. Values: User = X, Profile = Initialization SQL Statement – Custom (you can also mention Initialization%Custom)
5. Under user field enter the following


[ANY_IDENTIFIER] = Any value to identify the trace file.

LEVEL 12 – Both Binds and Waits
LEVEL 4 – Only BIND Variables

All above levels will include the sql statements being executed and not only the binds and waits.

6. If tracing had been enabled at the concurrent program level as mentioned in this link, please disable it by removing the check box. [IMPORTANT]. If you have trace enabled for the concurrent program, then it will take precedence for the profile option.

7. Kill / cancel the long running concurrent request (if its still running)
8. Request X to login to ERP and instruct X to submit the concurrent request again and logout. Nothing else. This will ensure we capture the trace information specific to the concurrent request and not for other actions.

At this stage, you will find a trace file generated at $ORACLE_HOME/admin/SID_Hostname/udump with [ANY_IDENTIFIER] as part of it.

Inside the trace file you will be able to see information about the bind variable values being passed to and used by the concurrent program.


oacdty=12 mxl=07(07) mxlc=00 mal=00 scl=00 pre=00
oacflg=13 fl2=206001 frm=00 csi=00 siz=8 off=0
kxsbbbfp=9fffffffbf2e88d8 bln=07 avl=07 flg=09

oacdty=12 mxl=07(07) mxlc=00 mal=00 scl=00 pre=00
oacflg=13 fl2=206001 frm=00 csi=00 siz=8 off=0
kxsbbbfp=9fffffffbf2e88f8 bln=07 avl=07 flg=09

Note: Make sure you remove the profile option value for X after the job is done, else every action of X will generate a trace file in udump directory.

Happy Trouble shooting.


Read Full Post | Make a Comment ( 2 so far )

How to clear cache for a specific component in R12?

Posted on November 11, 2008. Filed under: Apps Technology Stack, Configuration, DBA/Admin Tips, Diagnostics, R12, Technology, TroubleShooting | Tags: , , , , |

As we all know the normal tendency for clearing cache in Oracle Apps is to delete the $COMMON_TOP/_pages directory. In 11i, after deleting we dont have to recompile the jsps but in R12 we have to recompile the pages since s_jsp_main_mode is set to justrun by default which is in turn updated in orion-web.xml. Click here for more info.

Ok coming to this post, how can i clear cache for a specific component of Oracle Apps R12 viz., iprocurement, istore etc., Is this even possible ? Yes it is and this option comes handy when somebody modifies a jsp page which is part of say istore, then it makes sense to clear cache for istore and not for the whole server.

Clearing cache for specific Component/Application in R12

Login as sysadmin
Select “Functional Administrator” responsibility
Go to Core Services ==> Caching Framework ==> Tuning
Query the application (for example iStore) or by Name or code.
Select the component and clear the cache.

Note: For some specific configuration changes, we need to bounce Apache also.

Clearing Global Cache in R12

a) Go to Functional Administrator Responsibility
b) Go to Core Services ==> Caching Framework
c) Go to Global configuration and clear all the cache from there.(Only If you want to clear all the  cache)

Hope this helps…..


About Sam: Sam is a guest author for this blog and had performed various installations & upgrades. He is well versed with tech stack components and applications maintenance. He can be contacted at

Read Full Post | Make a Comment ( 1 so far )

OPATCH_JAVA_ERROR : An exception of type “OPatchException” has occurred

Posted on November 4, 2008. Filed under: Apps Technology Stack, Database, DBA/Admin Tips, opatch, Patch, R12, TroubleShooting | Tags: , , |

While trying to apply a patch or listing inventory, using opatch utility (be it on the Database node or for the Applications Tech Stack) below error is encountered:

LsInventory: OPatch Exception while accessing O2O

OPATCH_JAVA_ERROR  : An exception of type “OPatchException” has occurred:
OPatch Exception:
OUI found no such ORACLE_HOME set in the environment
Can not get details for given Oracle Home
An exception occurs

There is no Interim Patch

OPATCH_JAVA_ERROR  : An exception of type “OPatchException” has occurred:
Can not get a list of inventory on this home.

ERROR: OPatch failed because of Inventory problem.


Imagine you are trying to apply a patch to 10.1.3 Tech Stack Oracle Home on R12.

What did i do wrong ?

1. Did i source the corrent environment file ?

Yes, i did, my echo $ORACLE_HOME command points to 10.1.3 home instead of the default 10.1.2 home. hmmm ok

2. Am i using the correct opatch utility?

yes, which opatch is pointing to $ORACLE_HOME(10.1.3)/Opatch/opatch

3. Is the inventory location accessible ?

Yes, oraInst.loc under Oracle Home (10.1.3) is pointing to a valid directory.

$ cat oraInst.loc

4. Does the inventory has information about the Oracle Home you are trying to patch?

How do i check that ….

$ cd /var/opt/oracle/oraInventory

$ ls -latr
total 112
drwxrwxr-x   3 username    dba  ..
drwxrwxr-x   2 username    dba  oui
-rwxrwxr-x   1 username    dba  sessionContext.ser
drwxrwxr-x   2 username    dba  ContentsXML
-rwxrwxr-x   1 username    dba
drwxrwxr-x   2 username    dba  Contents
drwxrwx—   3 username    dba  logs
-rwxrwxr-x   1 username    dba  install.platform
drwxrwxr-x   6 username    dba  .

$ cd ContentsXML
$ vi inventory.xml

<HOME NAME=”SID_TOOLS__masked_oracle_home_tech_st_10_1_2″ LOC=”[masked_oracle_home]/tech_st/10.1.2″ TYPE=”O” IDX=”1″/>


ooops … i dont …. THIS IS THE ISSUE

So what should i do now?

Pretty simple, add an entry for this Oracle Home just above or below HOME_LIST tag.

<HOME NAME=”SID_TOOLS__masked_oracle_home_tech_st_10_1_3″ LOC=”[masked_oracle_home]/tech_st/10.1.3″ TYPE=”O” IDX=”2″/>

After making changes,

<HOME NAME=”SID_TOOLS__masked_oracle_home_tech_st_10_1_2″ LOC=”[masked_oracle_home]/tech_st/10.1.2″ TYPE=”O” IDX=”1″/>
<HOME NAME=”SID_WEBOH__masked_oracle_home_tech_st_10_1_3″ LOC=”[masked_oracle_home]/tech_st/10.1.3″ TYPE=”O” IDX=”2″/>

Save the file.

Now you retry the same command it should go through.

If you dont want to update your inventory.xml which is under the global inventory location, you can pass a parameter to the opatch utility to use a different inventory location provided that inventory has information about this oracle home.

opatch -invPtrLoc [New .loc file location]

Happy troubleshooting !!!!


Read Full Post | Make a Comment ( 3 so far )

How to enable / generate trace for a form 11i / R12

Posted on October 6, 2008. Filed under: DBA/Admin Tips, Diagnostics, R12, TroubleShooting | Tags: , |

This step can be performed by any user who has access to the professional approach (ie., the forms not the jsp). By access i meant any responsibility which launches the form.

Steps: (In the forms window)

1. Click on Help -> Diagnostics -> Trace

2. Under trace you have 5 options – select the appropriate option

3. Key in the apps password in the prompted window.

4. The next window will specify the logfile name and location.


Read Full Post | Make a Comment ( None so far )

How to identify if a port is listening on a unix server

Posted on October 6, 2008. Filed under: Apps Technology Stack, DBA/Admin Tips, Diagnostics, Operating System, R12, TroubleShooting | Tags: , , , |

We often get into a situation wherein we have to check if a particular port is being used by some process.

netstat -an | grep [port number] | grep LISTEN

Command to identify the OS process associated with a port

lsof -i tcp:[port number]



Read Full Post | Make a Comment ( 1 so far )

httpd: bad group name dba

Posted on October 1, 2008. Filed under: Configuration, DBA/Admin Tips, Operating System, R12, SOA, Technology, TroubleShooting | Tags: , , , |

All were working fine on our SOA server untill we restarted the services. All the custom oc4j components came up fine except the HTTP component.

Below is the error in opmn/logs/HTTP_Server~1.log

apachectl startssl: execing httpd
httpd: bad group name dba

Executing the id command for the SOA owner returned

$ id
uid=5352(username_masked) gid=500()

Note the blank open and closed brackets next to 500.

The root cause of the problem was read access on file /etc/group had been revoked by the root user.
$ ls -l /etc/group
-r–r—–   1 root       3              955 Oct  1 06:36 /etc/group

Both username and group information is stored in the httpd.conf. This file is read everytime you start apache. Since there was no read access to the /etc/group file and also no group information available for the SOA owner httpd service failed to start.

After granting read access to everyone, the services came up fine.

Above solution not only holds good to SOA but also to any Oracle product which uses apache viz., Oracle Apps, Oracle 10gAS.


Read Full Post | Make a Comment ( None so far )

Blank AppsLogin / AppsLocalLogin Page in R12?

Posted on September 12, 2008. Filed under: Apps Technology Stack, Configuration, Diagnostics, R12, TroubleShooting | Tags: , , , |

There were quite a few times i had encountered this blank page issue and the root cause was not the same all the times. There are many reasons for this issue to occur. Listed below are some of them.

Scenario 1: After a clone

Solution :

1. stopall — to shutdown opmn managed services viz., opmn, forms, oacore & oc4j
2. cd $COMMON_TOP/_pages
3. mv _pages _pages_old
4. cd $FND_TOP/patch/115/bin or export PATH=$FND_TOP/patch/115/bin:$PATH
5. Compile all jsps.

perl –compile –flush -p 2

(-p represents parallel threads, if you have enough horse power on your server, you can increase it to 8 or 10)

6. startall

Scenario 2: Database Listener is down

Solution: Obvious answer, start the database listener.

Scenario 3: Database Archive Locaion is 100% full

Solution: Either temporarily move the archive files to another location or delete older archive files (provided you had taken a cold backup sometime).

Scenario 4: IAS Cache initialization failed

This issue occurs only if you have more than one mid-tier and you had enabled Distributed Java Caching.

Error Message in log file:

Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException:
IAS Cache initialization failed.
The Distributed Caching System failed to initialize on port: 12357.
The list of hosts in the distributed caching system is: [node 1] [node 2].
The port [port no] should be free on each host running the JVMs.
at oracle.apps.jtf.cache.IASCacheProvider.init(
at oracle.apps.jtf.cache.CacheManager.activateCache(

In this scenario, s_java_object_cache_port value was listening on node 2 but not on node 1. So whichever connection went to node 1 it returned a blank page and occasionally it displayed 500 Internal Server error.



1) Bring down opmn services on both the boxes.
2) Verify no process is listening for s_java_object_cache_port value. click here to view the commands.
2) Change s_java_object_cache_port on all the nodes in the context file
3) Run autoconfig


s_java_object_cache_port value is referrenced in two places

1) $INST_TOP/ora/10.1.3/javacache/admin/javacache.xml
2) Profile option name JTF_DIST_CACHE_PORT

Change the port value to a new number and bounce opmnctl services. Dont forget to change it in $CONTEXT_FILE. (To take effect during next autoconfig)


Use commands listed in step 2 of autoconfig option.

If the new port mentioned still doesnt listen on the boxes you can use below workaround. This workaround will resolve the “IAS Cache initialization failed” even if the port is not listening.


Set LONG_RUNNING_JVM=false in $INST_TOP/ora/10.1.3/j2ee/oacore/config/
Bounce opmn services.

In 11i, if the database listener is down or if the archive location is full, atleast we get the internal server page while trying to access the login page, but in R12 nothing happens except for the blank page and apache error log will have an entry “oc4j_socket_recvfull timed out” with additional error message mod_oc4j: request to OC4J [mid-tier servername:ajp port] failed: Connect failed

So the moral of the story is recompiling the jsps is not always the solution if you get a blank login page.


Read Full Post | Make a Comment ( 2 so far )

« Previous Entries

Liked it here?
Why not try sites on the blogroll...