Archive for September, 2008

How to route / send all workflow mailer notifications to a single email id / address ?

Posted on September 12, 2008. Filed under: Clone, Configuration, DBA/Admin Tips, R12, Workflow Mailer | Tags: , , , |

Mostly, this option will come handy after a clone, wherein you want to retain all the production / source instance data and continue to run workflow mailer but without sending emails to the real end users.

To achieve this you can set an Override address to route all the notifications to a single email address.

Steps

1. Login as sysadmin
2. Select System Administration Responsibility
3. Click Workflow under Oracle Applications Manager
4. Click on the tick mark or page icon next to Notification Mailers
5. Click on Workflow Notification Mailer under Name column
6. Click “Set Override Address” button next to Test Mailer button
7. Key in the new email address and click submit.

After submitting, Java mailer will send a verification email to the mentioned email address with a verification code and link to activate the override entry.

Either you can request the email id owner to send you the email with the verification code so that you can key-in the code or if the email address owner has access to Oracle Apps, he/she can click the link and enter the verification code.

cheers,
OraclePitStop.

Advertisements
Read Full Post | Make a Comment ( 2 so far )

Blank AppsLogin / AppsLocalLogin Page in R12?

Posted on September 12, 2008. Filed under: Apps Technology Stack, Configuration, Diagnostics, R12, TroubleShooting | Tags: , , , |

There were quite a few times i had encountered this blank page issue and the root cause was not the same all the times. There are many reasons for this issue to occur. Listed below are some of them.

Scenario 1: After a clone

Solution :

1. adopmnctl.sh stopall — to shutdown opmn managed services viz., opmn, forms, oacore & oc4j
2. cd $COMMON_TOP/_pages
3. mv _pages _pages_old
4. cd $FND_TOP/patch/115/bin or export PATH=$FND_TOP/patch/115/bin:$PATH
5. Compile all jsps.

perl ojspCompile.pl –compile –flush -p 2

(-p represents parallel threads, if you have enough horse power on your server, you can increase it to 8 or 10)

6. adopmnctl.sh startall

Scenario 2: Database Listener is down

Solution: Obvious answer, start the database listener.

Scenario 3: Database Archive Locaion is 100% full

Solution: Either temporarily move the archive files to another location or delete older archive files (provided you had taken a cold backup sometime).

Scenario 4: IAS Cache initialization failed

This issue occurs only if you have more than one mid-tier and you had enabled Distributed Java Caching.

Error Message in log file:

Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException:
IAS Cache initialization failed.
The Distributed Caching System failed to initialize on port: 12357.
The list of hosts in the distributed caching system is: [node 1] [node 2].
The port [port no] should be free on each host running the JVMs.
at oracle.apps.jtf.cache.IASCacheProvider.init(IASCacheProvider.java:214)
at oracle.apps.jtf.cache.CacheManager.activateCache(CacheManager.java:1451)

In this scenario, s_java_object_cache_port value was listening on node 2 but not on node 1. So whichever connection went to node 1 it returned a blank page and occasionally it displayed 500 Internal Server error.

Solution:

Autoconfig

1) Bring down opmn services on both the boxes.
2) Verify no process is listening for s_java_object_cache_port value. click here to view the commands.
2) Change s_java_object_cache_port on all the nodes in the context file
3) Run autoconfig

Non-Autoconfig

s_java_object_cache_port value is referrenced in two places

1) $INST_TOP/ora/10.1.3/javacache/admin/javacache.xml
2) Profile option name JTF_DIST_CACHE_PORT

Change the port value to a new number and bounce opmnctl services. Dont forget to change it in $CONTEXT_FILE. (To take effect during next autoconfig)

Verification:

Use commands listed in step 2 of autoconfig option.

If the new port mentioned still doesnt listen on the boxes you can use below workaround. This workaround will resolve the “IAS Cache initialization failed” even if the port is not listening.

Workaround

Set LONG_RUNNING_JVM=false in $INST_TOP/ora/10.1.3/j2ee/oacore/config/oc4j.properties
Bounce opmn services.

In 11i, if the database listener is down or if the archive location is full, atleast we get the internal server page while trying to access the login page, but in R12 nothing happens except for the blank page and apache error log will have an entry “oc4j_socket_recvfull timed out” with additional error message mod_oc4j: request to OC4J [mid-tier servername:ajp port] failed: Connect failed

So the moral of the story is recompiling the jsps is not always the solution if you get a blank login page.

cheers,
Ram.

Read Full Post | Make a Comment ( 2 so far )

How to change Java heap size in R12 to avoid java.lang.OutOfMemoryError: Java heap space error?

Posted on September 11, 2008. Filed under: Apps Technology Stack, Configuration, Diagnostics, Operating System, R12, Technology, TroubleShooting | Tags: , , , , |

Recently we had encountered a situation, wherein end users where not able to get the R12 login page and
it just hangs, but all the opmn processess were up and running – adopmnctl.sh status gives status as
alive for all the components viz., HTTP, oafm, oacore & forms. The same environment was available and
end users were able to access without any isuses 15 minutes before. This error happens only when
multiple people login to the same page and perform simillar activity like employee self service form
etc. So where and what exactly could be the problem.

This is how we approached and resolved the issue.

First we checked apache error log. The following error was reported in the error log.

==============================================================
[warn] [client IP Address ] oc4j_socket_recvfull timed out
[error] [client IP Address ] mod_oc4j: There is no oc4j process (for destination: application://oacore) available to service request.
[error] [client  IP Address] mod_oc4j: request to OC4J [servername:OC4J AJP Port Range for Oacore]
failed: Connect failed
==============================================================

Above error message does give us a hint that the problem is with oacore, but as i said earlier the oacore is alive according to opmn.

Next, we verified the oacore log file.

($INST_TOP/logs/ora/10.1.3/opmn/oacore_default_group_1/oacorestd.err)

Error Message in the file

=======================================================
Exception in thread “OC4JMonitorThread” java.lang.OutOfMemoryError: Java heap space
=======================================================

While checking the error in file, parallely we opened another window to see the CPU and memory usage and we could see one java process was taking more than 100% CPU. This process was spawned by the opmn -d process and the process id didnt match with the oacore process id. Hint: adopmnctl.sh status will give the status as well the process id. It looked like a end client java process.

At this stage we had 3 options.

1. Kill the java process which is consuming high cpu.
2. Bounce oacore component
3. Adjust java (jvm) memory parameters

Each of the above options has its own implications and advantages. To minimize the downtime we decided  to kill the java processes, and the moment we killed the java process, all the browsers which were hanging reported instantaneously Internal Server Error. This proved to be a bad decision.

So we moved to the next option, bouncing the oacore service which we are sure will resolve the issue  (temporarily) and it did as expected. Basically when you bounce the services all the existing
connections and processess will be released which results in more available memory when you re-start the services.

Ok, now we tackled the problem and provided a temporary solution but we need to find a long term solution. This is option 3, adjusting java memory size.

Steps to change the heap size.

First, you need to identify how much is the maximum heap size that you can set. Click here.

Once you had identified the maximum heap size, we need to change the configuration files to make it
permanent.

Step 1: Edit opmn.xml file.

Location : $INST_TOP/ora/10.1.3/opmn/conf/

Open opmn.xml

Search for string Xms or Xmx or module-id=”OC4J”

This search should lead you to below location

‘<‘process-type id=”oacore” module-id=”OC4J” status=”enabled” working-dir=”$ORACLE_HOME/j2ee/home”‘>’
‘<‘category id=”start-parameters”‘>’
‘<‘data id=”java-options” value=”-server -verbose:gc -Xmx512M -Xms128M ……]

The default value for Maximum (-Xmx) and Minimum (-Xms) heap sizes are 512M and 128M respectively.

Again here you have options, you can set both Xms and Xmx has the same value as Xmx if you feel all your sessoins require higher memory or set a lower value for Xms and the maximum value for Xmx. Dont forget to change the values under ‘<‘category id=”stop-parameters”‘>’

opmn.xml also contains jvm configurations for other components – oafm & forms.

Step 2: Edit oc4j.properties file.

Location : ($INST_TOP/ora/10.1.3/j2ee/oacore/config)

This step is optional since we had already made changes in opmn.xml but there is no harm in making the change here. This step will come handy for troubleshooting specific components of Oracle viz., configurator, iSupplier or any other option which heavily utilizes/consumes CPU/memory.

Search for string Xms or Xmx or wrapper.

Option 1: If you find any of the above parameters change the values corresponding to the value you had  mentioned in opmn.xml or even more than that, as long as you dont exceed the maximum heap size limit.

Option 2: If you DO NOT find any of the above parameters, then make an entry like this, under the heading “# Java Object Cache Configuration Parameters”

wrapper.bin.parameters=-Xms[Value]M -Xmx[Value]M -XX:NewSize=256M -XX:MaxNewSize=256M

Step 3: Edit Applications Context file

vi $CONTEXT_FILE

Location: $INST_TOP/appl/admin/SID_hostname.xml

search for string s_oacore_jvm_start_options

Change Xms and Xmx value. Repeat the same step for parameter s_oacore_jvm_stop_options.

Changes made in Step 3 will take effect the next time you run autoconfig, whereas Step 1 & 2 changes will take effect the next time you bounce opmn services, but the values are not permanent in the sense these will be wiped off next time you run autoconfig. Yes you can preserve the changes by placing it inbetween Begin and End customizations.

You can also increase the Garbage Collection threads parameter (-XX:ParallelGCThreads) to a higher value if you have JDK 1.5 or more than 2 cpus or more memory. For more information on this you can refer to Metalink Note: 362851 – Guidelines to setup the JVM in Apps Ebusiness Suite 11i and R12.

cheers,
oraclePitStop.

Read Full Post | Make a Comment ( 8 so far )

How to identify/determine the maximum JVM Heap Size for OC4j components in R12 / 10gAS

Posted on September 11, 2008. Filed under: Apps Technology Stack, Configuration, Operating System, R12, TroubleShooting | Tags: , , , , , , |

Below command will help you to identify the maximum memory heap size than can be allocated to a JVM
process.

java -mx[value]m -version

Start with a higher value like 4G or higher and slowly cut down to the maximum allowed value.

Sample

$ java -mx4096m -version

Invalid maximum heap size: -Xmx4096m
The specified size exceeds the maximum representable size.
Could not create the Java virtual machine.

As you can see my system cant support 4Gig for a JVM.

Now lets reduce to a lesser value, viz., 3Gig

$ java -mx3072m -version
java version “1.5.0.04”
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0.04-_27_jul_2006_10_52)
Java HotSpot(TM) Server VM (build 1.5.0.04 jinteg:07.27.06-16:18 IA64, mixed mode)

Now its clear that the maximum allowed value is 3gig.

cheers,
OraclePitStop.

Read Full Post | Make a Comment ( 3 so far )

R12 – Responsibilities not visible to user

Posted on September 4, 2008. Filed under: DBA/Admin Tips, R12, TroubleShooting, User, Workflow Mailer | Tags: , , , , , |

We recently encountered this issue on a R12 instance fully functional instance.

1) Login as sysadmin or any user who as privileges to create user/add responsibilities
2) Create a new user and add responsibilities or add additional seeded/custom responsibilities to existing user

Then,
3) New/existing user logs in but he/she cant see the new responsibility.

Strange isnt it. I had encountered a simillar situation post 11.5.x to 11.5.10.2 upgrade, but the underlying problem is different here.

To tackle this situation, we ran the concurrent program “Workflow Directory Services User/Role Validation” with below parameters:

p_BatchSize – 10000 (Default Value 10000)
p_Check_Dangling – Yes (Default value No)
Add missing user/role assignments – Yes (Default Value No)
Update WHO columns in WF tables – No (Default Value No)

So what does this concurrent program do. It syncs all the user and role related WF tables with latest information viz., WF_LOCAL_ROLES, WF_LOCAL_USER_ROLES, WF_USER_ROLE_ASSIGNMENTS etc.,

The time taken to complete this program depends on the number of users/roles to be synched. After completion of this request the affected user can log back in to see the missing responsibilities.

You can also schedule this request to run every 15 minutes or so, but it depends how frequently you create/assign new users/roles/responsibilities.

cheers,
Ram.

Read Full Post | Make a Comment ( 6 so far )

Liked it here?
Why not try sites on the blogroll...