Operating System

Is user equivelance really required at the OS level after RAC installation?

Posted on December 18, 2008. Filed under: Database, DBA/Admin Tips, Operating System, RAC, TroubleShooting, Uncategorized | Tags: , , , |

A wild though came to my head late this evening about the user equivelance in RAC. What will happen if i remove the user equivelance for the oracle home owner between the RAC nodes. What do you think – will this mess up the normal operation of the database ???

BTW, User Equivelance = Ability to ssh between rac nodes without password.

Assumptions

RAC Servers: RAC_NODE_1, RAC_NODE_2
DB Name : opitstop
DB Owner : orapstop

orapstop@RAC_NODE_1

1. srvctl stop database -d OPITSTOP
2. cd $HOME/.ssh && mv authorized_keys authorized_keys_old [ authorized_keys file holds the rsa/dsa keys, which enables ssh to other boxes without password]

At this stage, opitstop DB is down and RAC_NODE_2 cannot ssh to RAC_NODE_1 since RAC_NODE_1 authorized_keys file had been moved.

orapstop@RAC_NODE_2

cd $HOME/.ssh && mv authorized_keys authorized_keys_old

Now both the nodes cannot do ssh between them without password.

orapstop@RAC_NODE_1

Lets start the database, using srvctl to see if it errors out

srvctl start database -d OPITSTOP

NOPE, it doesnt … it works as before, started the instances on both RAC_NODE_1 and RAC_NODE_2.

At this stage in the alert log file, the database was able to get information about the other node and a select on gv$instance shows both instances too. I let the db to run for sometime … still there was no error.

Now lets revert back the user equivelance on both nodes.

orapstop@RAC_NODE_1: cd $HOME/.ssh && mv authorized_keys_old authorized_keys
orapstop@RAC_NODE_2: cd $HOME/.ssh && mv authorized_keys_old authorized_keys

So based on this excercise my conclusion is, user equivelance is required only for RAC installation’s / RAC binary upgrades viz., 10.2.0.1 to 10.2.0.3 or 4 / patching using opatch option napply and not for normal operations of the database.

Oh yeah, its a good feeling to mess up with the setup and to get it back working.

Happy Messing !!!

cheers,
OraclePitStop

Read Full Post | Make a Comment ( None so far )

How to identify if a port is listening on a unix server

Posted on October 6, 2008. Filed under: Apps Technology Stack, DBA/Admin Tips, Diagnostics, Operating System, R12, TroubleShooting | Tags: , , , |

We often get into a situation wherein we have to check if a particular port is being used by some process.
Command

netstat -an | grep [port number] | grep LISTEN

Command to identify the OS process associated with a port

lsof -i tcp:[port number]

cheers,

OraclePitStop

Read Full Post | Make a Comment ( 1 so far )

httpd: bad group name dba

Posted on October 1, 2008. Filed under: Configuration, DBA/Admin Tips, Operating System, R12, SOA, Technology, TroubleShooting | Tags: , , , |

All were working fine on our SOA server untill we restarted the services. All the custom oc4j components came up fine except the HTTP component.

Below is the error in opmn/logs/HTTP_Server~1.log

apachectl startssl: execing httpd
httpd: bad group name dba

Executing the id command for the SOA owner returned

$ id
uid=5352(username_masked) gid=500()

Note the blank open and closed brackets next to 500.

The root cause of the problem was read access on file /etc/group had been revoked by the root user.
$ ls -l /etc/group
-r–r—–   1 root       3              955 Oct  1 06:36 /etc/group

Both username and group information is stored in the httpd.conf. This file is read everytime you start apache. Since there was no read access to the /etc/group file and also no group information available for the SOA owner httpd service failed to start.

After granting read access to everyone, the services came up fine.

Above solution not only holds good to SOA but also to any Oracle product which uses apache viz., Oracle Apps, Oracle 10gAS.

cheers,
OraclePitStop.

Read Full Post | Make a Comment ( None so far )

How to change Java heap size in R12 to avoid java.lang.OutOfMemoryError: Java heap space error?

Posted on September 11, 2008. Filed under: Apps Technology Stack, Configuration, Diagnostics, Operating System, R12, Technology, TroubleShooting | Tags: , , , , |

Recently we had encountered a situation, wherein end users where not able to get the R12 login page and
it just hangs, but all the opmn processess were up and running – adopmnctl.sh status gives status as
alive for all the components viz., HTTP, oafm, oacore & forms. The same environment was available and
end users were able to access without any isuses 15 minutes before. This error happens only when
multiple people login to the same page and perform simillar activity like employee self service form
etc. So where and what exactly could be the problem.

This is how we approached and resolved the issue.

First we checked apache error log. The following error was reported in the error log.

==============================================================
[warn] [client IP Address ] oc4j_socket_recvfull timed out
[error] [client IP Address ] mod_oc4j: There is no oc4j process (for destination: application://oacore) available to service request.
[error] [client  IP Address] mod_oc4j: request to OC4J [servername:OC4J AJP Port Range for Oacore]
failed: Connect failed
==============================================================

Above error message does give us a hint that the problem is with oacore, but as i said earlier the oacore is alive according to opmn.

Next, we verified the oacore log file.

($INST_TOP/logs/ora/10.1.3/opmn/oacore_default_group_1/oacorestd.err)

Error Message in the file

=======================================================
Exception in thread “OC4JMonitorThread” java.lang.OutOfMemoryError: Java heap space
=======================================================

While checking the error in file, parallely we opened another window to see the CPU and memory usage and we could see one java process was taking more than 100% CPU. This process was spawned by the opmn -d process and the process id didnt match with the oacore process id. Hint: adopmnctl.sh status will give the status as well the process id. It looked like a end client java process.

At this stage we had 3 options.

1. Kill the java process which is consuming high cpu.
2. Bounce oacore component
3. Adjust java (jvm) memory parameters

Each of the above options has its own implications and advantages. To minimize the downtime we decided  to kill the java processes, and the moment we killed the java process, all the browsers which were hanging reported instantaneously Internal Server Error. This proved to be a bad decision.

So we moved to the next option, bouncing the oacore service which we are sure will resolve the issue  (temporarily) and it did as expected. Basically when you bounce the services all the existing
connections and processess will be released which results in more available memory when you re-start the services.

Ok, now we tackled the problem and provided a temporary solution but we need to find a long term solution. This is option 3, adjusting java memory size.

Steps to change the heap size.

First, you need to identify how much is the maximum heap size that you can set. Click here.

Once you had identified the maximum heap size, we need to change the configuration files to make it
permanent.

Step 1: Edit opmn.xml file.

Location : $INST_TOP/ora/10.1.3/opmn/conf/

Open opmn.xml

Search for string Xms or Xmx or module-id=”OC4J”

This search should lead you to below location

‘<‘process-type id=”oacore” module-id=”OC4J” status=”enabled” working-dir=”$ORACLE_HOME/j2ee/home”‘>’
‘<‘category id=”start-parameters”‘>’
‘<‘data id=”java-options” value=”-server -verbose:gc -Xmx512M -Xms128M ……]

The default value for Maximum (-Xmx) and Minimum (-Xms) heap sizes are 512M and 128M respectively.

Again here you have options, you can set both Xms and Xmx has the same value as Xmx if you feel all your sessoins require higher memory or set a lower value for Xms and the maximum value for Xmx. Dont forget to change the values under ‘<‘category id=”stop-parameters”‘>’

opmn.xml also contains jvm configurations for other components – oafm & forms.

Step 2: Edit oc4j.properties file.

Location : ($INST_TOP/ora/10.1.3/j2ee/oacore/config)

This step is optional since we had already made changes in opmn.xml but there is no harm in making the change here. This step will come handy for troubleshooting specific components of Oracle viz., configurator, iSupplier or any other option which heavily utilizes/consumes CPU/memory.

Search for string Xms or Xmx or wrapper.

Option 1: If you find any of the above parameters change the values corresponding to the value you had  mentioned in opmn.xml or even more than that, as long as you dont exceed the maximum heap size limit.

Option 2: If you DO NOT find any of the above parameters, then make an entry like this, under the heading “# Java Object Cache Configuration Parameters”

wrapper.bin.parameters=-Xms[Value]M -Xmx[Value]M -XX:NewSize=256M -XX:MaxNewSize=256M

Step 3: Edit Applications Context file

vi $CONTEXT_FILE

Location: $INST_TOP/appl/admin/SID_hostname.xml

search for string s_oacore_jvm_start_options

Change Xms and Xmx value. Repeat the same step for parameter s_oacore_jvm_stop_options.

Changes made in Step 3 will take effect the next time you run autoconfig, whereas Step 1 & 2 changes will take effect the next time you bounce opmn services, but the values are not permanent in the sense these will be wiped off next time you run autoconfig. Yes you can preserve the changes by placing it inbetween Begin and End customizations.

You can also increase the Garbage Collection threads parameter (-XX:ParallelGCThreads) to a higher value if you have JDK 1.5 or more than 2 cpus or more memory. For more information on this you can refer to Metalink Note: 362851 – Guidelines to setup the JVM in Apps Ebusiness Suite 11i and R12.

cheers,
oraclePitStop.

Read Full Post | Make a Comment ( 8 so far )

How to identify/determine the maximum JVM Heap Size for OC4j components in R12 / 10gAS

Posted on September 11, 2008. Filed under: Apps Technology Stack, Configuration, Operating System, R12, TroubleShooting | Tags: , , , , , , |

Below command will help you to identify the maximum memory heap size than can be allocated to a JVM
process.

java -mx[value]m -version

Start with a higher value like 4G or higher and slowly cut down to the maximum allowed value.

Sample

$ java -mx4096m -version

Invalid maximum heap size: -Xmx4096m
The specified size exceeds the maximum representable size.
Could not create the Java virtual machine.

As you can see my system cant support 4Gig for a JVM.

Now lets reduce to a lesser value, viz., 3Gig

$ java -mx3072m -version
java version “1.5.0.04”
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0.04-_27_jul_2006_10_52)
Java HotSpot(TM) Server VM (build 1.5.0.04 jinteg:07.27.06-16:18 IA64, mixed mode)

Now its clear that the maximum allowed value is 3gig.

cheers,
OraclePitStop.

Read Full Post | Make a Comment ( 3 so far )

How to submit a concurrent program from command / shell prompt

Posted on July 28, 2008. Filed under: DBA/Admin Tips, Operating System, R12 | Tags: , , , , , |

We can submit concurrent programs from OS command / shell prompt using CONCSUB utility. Heres the full length syntax:

CONCSUB <APPS username>/<APPS password> \
<responsibility application short name> \
<responsibility name> \
<username> \
[WAIT=N|Y|<n seconds>] \
CONCURRENT \
<program application short name> \
<program name> \
[PROGRAM_NAME=”<description>”] \
[REPEAT_TIME=<resubmission time>] \
[REPEAT_INTERVAL= <number>] \
[REPEAT_INTERVAL_UNIT=< resubmission unit>] \
[REPEAT_INTERVAL_TYPE=< resubmission type>] \
[REPEAT_END=<resubmission end date and time>] \
[NLS_LANGUAGE=<language of the request>] \
[NLS_TERRITORY=<territory of the request>] \
[START=<date>] \
[IMPLICIT=< type of concurrent request> \
[<parameter 1> … <parameter n>]

Examples:

Active Users

CONCSUB apps/[apps_pwd] SYSADMIN “System Administrator” SYSADMIN WAIT=N CONCURRENT FND FNDSCURS

Prints Environment Variables

CONCSUB apps/[apps_pwd] SYSADMIN “System Administrator” SYSADMIN WAIT=N CONCURRENT FND FNDPRNEV APPL_TOP

cheers,
OraclePitStop

Read Full Post | Make a Comment ( None so far )

Offtopic: Windows Vista Update failed with error 80070005

Posted on May 24, 2008. Filed under: Operating System, Windows | Tags: , , |

I have a Vista Home Premium edition and for no reason the windows update failed with error 80070005 one day. There was no new applications installed and i dont remember changing any settings and the worst part is i dont get to find a solution on the vendor website.

I tried all sort of options provided in this microsoft forum link but nothing worked.

Thanks to Aaron Stebner for the wonderful post. (http://blogs.msdn.com/astebner/archive/2006/09/04/739820.aspx) which just worked like a charm. I just followed the steps mentioned on this blog, and voila the error vanished.

cheers,
OraclePitStop

Read Full Post | Make a Comment ( 3 so far )

Liked it here?
Why not try sites on the blogroll...