The other day while reviewing an instance's alert log I
found an interesting message:
WARNING:Oracle instance running on a system with low
open file descriptor
limit.
Tune your system to increase this limit to avoid
severe
performance degradation.
A quick search of MOS pointed me to bug 5862719. It turns out on our RAC system (10.2.0.2) on
RHEL 4, the /etc/init.d/init.crsd file has the command "ulimit -n
unlimited" in it, which fails and drops the file descriptor limit to its
default, which in our case is 1024.
% ps -ef | grep smon_dchdb
oracle
20720 1 0 Oct31 ? 00:00:16 ora_smon_dchdb1
% grep "open files" /proc/20720/limits
Max open files 1024 1024 files
The system recently crashed (which was why I was checking
the alert log) and so all auto-started Oracle processes inherited their ulimits
from the init processes.
My point of this isn't to re-document an Oracle bug but
emphasize the importance of taking a step back periodically and thinking things
through. Is an FD limit of 1024
potentially hurting our system, performance-wise? If so, can I duplicate this elsewhere to
prove the impact, justifying an emergency fix?
On Linux it's pretty easy to check on open file counts for
processes running under the same account.
As "oracle" I issued the following:
#------------------------------------------------------------------------------------|
# Check all "oracle" processes associated
with database "dchdb" and ASM, grab the PID
# and current command, then count how many files are
opened and display the results.
#------------------------------------------------------------------------------------|
ps -f -u oracle | grep -E '(dchdb1|ASM)' | awk
'{print $2" "$8}' | \
while read
IN_PID IN_CMD
do
echo
"Files opened for PID $IN_PID $IN_CMD: `ls -1 /proc/${IN_PID}/fd 2>/dev/null
| wc -l`"
done | sort
-nr -k7,7 | head -6
#-- output --#
Files opened for PID 21810 ora_rbal_dchdb1: 28
Files opened for PID 21790 ora_asmb_dchdb1: 28
Files opened for PID 22080 ora_arc1_dchdb1: 27
Files opened for PID 22174 ora_qmnc_dchdb1: 26
Files opened for PID 22078 ora_arc0_dchdb1: 26
Files opened for PID 20859 ora_lck0_dchdb1: 24
No process has more than 28 files open, so a limit of 1024
seems pretty safe. Yet the database has
a few hundred datafiles, so why isn't the number higher? This is where ASM comes in to play. ASM bypasses the file system so database
"files" aren't managed by the OS and FD's, so for the most part FD
limits have a different context when dealing with ASM.
Result? While I believe
it's not always wise to leave code as is even though 1 or more commands/parts
are failing each time, in this case spending a bit of time to think things
through allowed this change to be scheduled on a lower priority.
