With OEM 12c came a feature that I've come to appreciate, the ability to discover all targets on a given environment. Part of my appreciation is the removal of the mundane - why should I tell and define for OEM what exists on an environment when I can make it discover this information for itself? But of course, with automation comes highlighting our own mistakes or possibly misunderstanding of our own assumptions.
We recently needed to discover a 2-node database, which was a copy of a separate database. Simple enough. The new database was "orcldb" with instances "orcldb1" and "orcldb2" on nodes 1 and 2, respectively. Yet the discovery results in OEM showed as "orcldb" and "orcldb2" on node2, "ORCLDB.us.dnb.com" and "ORCLDB.us.dnb.com_orcldb1" on node1. So not only 2 separate databases but a difference in character case along with the domain involved.
A bit of knowledge about what's involved in the discovery process helped identify the problem. When monitoring the discovery process I noticed that a process was forked with the following command:
$AGENT_HOME/perl/bin/perl $AGENT_BASE/plugins/oracle.sysman.db.discovery.plugin_12.1.0.7.0/discover/oracledb.pl
$AGENT_HOME <host being discovered>
Also, not only is some information written to $AGENT_INSTANCE_HOME/sysman/log/emagent_perl.trc but much more information is detailed in $AGENT_INSTANCE_HOME/sysman/emd/state/fetchlet_state/Discovery/oracle.sysman.db_discover_oracledb.pl.trc. A search on this trace file lead me to the following:
oracledb.pl: Fri Feb 15 17:57:48 2019: DB_LISTENER_DISCOVERY: SID=orcldb excluded from discovery as it does not have a running pmon process, "/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/lkORCLDB file or initialization file.
While I knew the database had both instances open, I noticed on node1, under $ORACLE_HOME/dbs, there was a leftover init.ora file from configuring the new database - initorcldb1.ora. This file was a full "pfile" and included "CLUSTER_DATABASE=FALSE". Bingo!
I removed the init.ora file (and made sure nothing similar was on node2 for this database), then validated "Spfile" was set properly in the OCR. Then I removed the discovered targets in OEM and tried again, this time the discovered targets being exactly what I expected.
One more item has been added to my notes on sources for the discovery process - $ORACLE_HOME/dbs/init*!
