OEM Target Discovery and One Potential Source of Duplicates

With OEM 12c came a feature that I've come to appreciate, the ability to discover all targets on a given environment.  Part of my appreciation is the removal of the mundane - why should I tell and define for OEM what exists on an environment when I can make it discover this information for itself?  But of course, with automation comes highlighting our own mistakes or possibly misunderstanding of our own assumptions.

We recently needed to discover a 2-node database, which was a copy of a separate database.  Simple enough.  The new database was "orcldb" with instances "orcldb1" and "orcldb2" on nodes 1 and 2, respectively.  Yet the discovery results in OEM showed as "orcldb" and "orcldb2" on node2, "ORCLDB.us.dnb.com" and "ORCLDB.us.dnb.com_orcldb1" on node1.  So not only 2 separate databases but a difference in character case along with the domain involved.

A bit of knowledge about what's involved in the discovery process helped identify the problem.  When monitoring the discovery process I noticed that a process was forked with the following command:

$AGENT_HOME/perl/bin/perl $AGENT_BASE/plugins/oracle.sysman.db.discovery.plugin_12.1.0.7.0/discover/oracledb.pl
  $AGENT_HOME <host being discovered>

Also, not only is some information written to $AGENT_INSTANCE_HOME/sysman/log/emagent_perl.trc but much more information is detailed in $AGENT_INSTANCE_HOME/sysman/emd/state/fetchlet_state/Discovery/oracle.sysman.db_discover_oracledb.pl.trc.  A search on this trace file lead me to the following:

oracledb.pl: Fri Feb 15 17:57:48 2019:  DB_LISTENER_DISCOVERY:  SID=orcldb excluded from discovery as it does not have a
running pmon process, "/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/lkORCLDB file or initialization file.

While I knew the database had both instances open, I noticed on node1, under $ORACLE_HOME/dbs, there was a leftover init.ora file from configuring the new database - initorcldb1.ora.  This file was a full "pfile" and included "CLUSTER_DATABASE=FALSE".  Bingo!

I removed the init.ora file (and made sure nothing similar was on node2 for this database), then validated "Spfile" was set properly in the OCR.  Then I removed the discovered targets in OEM and tried again, this time the discovered targets being exactly what I expected.

One more item has been added to my notes on sources for the discovery process - $ORACLE_HOME/dbs/init*!