Every once-in-a-while I'll be reviewing a logfile or trace file in search of one thing and I'll come across something else which doesn't look quite right. As the title of this blog suggests, I found a frequently repeating error in our development OEM's "emoms.trc" file, specifically:
[2018-08-23T17:26:18.819-04:00] [EMGC_OMS1] [ERROR:32] [] [oracle.sysman.core.pbs.receiver.AbstractOMSHandshake] [tid: [ACTIVE].ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: <anonymous>] [ecid: 0000MJPAUGO1JfV0U3J7Fs1RMaUD00000B,0:7540:1] [APP: empbs] [URI: /empbs/upload] OMSHandshake failed.(AGENT URL = https://<agent host>:3872/emd/main/)(ERROR = KEY_MISMATCH as Last AgentURL is null and the Agent Key doesn't exist in repos)
A few things caught my attention with this message. First, the error was from an agent on a server that I know has been decommissioned, or at least removed from OEM. Second, the error was coming in every minute, which to me seemed like a real waste since the agent shouldn't exist.
It turns out that not only did the server still exist but the management agent was up. Apparently in-between our team shutting down all Oracle processes plus removing targets from OEM and the final step of decommissioning the hardware, the server itself suffered a reboot which caused all processes to start automatically. From then on, the management agent tried every minute to upload information to the repository yet the repository couldn't find a match for the agent.
Shutting down the management agent stopped the flow of error messages to "emoms.trc" (this would be the domain server's file under "
