Oracle: Issues when patching Cloud Control 13.5 from RU25 to RU27

Patching Oracle Enterprise Manager Cloud Control - ore shortly Cloud Control - is and was always a challenge. Especially when Cloud Control is running on Windows (don't do that!). But I must admit that the patching process was always improved during the last years.

Preparation

Before pathing begins, Cloud Control has to be stopped, but Admin Server has to be running. You can achieve this by

# emctl stop oms

which stopps the Cloud Control application, but leaves the Admin Server online.

Or, if you had already stopped Cloud Control before, i. e. because you patched the repository database, you can start the Admin Server, but leave Cloud Control offline:

# emctl start oms -admin_only

Since an early RU (like RU03 I guess), Cloud Control supports the Rapid Platform Update:

• omspatcher deploy creates a new Cloud Control home, which is than patched to the latest Release Update
• omspatcher apply stops Cloud Control and starts it from the new home

This leads to shorter downtimes, but needs some additional housekeeping, as the old Cloud Control home needs to be deinstalled after pathing succeeded.

In my case, I use traditional patching, as this is my customer's wish - he doesn't want to try something new 😄

Middleware Home not found

But sometimes, still errors occur - like this one:

$ $ORACLE_HOME/OMSPatcher/omspatcher apply -oh $ORACLE_HOME
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation. All rights reserved.


OMSPatcher version : 13.9.5.26.0
OUI version : 13.9.4.0.0
Running from : /oracle/middleware
Log file location : /oracle/middleware/cfgtoollogs/omspatcher/opatch2025-07-31_13-31-46PM_1.log

OMSPatcher log file: /oracle/middleware/cfgtoollogs/omspatcher/37784425/omspatcher_2025-07-31_13-36-03PM_apply.log

Middleware Home not found.
Please enter OMS weblogic admin server URL():>

First, I thought that I downloaded the wrong OMSPatcher.

But newly downloading it, and unzipping it to the Middleware home, the same error occurred.

After creating an Oracle Service Request (SR), they helped me quickly - and found out, that a file called domain-registry.xml was missing.

The XML file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<domain-registry xmlns="http://xmlns.oracle.com/weblogic/domain-registry">
  <domain location="/oracle/gc_inst/user_projects/domains/GCDomain"/>
</domain-registry>

So it's easy to recover this file - just change the entry "domain location" to the path of GCDomain.

OMSPatcher failed to establish JMX connection to weblogic server

Then, I restarted the pathing process:

$ $ORACLE_HOME/OMSPatcher/omspatcher apply
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation.  All rights reserved.

OMSPatcher version : 13.9.5.26.0
OUI version        : 13.9.4.0.0
Running from       : /oracle/middleware
Log file location  : /oracle/middleware/cfgtoollogs/omspatcher/opatch2025-08-01_06-21-18AM_1.log

OMSPatcher log file: /oracle/middleware/cfgtoollogs/omspatcher/37784425/omspatcher_2025-08-01_06-25-24AM_apply.log

Please enter OMS weblogic admin server URL(t3s://<server name>:<admin server port>):>
Please enter OMS weblogic admin server username(weblogic):>
Please enter OMS weblogic admin server password:>

OMSPatcher failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server might be down.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OMSPatcher.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/middleware/OMSPatcher/omspatcher apply -invPtrLoc /oracle/middleware/oraInst.loc  OMSPatcher.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.

Log file location: /oracle/middleware/cfgtoollogs/omspatcher/37784425/omspatcher_2025-08-01_06-25-24AM_apply.log

Recommended actions: Please verify the error details and try again.

OMSPatcher failed with error code 231

Checking the credentials of weblogic is quite easy: just open the URL for the admin server in a browser. The URL would look like this:

https://<server name>:<admin server port>/console

The website will look like this:

If this works, your credentials are ok.

So, let's see what the error messages say:

3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OMSPatcher.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/middleware/OMSPatcher/omspatcher apply -invPtrLoc /oracle/middleware/oraInst.loc  OMSPatcher.OMS_DISABLE_HOST_CHECK=true)

Maybe, the Cloud Control remembers the last patching process, which was to RU25 on the old server.

Give it a try:

$ $ORACLE_HOME/OMSPatcher/omspatcher apply -invPtrLoc /orahome/middleware135/oraInst.loc OMSPatcher.OMS_DISABLE_HOST_CHECK=true

Now, it looks even better:

$ $ORACLE_HOME/OMSPatcher/omspatcher resume -invPtrLoc /oracle/middleware/oraInst.loc  OMSPatcher.OMS_DISABLE_HOST_CHECK=true
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


OMSPatcher version : 13.9.5.26.0
OUI version        : 13.9.4.0.0
Running from       : /oracle/middleware
Log file location  : /oracle/middleware/cfgtoollogs/omspatcher/opatch2025-08-01_13-28-41PM_1.log

OMSPatcher log file: /oracle/middleware/cfgtoollogs/omspatcher/SystemPatch/omspatcher_2025-08-01_08-25-18AM_resume.log

[Info:Previous action was performed by user:'sys']
Enter 'sys' password :
Enter SYSMAN Password :
Inside the execute Resume cmd method
command echo /oracle/tmp/patch/37784425/37736763 >> /oracle/middleware/.phBaseFile2025-08-01_10-32-14AM.txt
 The filelist is /oracle/middleware/.phBaseFile2025-08-01_10-32-14AM.txt
command echo /oracle/tmp/patch/37784425/37736768 >> /oracle/middleware/.phBaseFile2025-08-01_10-32-14AM.txt
 The filelist is /oracle/middleware/.phBaseFile2025-08-01_10-32-14AM.txt
command echo /oracle/tmp/patch/37784425/37736785 >> /oracle/middleware/.phBaseFile2025-08-01_10-32-14AM.txt
 The filelist is /oracle/middleware/.phBaseFile2025-08-01_10-32-14AM.txt
command echo /oracle/tmp/patch/37784425/37736831 >> /oracle/middleware/.phBaseFile2025-08-01_10-32-14AM.txt
 The filelist is /oracle/middleware/.phBaseFile2025-08-01_10-32-14AM.txt
command echo /oracle/tmp/patch/37784425/37736852 >> /command echo /oracle/tmp/patch/37784425/37736852 >> /orahome/middleware135/.phBaseFile2025-08-01_10-32-14AM.txt/middleware/.phBaseFile2025-08-01_10-32-14AM.txt
 The filelist is /oracle/middleware/.phBaseFile2025-08-01_10-32-14AM.txt
command /oracle/middleware/OPatch/opatch napply -phBaseFile /oracle/middleware/.phBaseFile2025-08-01_10-32-14AM.txt -invPtrLoc /oracle/middleware/oraInst.loc -oh /oracle/middleware -silent

File permissions

After many minutes, OMSPatcher stopped again:

OMSPatcher failed to execute some of the patching steps. Please check the Patching summary,individual logs and try to resolve the issue. Once the issue is resolved,Please execute below command to complete patching session:
                                omspatcher resume

In the log files, I found out that the patching process unsuccessfully tried to copy files from the patch folder to the Middleware home:

[Aug 1, 2025 8:21:56 AM]     Exception Occured while executiong Apply Operation: ApplySession failed in system modification phase... 'ApplySession::apply failed: Copy failed from '/oracle/tmp/37784425/37736785/files/oracle.sysman.db.oms.plugin/13.5.1.0.0/em.plugin.common.symbol/oracle.sysman.db.oms.plugin_13.5.1.0.0/patched_metadata/13.5.1.0.0/default_collection/135100/cloud_db_common_metrics.xmlp' to '/oracle/middleware/plugins/oracle.sysman.db.oms.plugin_13.5.1.0.0/patched_metadata/13.5.1.0.0/default_collection/135100/cloud_db_common_metrics.xmlp'...
                             Copy failed from '/oracle/tmp/37784425/37736785/files/oracle.sysman.db.oms.plugin/13.5.1.0.0/em.plugin.common.symbol/oracle.sysman.db.oms.plugin_13.5.1.0.0/patched_metadata/13.5.1.0.0/default_collection/135100/cloud_schema_storage_metrics.xmlp' to '/oracle/middleware/plugins/oracle.sysman.db.oms.plugin_13.5.1.0.0/patched_metadata/13.5.1.0.0/default_collection/135100/cloud_schema_storage_metrics.xmlp'...
                             Copy failed from '/oracle/tmp/37784425/37736785/files/oracle.sysman.db.oms.plugin/13.5.1.0.0/em.plugin.common.symbol/oracle.sysman.db.oms.plugin_13.5.1.0.0/patched_metadata/13.5.1.0.0/default_collection/135100/cluster.xml' to '/oracle/middleware/plugins/oracle.sysman.db.oms.plugin_13.5.1.0.0/patched_metadata/13.5.1.0.0/default_collection/135100/cluster.xml'...

I tried to do an ls to this folder, but I got that:

[...]
ls: cannot access '/orahome/middleware135/plugins/oracle.sysman.db.oms.plugin_13.5.1.0.0/patched_metadata/13.5.1.0.0/default_collection/135100/gds_gsm.xml': Permission denied
ls: cannot access '/orahome/middleware135/plugins/oracle.sysman.db.oms.plugin_13.5.1.0.0/patched_metadata/13.5.1.0.0/default_collection/135100/database.xmlp': Permission denied
ls: cannot access '/orahome/middleware135/plugins/oracle.sysman.db.oms.plugin_13.5.1.0.0/patched_metadata/13.5.1.0.0/default_collection/135100/rac_database.xml': Permission denied
total 0
-????????? ? ? ? ?            ? rac_database.xml
-????????? ? ? ? ?            ? osm_proxy.xml
-????????? ? ? ? ?            ? osm_ioserver.xml
-????????? ? ? ? ?            ? osm_instance.xml
-????????? ? ? ? ?            ? osm_common.xmlp
-????????? ? ? ? ?            ? osm_cluster.xml
-????????? ? ? ? ?            ? oracle_pdb.xml
-????????? ? ? ? ?            ? oracle_listener.xml
[...]

And when I tried an ls in the parent folder, I got that:

[...]
drw-r--r--.  2 oracle oinstall 4.0K Apr  9 13:16 135100
[...]

The execution bit was missing - so no cd into that folder, nor copying files into that!

I tried to restart the pathing process, but after a few minutes, it failed again. Why?

In another logfile, I found that executing rcu failed. I found it here:

/oracle/middleware/oracle_common/bin/rcu

And again, execution bits were missing. I fixed it by

# chmod +x /orahome/middleware135/oracle_common/bin/rcu

After that, patches were successfully applied.

Conclusion

I don't know how all these errors were produced - but I can imagine that it was due to the fact, that I moved Cloud Control from an old server running RedHat 7 to a newer one, running RedHat 9. I did that using the procedures and documentations provided by Oracle, but they still seem to have errors.

I think, there will again be errors due to that in future, until I upgraded to Cloud Control 24ai...