GotoDBA Features,Infrastructure,Troubleshooting Many Usm Processes in Oracle Restart

Many Usm Processes in Oracle Restart

This is something I’ve seen for a while now. It all started when I upgraded an Oracle Restart environment from 11.2.0.4 to 18.7. Suddenly our monitoring system alerted that the number of processes (at the linux level) crossed the threshold (500).

I looked at the process list (ps -ef) and saw many processes starting with Usm (more than 150). Here is a partial list:

root      2138     2  0 10:48 ?        00:00:00 [UsmGeneral]
root      2140     2  0 10:48 ?        00:00:00 [UsmMonitor]
root      2201     2  0 10:48 ?        00:00:00 [UsmRdBb-01]
root      2207     2  0 10:48 ?        00:00:00 [UsmWrBb-00]
root      2404     2  0 10:48 ?        00:00:00 [UsmRdBb-30]
root      2405     2  0 10:48 ?        00:00:00 [UsmRdBb-31]
root      2406     2  0 10:48 ?        00:00:00 [UsmRdBb]
root      2347     2  0 10:48 ?        00:00:00 [UsmWrBb-31]
root      2348     2  0 10:48 ?        00:00:00 [UsmWrRb-27]
root      2349     2  0 10:48 ?        00:00:00 [UsmWrBb]

There is not a lot of information about this so it took me some time to actually understand that these are Oracle processes. They are, and they are related to ACFS.

In 18.7 (the version we upgraded to) we simply restarted the server after the upgrade and got rid of these processes, so all was good. But then we patched the database to 18.9. With 18.9 these processes appeared again and didn’t go away after a restart.

I opened an SR and apparently we had ACFS installed and running (we didn’t use ACFS and didn’t install it, it probably got installed and configured automatically with the 18c upgrade).

To check if you have ACFS installed and configured (look at the highlighted lines):

[oracle@ora18 ~]$ lsmod | grep oracle
oracleacfs           5711611  0
oracleadvm           1127521  0
oracleoks             714819  2 oracleacfs,oracleadvm
oracleasm              53886  1
[oracle@ora18 ~]$ acfsdriverstate loaded
ACFS-9203: true
[oracle@ora18 ~]$ acfsdriverstate installed
ACFS-9203: true
[oracle@ora18 ~]$ acfsdriverstate supported
ACFS-9200: Supported

If you’re not using ACFS, you can simply deinstall it as described in MOS note 1371067.1 (requires downtime).

Before you start, Make sure you don’t have ACFS filesystem mounted. Then follow these steps (these are for Oracle Restart only, for RAC, check the note):

[root@ora18 ~]# export ORACLE_HOME=/oracle/software/18/grid

[root@ora18 ~]# $ORACLE_HOME/bin/crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ora18'
CRS-2673: Attempting to stop 'ora.ora18.db' on 'ora18'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'ora18'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'ora18' succeeded
CRS-2677: Stop of 'ora.ora18.db' on 'ora18' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'ora18'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'ora18'
CRS-2677: Stop of 'ora.DATA.dg' on 'ora18' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'ora18'
CRS-2677: Stop of 'ora.evmd' on 'ora18' succeeded
CRS-2677: Stop of 'ora.asm' on 'ora18' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'ora18'
CRS-2677: Stop of 'ora.cssd' on 'ora18' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ora18' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@ora18 ~]# $ORACLE_HOME/bin/acfsload stop

[root@ora18 ~]# lsmod | grep oracle
oracleasm              53886  1

[root@ora18 ~]# $ORACLE_HOME/bin/acfsroot uninstall
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
stop: unrecognized service
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf

[root@ora18 ~]# $ORACLE_HOME/bin/crsctl start has
CRS-4123: Oracle High Availability Services has been started.

[root@ora18 ~]# ps -ef|grep Usm
root      8109  3294  0 11:25 pts/0    00:00:00 grep Usm

Patching Issue

After I disabled ACFS on 18.9 I patched the environment to 18.10 and lo and behold, ACFS is back! Apparently RU brings ACFS as expected behavior.

Conclusion – after you patch the environment you’ll have to disable ACFS again. Oracle support said that this is expected, but it doesn’t make sense to me. I will continue checking this subject.

1 thought on “Many Usm Processes in Oracle Restart”

  1. This remains me of Doc 2441164.1: How To – Disable Docker Process and docker0 Interface on Exadata If Not Needed

    The Docker application was added to Exadata images beginning with Exadata image releases in 2018. The services and related docker0 network interface are enabled by default.

    This document details how to disable Docker and the docker0 interface if it is not required in the customer’s Exadata environment or if the docker0 interface is conflicting with other network interfaces/routes.

    ============================================================

    So, if you upgrade and have same network “somewhere”.. you will have good time trying to see why suddenly something is failing… 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Post