This is something I’ve seen for a while now. It all started when I upgraded an Oracle Restart environment from 11.2.0.4 to 18.7. Suddenly our monitoring system alerted that the number of processes (at the linux level) crossed the threshold (500).
I looked at the process list (ps -ef) and saw many processes starting with Usm (more than 150). Here is a partial list:
root 2138 2 0 10:48 ? 00:00:00 [UsmGeneral]
root 2140 2 0 10:48 ? 00:00:00 [UsmMonitor]
root 2201 2 0 10:48 ? 00:00:00 [UsmRdBb-01]
root 2207 2 0 10:48 ? 00:00:00 [UsmWrBb-00]
root 2404 2 0 10:48 ? 00:00:00 [UsmRdBb-30]
root 2405 2 0 10:48 ? 00:00:00 [UsmRdBb-31]
root 2406 2 0 10:48 ? 00:00:00 [UsmRdBb]
root 2347 2 0 10:48 ? 00:00:00 [UsmWrBb-31]
root 2348 2 0 10:48 ? 00:00:00 [UsmWrRb-27]
root 2349 2 0 10:48 ? 00:00:00 [UsmWrBb]
There is not a lot of information about this so it took me some time to actually understand that these are Oracle processes. They are, and they are related to ACFS.
In 18.7 (the version we upgraded to) we simply restarted the server after the upgrade and got rid of these processes, so all was good. But then we patched the database to 18.9. With 18.9 these processes appeared again and didn’t go away after a restart.
I opened an SR and apparently we had ACFS installed and running (we didn’t use ACFS and didn’t install it, it probably got installed and configured automatically with the 18c upgrade).
To check if you have ACFS installed and configured (look at the highlighted lines):
[oracle@ora18 ~]$ lsmod | grep oracle
oracleacfs 5711611 0
oracleadvm 1127521 0
oracleoks 714819 2 oracleacfs,oracleadvm
oracleasm 53886 1
[oracle@ora18 ~]$ acfsdriverstate loaded
ACFS-9203: true
[oracle@ora18 ~]$ acfsdriverstate installed
ACFS-9203: true
[oracle@ora18 ~]$ acfsdriverstate supported
ACFS-9200: Supported
If you’re not using ACFS, you can simply deinstall it as described in MOS note 1371067.1 (requires downtime).
Before you start, Make sure you don’t have ACFS filesystem mounted. Then follow these steps (these are for Oracle Restart only, for RAC, check the note):
[root@ora18 ~]# export ORACLE_HOME=/oracle/software/18/grid
[root@ora18 ~]# $ORACLE_HOME/bin/crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ora18'
CRS-2673: Attempting to stop 'ora.ora18.db' on 'ora18'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'ora18'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'ora18' succeeded
CRS-2677: Stop of 'ora.ora18.db' on 'ora18' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'ora18'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'ora18'
CRS-2677: Stop of 'ora.DATA.dg' on 'ora18' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'ora18'
CRS-2677: Stop of 'ora.evmd' on 'ora18' succeeded
CRS-2677: Stop of 'ora.asm' on 'ora18' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'ora18'
CRS-2677: Stop of 'ora.cssd' on 'ora18' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ora18' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@ora18 ~]# $ORACLE_HOME/bin/acfsload stop
[root@ora18 ~]# lsmod | grep oracle
oracleasm 53886 1
[root@ora18 ~]# $ORACLE_HOME/bin/acfsroot uninstall
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
stop: unrecognized service
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
[root@ora18 ~]# $ORACLE_HOME/bin/crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[root@ora18 ~]# ps -ef|grep Usm
root 8109 3294 0 11:25 pts/0 00:00:00 grep Usm
Patching Issue
After I disabled ACFS on 18.9 I patched the environment to 18.10 and lo and behold, ACFS is back! Apparently RU brings ACFS as expected behavior.
Conclusion – after you patch the environment you’ll have to disable ACFS again. Oracle support said that this is expected, but it doesn’t make sense to me. I will continue checking this subject.
This remains me of Doc 2441164.1: How To – Disable Docker Process and docker0 Interface on Exadata If Not Needed
The Docker application was added to Exadata images beginning with Exadata image releases in 2018. The services and related docker0 network interface are enabled by default.
This document details how to disable Docker and the docker0 interface if it is not required in the customer’s Exadata environment or if the docker0 interface is conflicting with other network interfaces/routes.
============================================================
So, if you upgrade and have same network “somewhere”.. you will have good time trying to see why suddenly something is failing… 🙂