I wrote lately a couple of post about upgrading to 18c (why research is important and OCR and Voting disks on 18c and 19c). These posts were the result of an upgrade planning for a client where the database resides on NFS. After I finished the research and prepared a plan, it was time for testing.
Creating ASM
As this is Oracle 12.1.0.2 on NFS, the OCR and Voting disks were on NFS. In 18c this is not allowed, so I had to create ASM. This is how you do it:
Make sure your NFS mount is configured with the following settings:
rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0
Under this mount (let’s call it /oracle/nfs), create a directory and file(s) for ASM disks (this is one file of 2GB, but I’ll talk about space later):
root$ mkdir /oracle/nfs/asm
root$ dd if=/dev/zero of=/oracle/nfs/asm/asmdisk1 bs=1024k count=2000 oflag=direct
root$ chown -R grid:asmadmin /oracle/nfs/asm/asmdisk1
root$ chmod 660 /oracle/nfs/asm/asmdisk1
Now connect as grid and start asmca from the 12.1 home. This will start the ASM instances on the nodes. Then change the parameter “ASM_DISKSTRING” to /oracle/nfs/asm and you’ll be able to create a diskgroup on the created empty file(s). In this case I created the group +GRID.
Moving OCR and Voting Disk
When the ASM is up and the diskgroup is ready, we need to move the OCR to the ASM (as root):
root$ $GRID_HOME/bin/ocrconfig -add +GRID
root$ $GRID_HOME/bin/ocrconfig -delete /oracle/nfs/ocr
root$ $GRID_HOME/bin/ocrcheck
And then move the Voting disk (as grid):
grid$ $GRID_HOME/bin/crsctl replace votedisk +GRID
grid$ $GRID_HOME/bin/crsctl query css votedisk
MGMTDB Problem
One thing that I didn’t consider is the MGMTDB. In 12.1 I have it on NFS, just like the other databases, but when the upgrade runs it drops the existing MGMTDB and creates a new one. The GUI doesn’t let you choose location for the MGMTDB, and it seems that it creates it automatically on the ASM in the same diskgroup as the OCR and Voting disk (at least in my case it did).
In my case, I created the +GRID diskgroup with 2GB of space, which is not enough for MGMTDB. According to the documentation, the OCR and Voting disk DG needs 1GB of space (for external redundancy). The MGMTDB DG need 28GB (for up to 4 nodes).
When I executed the upgrade everything went well including the rootupgrade.sh scripts on both nodes. During the configuration step, Oracle drops the existing MDMGT, creates a new CDB (step called “Creating Oracle Grid Infrastructure Management Repository”) and then created the MGMTDB PDB (a step called “Grid Infrastructure Management Repository Configuration Assistant”) and this step failed (as it tries to create the relevant tablespaces and I didn’t have enough space). The funny thing is that it doesn’t fail on the tablespace creation, but on the next step when it tries to use the CHM user and gets “ORA-01917: user or role ‘CHM’ does not exist”.
MGMTDB Solution
The solution here is to create the CDB manually and then retry the Configuration Assistant in the GUI.
When you create the MGMTDB CDB manually, you can actually choose the location and place it wherever you’d like (including NFS, it is supported in 18c):
# run this as root on all nodes
root$ $GRID_HOME/bin/crsctl stop res ora.crf -init
root$ $GRID_HOME/bin/crsctl modify res ora.crf -attr ENABLED=0 -init
# run this as grid on the node that holds the MGMTDB resource
grid$ $GRID_HOME/bin/dbca -silent -deleteDatabase -sourceDB -MGMTDB
grid$ $GRID_HOME/bin/dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType FS -datafileDestination /oracle/nfs/mgmtdb -datafileJarLocation $ORACLE_HOME/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck
# run this as root on all nodes
root$ $GRID_HOME/bin/crsctl modify res ora.crf -attr ENABLED=1 -init
root$ $GRID_HOME/bin/crsctl start res ora.crf -init
Now when you click “retry” in the GUI, the mgmtca will connect to the existing database and will create everything in the same location (NFS in my case). You can, of course, create a different ASM DG (or add disks to the existing +GRID one) and create the MGMTDB there (Oracle recommends to separate the OCR and Voting from the MGMTDB, I don’t know why the upgrade GUI doesn’t let you choose location for the MGMTDB).