SEHA is a new Oracle Database 19c (>=19.7) feature to allow Standard Edition to integrate with Grid Infrastructure (GI) and allow a simple fail-over cluster option (after Oracle has removed the RAC option from Standard Edition). You can read more about what it is in my introduction to SEHA blog post, and more about how it actually works under different circumstances in my testing SEHA post. In this post it’s time to upgrade SEHA.
Introduction
As I explained in my first post about SEHA, for SEHA to work you’ll need Oracle 19.7 or later and I installed 19.8. So in my environment the GI and DB are 19.8 and last week 19.9 was released. What interested me is how the patch works: Will opatchauto patch the Standard Edition home as well? Will it add the OCW patch? How about running the sqlpatch scripts? And in general, what is the best way to patch this?
The Patch Process
In my case, I have 2 servers: SEHA1 and SEHA2, and 2 databases: se1 (SEHA-enabled, currently on SEHA1) and se2 (SEHA-disabled, runnning on SEHA2). This is how I’m going to patch the entire environment:
- Patch SEHA2 (se2 will be down during the patch, se1 will be on SEHA1)
- Move se1 to SEHA2 and patch it
- Patch SEHA1
Patch Node SEHA2
Initial status:
[oracle@seha2 ~]$ srvctl status database -db se1
Instance se1 is running on node seha1
[oracle@seha2 ~]$ srvctl status database -db se2
Instance se2 is running on node seha2
Installing the patch:
[root@seha2 ~]# export ORACLE_HOME=/oracle/grid/19
[root@seha2 ~]# $ORACLE_HOME/OPatch/opatchauto apply /oracle/install/31750108
OPatchauto session is initiated at Wed Oct 28 14:39:46 2020
System initialization log file is /oracle/grid/19/cfgtoollogs/opatchautodb/systemconfig2020-10-28_02-39-52PM.log.
Session log file is /oracle/grid/19/cfgtoollogs/opatchauto/opatchauto2020-10-28_02-40-41PM.log
The id for this session is U8ZB
Executing OPatch prereq operations to verify patch applicability on home /oracle/grid/19
Executing OPatch prereq operations to verify patch applicability on home /oracle/db/19
Patch applicability verified successfully on home /oracle/db/19
Patch applicability verified successfully on home /oracle/grid/19
Verifying SQL patch applicability on home /oracle/db/19
SQL patch applicability verified successfully on home /oracle/db/19
Preparing to bring down database service on home /oracle/db/19
Successfully prepared home /oracle/db/19 to bring down database service
Bringing down database service on home /oracle/db/19
Following database has been stopped and will be restarted later during the session: se2
Database service successfully brought down on home /oracle/db/19
Bringing down CRS service on home /oracle/grid/19
CRS service brought down successfully on home /oracle/grid/19
Start applying binary patch on home /oracle/db/19
Binary patch applied successfully on home /oracle/db/19
Start applying binary patch on home /oracle/grid/19
Binary patch applied successfully on home /oracle/grid/19
Starting CRS service on home /oracle/grid/19
CRS service started successfully on home /oracle/grid/19
Starting database service on home /oracle/db/19
Database service successfully started on home /oracle/db/19
Preparing home /oracle/db/19 after database service restarted
No step execution required.........
Trying to apply SQL patch on home /oracle/db/19
SQL patch applied successfully on home /oracle/db/19
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:seha2
SIDB Home:/oracle/db/19
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /oracle/install/31750108/31773437
Reason: This patch is not applicable to this specified target type - "oracle_database"
Patch: /oracle/install/31750108/31780966
Reason: This patch is not applicable to this specified target type - "oracle_database"
==Following patches were SUCCESSFULLY applied:
Patch: /oracle/install/31750108/31771877
Log: /oracle/db/19/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-28_14-45-37PM_1.log
Patch: /oracle/install/31750108/31772784
Log: /oracle/db/19/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-28_14-45-37PM_1.log
Host:seha2
CRS Home:/oracle/grid/19
Version:19.0.0.0.0
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /oracle/install/31750108/31771877
Log: /oracle/grid/19/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-28_15-01-38PM_1.log
Patch: /oracle/install/31750108/31772784
Log: /oracle/grid/19/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-28_15-01-38PM_1.log
Patch: /oracle/install/31750108/31773437
Log: /oracle/grid/19/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-28_15-01-38PM_1.log
Patch: /oracle/install/31750108/31780966
Log: /oracle/grid/19/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-28_15-01-38PM_1.log
OPatchauto session completed at Wed Oct 28 15:41:00 2020
Time taken to complete the session 61 minutes, 15 seconds
Status After SEHA2 Patch
After we finished patching the node, let’s see what the status of the databases is:
[oracle@seha1 ~]$ srvctl status database -db se1
Instance se1 is running on node seha1
[oracle@seha1 ~]$ srvctl status database -db se2
Instance se2 is running on node seha2
# on SEHA1
[oracle@seha1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
[oracle@seha1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Oct 28 15:44:09 2020
Version 19.8.0.0.0
Copyright (c) 1982, 2020, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Standard Edition 2 Release 19.0.0.0.0 - Production
Version 19.8.0.0.0
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
se1
SQL> select comp_id,version_full from dba_registry where comp_id='CATALOG';
COMP_ID VERSION_FULL
------------------------------ ------------------------------
CATALOG 19.8.0.0.0
# on SEHA2
[oracle@seha2 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)
31771877;Database Release Update : 19.9.0.0.201020 (31771877)
OPatch succeeded.
[oracle@seha2 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Oct 28 15:47:19 2020
Version 19.9.0.0.0
Copyright (c) 1982, 2020, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Standard Edition 2 Release 19.0.0.0.0 - Production
Version 19.9.0.0.0
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
se2
SQL> select comp_id,version_full from dba_registry where comp_id='CATALOG';
COMP_ID VERSION_FULL
------------------------------ ------------------------------
CATALOG 19.9.0.0.0
We can see here a few things:
- The databases are still located on the same nodes
- se1 database is 19.8 and wasn’t affected by SEHA2 patch
- se2 was upgraded to 19.9
- SEHA2 now has 19.9 RU and 19.9 OCW. This is great because now we can use srvctl from the DB home (and if you have no idea what I’m talking about, check my introduction to SEHA post)
Relocate and Patch DB se1
After we patched the SEHA2 node without downtime to the SEHA database se1, it’s time to patch se1. I’ll do this by moving it to the patched node (minimum downtime) and use datapach. Note that datapatch can be executed online, so basically we have a very short downtime during the relocate itself and that’s it.
I will do everything from SEHA2 node:
[oracle@seha2 ~]$ srvctl relocate database -db se1 -node seha2
[oracle@seha2 ~]$ srvctl status database -db se1
Instance se1 is running on node seha2
[oracle@seha2 OPatch]$ ./datapatch -verbose -db se1
SQL Patching tool version 19.9.0.0.0 Production on Wed Oct 28 15:58:58 2020
Copyright (c) 2012, 2020, Oracle. All rights reserved.
Log file for this invocation: /oracle/db/cfgtoollogs/sqlpatch/sqlpatch_23299_2020_10_28_15_58_58/sqlpatch_invocation.log
Connecting to database...OK
Gathering database info...done
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of interim SQL patches:
No interim patches found
Current state of release update SQL patches:
Binary registry:
19.9.0.0.0 Release_Update 200930183249: Installed
PDB CDB$ROOT:
Applied 19.8.0.0.0 Release_Update 200703031501 successfully on 27-OCT-20 11.14.43.728061 PM
PDB PDB:
Applied 19.8.0.0.0 Release_Update 200703031501 successfully on 27-OCT-20 11.14.47.991871 PM
PDB PDB$SEED:
Applied 19.8.0.0.0 Release_Update 200703031501 successfully on 27-OCT-20 11.14.47.991871 PM
Adding patches to installation queue and performing prereq checks...done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED PDB
No interim patches need to be rolled back
Patch 31771877 (Database Release Update : 19.9.0.0.201020 (31771877)):
Apply from 19.8.0.0.0 Release_Update 200703031501 to 19.9.0.0.0 Release_Update 200930183249
No interim patches need to be applied
Installing patches...
Patch installation complete. Total patches installed: 3
Validating logfiles...done
Patch 31771877 apply (pdb CDB$ROOT): SUCCESS
logfile: /oracle/db/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply_SE1_CDBROOT_2020Oct28_16_01_06.log (no errors)
Patch 31771877 apply (pdb PDB$SEED): SUCCESS
logfile: /oracle/db/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply_SE1_PDBSEED_2020Oct28_16_02_53.log (no errors)
Patch 31771877 apply (pdb PDB): SUCCESS
logfile: /oracle/db/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply_SE1_PDB_2020Oct28_16_02_53.log (no errors)
SQL Patching tool complete on Wed Oct 28 16:05:32 2020
[oracle@seha2 OPatch]$ export ORACLE_SID=se1
[oracle@seha2 OPatch]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Oct 28 16:05:42 2020
Version 19.9.0.0.0
Copyright (c) 1982, 2020, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Standard Edition 2 Release 19.0.0.0.0 - Production
Version 19.9.0.0.0
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
se1
SQL> select comp_id,version_full from dba_registry where comp_id='CATALOG';
COMP_ID VERSION_FULL
------------------------------ ------------------------------
CATALOG 19.9.0.0.0
Patch Node SEHA1
Now both se1 and se2 databases are on SEHA2 node and both are 19.9. All we’ve got to do left is to patch SEHA1 node:
[root@seha1 ~]# export ORACLE_HOME=/oracle/grid/19
[root@seha1 ~]# $ORACLE_HOME/OPatch/opatchauto apply /oracle/install/31750108
OPatchauto session is initiated at Wed Oct 28 16:07:22 2020
System initialization log file is /oracle/grid/19/cfgtoollogs/opatchautodb/systemconfig2020-10-28_04-07-29PM.log.
Session log file is /oracle/grid/19/cfgtoollogs/opatchauto/opatchauto2020-10-28_04-08-13PM.log
The id for this session is MR9L
Executing OPatch prereq operations to verify patch applicability on home /oracle/grid/19
Patch applicability verified successfully on home /oracle/grid/19
Bringing down CRS service on home /oracle/grid/19
CRS service brought down successfully on home /oracle/grid/19
Start applying binary patch on home /oracle/grid/19
Binary patch applied successfully on home /oracle/grid/19
Starting CRS service on home /oracle/grid/19
CRS service started successfully on home /oracle/grid/19
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:seha1
CRS Home:/oracle/grid/19
Version:19.0.0.0.0
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /oracle/install/31750108/31771877
Log: /oracle/grid/19/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-28_16-12-28PM_1.log
Patch: /oracle/install/31750108/31772784
Log: /oracle/grid/19/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-28_16-12-28PM_1.log
Patch: /oracle/install/31750108/31773437
Log: /oracle/grid/19/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-28_16-12-28PM_1.log
Patch: /oracle/install/31750108/31780966
Log: /oracle/grid/19/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-28_16-12-28PM_1.log
OPatchauto session completed at Wed Oct 28 16:51:13 2020
Time taken to complete the session 43 minutes, 51 seconds
Status After SEHA1 Patch
Let’s check the DB home:
[oracle@seha1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
OPatch succeeded.
Wait, what? How come it’s not patched?
I don’t know if you’ve noticed, but if you look now at the patch process, you’ll see that only the GI was patched and not the database. But we didn’t have this problem on SEHA2, so why here?
When patching Oracle software, opatchauto will patch the GI and ACTIVE DB homes. This means that only homes that have running databases will be patched. On SEHA1 we don’t have any running databases, so opatchauto will not patch any DB home. If you remember, on SEHA2, because of my specific configuration, we did have the se2 database up, so its home was patched.
If this is the case, we are not done, we still have to patch the DB home on SEHA1:
[root@seha1 ~]# export ORACLE_HOME=/oracle/db/19
[root@seha1 ~]# $ORACLE_HOME/OPatch/opatchauto apply /oracle/install/31750108 -oh /oracle/db/19
OPatchauto session is initiated at Wed Oct 28 17:08:20 2020
System initialization log file is /oracle/db/19/cfgtoollogs/opatchautodb/systemconfig2020-10-28_05-08-30PM.log.
Session log file is /oracle/db/19/cfgtoollogs/opatchauto/opatchauto2020-10-28_05-09-58PM.log
The id for this session is ICNW
Executing OPatch prereq operations to verify patch applicability on home /oracle/db/19
Patch applicability verified successfully on home /oracle/db/19
Verifying SQL patch applicability on home /oracle/db/19
No step execution required.........
Preparing to bring down database service on home /oracle/db/19
No step execution required.........
Performing prepatch operation on home /oracle/db/19
Perpatch operation completed successfully on home /oracle/db/19
Start applying binary patch on home /oracle/db/19
Binary patch applied successfully on home /oracle/db/19
Performing postpatch operation on home /oracle/db/19
Postpatch operation completed successfully on home /oracle/db/19
Preparing home /oracle/db/19 after database service restarted
No step execution required.........
Trying to apply SQL patch on home /oracle/db/19
No step execution required.........
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:seha1
RAC Home:/oracle/db/19
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /oracle/install/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /oracle/install/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"
==Following patches were SUCCESSFULLY applied:
Patch: /oracle/install/31750108/31771877
Log: /oracle/db/19/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-28_17-12-19PM_1.log
Patch: /oracle/install/31750108/31772784
Log: /oracle/db/19/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-28_17-12-19PM_1.log
OPatchauto session completed at Wed Oct 28 17:34:59 2020
Time taken to complete the session 26 minutes, 40 seconds
[root@seha1 ~]# su - oracle
Last login: Wed Oct 28 17:34:57 PDT 2020
[oracle@seha1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)
31771877;Database Release Update : 19.9.0.0.201020 (31771877)
OPatch succeeded.
Now the DB home on SEHA1 node is patched (with OCW patch as well). We can leave se1 database on SEHA2 or relocate it back to SEHA1, in any case both servers and both databases are up to date and the patch process is completed.
What Else Can Be Done
One thing I could have done differently is using the Out Of Place (OOP) GI patching. Using this feature we can patch the GI while the database is still up and running, resulting in a shorter downtime (the downtime will be necessary for the DB home patch only). However, in our case, to reduce the downtime we relocated the database, which was done after GI and DB patch, so we actually got an even shorter downtime for se1 (se2 could have benefited from a shorter downtime using the OOP GI patch).
Note: this feature is documented for GI patching using RAC databases. I’m not sure it’s supported for SEHA databases (and officially it’s not supported for Oracle Restart).
For more information about OOP GI patching check MOS note 2662762.1.
Conclusion
Patching SEHA environment is not very different from RAC environment, but there are a few things we need to remember:
- SEHA database will only run on one node, so make sure where you would like it to run before patching a specific node
- opatchauto will not patch a home without a running database, so if you don’t have active databases on this home, you’ll have to patch this home manually
- If you relocated an unpatched database to a patched home, don’t forget to run datapatch manually to patch the database