Installing PSU (GI+RAC+JVM)

I’ve installed quite a few PSUs in my professional life, but this time, it was something else. I’m talking about a RAC environment with 2 nodes and a database containing JVM, so this was a complex patch to begin with. The problem is the the readme is incomplete and on top of that I ran into a few issues (some I could find on MOS but some I couldn’t). I’m writing this to assist anyone who needs to install this patch by providing the steps that worked for me. Please note that I patched 12.1.0.2 home, for other versions the steps might be a bit different.

OPatch

As always, it’s best (and sometimes required) to use the latest OPatch, so go on and download patch 6880880 . Upgrading OPatch is done by simply renaming the OPatch directory under ORACLE_HOME and copying the new one from the zip file. Remember, copy the new OPatch to the GRID_HOME and all ORACLE_HOMEs on all servers.

Pre-steps

Download the patch and extract it on all servers. There are quite a few patches out there, so you need to pick the right one. For database PSU go to MOS note 756671.1 and go to your version. For every version there are a few patches, these are the main ones (for versions < 12.2):

• Database PSU (use this one if you have only database home on the server)
• GI PSU (use this one if you have GI home as well, like RAC or Oracle Restart, it includes the PSU for the database home as well, so this is the only one you’ll need)
• COMBO database and JVM (use this one if you have only database home, but you also need to patch the JVM in the database)
• COMBO GI and JVM (use this one if you have GI home and you need to patch the JVM as well) – this is the one I used, as since is was 12.1.0.2, I downloaded patch 26636286

Before starting the patch, don’t forget to backup your database and have a fallback plan in case something goes terribly wrong.

COMBO Patch

The combo patch only saves us the time to download 2 patches (PSU and JVM), but that’s it. It includes 2 different patches that we need to install and handle separately. The readme doesn’t say anything besides “follow the readme of each patch”.
I’m not sure what is the correct order, if there is one. I installed the PSU first and then the JVM.

PSU

The PSU part of the patch is quite simple. It’s a rolling upgrade, meaning we patch one node at a time, while the others are up and running. Note that the application and sessions will still have to failover (more than once) during the PSU installation.
As root on the first node:

1. export ORACLE_HOME=<grid_home>
2. export PATH=$PATH:$ORACLE_HOME/OPatch
3. Use the opatch command from the readme (in my case: opatchauto apply <UNZIPPED_PATCH_LOCATION>/26635815)
4. Repeat steps 1-3 on the other nodes, one by one

Be aware that the opatchauto command will stop the database and the clusterware, patch them and restart them without any confirmation.
In 11.2 I had a few cases where the patching step failed. I simply executed the same opatchauto command again and it worked.

JDBC Patch for GI

The JVM patch itself is for the database home only, but it includes a JDBC patch for the GI home (and client homes if you have them installed). It took me a while to find the correct steps for this patch, so here they are. This is a rolling patch as well, so you have to start with the environment up and running (at least 2 nodes should be up for that one):
As root on the first node

1. export ORACLE_HOME=<grid_home>
2. Stop the clusterware (crsctl stop has)
3. unlock the GI home: $ORACLE_HOME/perl/bin/perl -I$ORACLE_HOME/perl/lib -I$ORACLE_HOME/crs/install $ORACLE_HOME/crs/install/rootcrs.pl -prepatch

As the GI owner on the same node:

1. export PATH=$PATH:$ORACLE_HOME/OPatch
2. Go to the patch dir (the specific JVM patch dir in the directory of the combo patch)
3. Run opatch: opatch apply -local

As root on the same node:

1. Lock the GI home: $ORACLE_HOME/perl/bin/perl -I$ORACLE_HOME/perl/lib -I$ORACLE_HOME/crs/install $ORACLE_HOME/crs/install/rootcrs.pl -postpatch

Repeat all the steps above (1-3 as root, 1-3 as GI and 1 as root) on the other nodes, one by one.

JVM Patch for the Database

This is the last step, patch the databsae JVM. This is not a rolling patch so for that we will need a full downtime of the database.
As root on all nodes:

1. Stop the clusterware (crsctl stop has)

As the database owner on all nodes:

1. Go to the patch dir (the specific JVM patch dir in the directory of the combo patch)
2. export PATH=$ORACLE_HOME/perl/bin:$PATH:$ORACLE_HOME/OPatch
3. export PERL5LIB=$ORACLE_HOME/perl/lib (as per note 2002334.1)
4. opatch apply

As root on one node:

1. start crs
2. chown -R oracle:oinstall $ORACLE_BASE/cfgtoollogs/sqlpatch (as per note 2242227.1)

As database user on the same node:

1. Disable RAC database (in sqlplus): alter system set cluster_database=false scope=spfile;
2. Stop the database: shutdown immediate
3. Start the database in upgrade mode: startup upgrade
4. Exit sqlplus and go to OPatch dir: cd $ORACLE_HOME/OPatch
5. Execute the post patch update: ./datapatch -verbose
6. Enable RAC database (in sqlplus): alter system set cluster_database=true scope=spfile;
7. Stop the database: shutdown immediate
8. Exit sqlplus and start the database: srvctl start database -d <DB>
9. Connect to sqlplus and recompile objects: @?/rdbms/admin/utlrp.sql

repeat the last block of commands (1-9) on every database you have in this ORACLE_HOME.
After you finished to patch all the databases, start the clusterware as root on all other nodes (crsctl start has). This should start all the resources and the patch installation is complete.