Patching Oracle Exalogic - Updating the ZFS 7320 Storage Appliance I
Back to overview

Patching Oracle Exalogic - Updating the ZFS 7320 Storage Appliance I

Published on: Category: Oracle

Part 3a

In my previous post we examined the rolling upgrade process for the network infrastructure (Infiniband Gateway switches), so next in line is the upgrade process for the storage infra, i.e. the ZFS 7320 storage appliance.

Like every piece of hardware in the Exalogic, the ZFS 7320 storage appliance has high availability built-in. In this case we have two ‘storage heads’ that are working in active-passive mode to ensure high availability. This enables us to perform a rolling upgrade.

I will demonstrate that the ZFS storage heads can be upgraded in rolling fashion, without severely impacting the storage services and keeping the Exalogic and everything running on it online while doing it!

1.1     Checking the documentation

First of all we should again look at the README.html and additional documentation for patch 13113092 and carefully follow the instructions pertaining to the ZFS 7320 part.

ZFS Storage Appliance (ZFS_Storage_7320)
Upgrading the software and ILOM on each storage head involves the following steps:

  1. Updating storage software to version Q3.2.1
  2. Applying the ILOM update to the storage nodes
  3. Updating storage software to version Q3.3

The factory default version of the ZFS storage 7320 software in Exalogic v1.0 and v1.0.0.1.0 systems is Q3.1.1. Upgrading that version directly to Q3.3 is not supported. So apparently there are two types of firmware to be updated: storage software and ILOM software. For the first type we have to apply an intermediate update first, upgrading directly to Q3.3 should not be done.

1.2     Checking current versions on the storage heads.

As before, we should first check the current firmware versions. For this we should login to the service processor on the storage head ILOM interfaces.

Checking the storage software versions:

  1. % <strong>ssh root@xxxxsn1-c.qualogy.com
  2. </strong>Password:
  3.  
  4. Oracle(R) Integrated Lights Out Manager
  5.  
  6. Version 3.0.14.11 r61398
  7.  
  8. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
  9.  
  10. -&gt; <strong>start /SP/console
  11. </strong>Are you sure you want to start /SP/console (y/n)? <strong>y
  12. </strong> 
  13. Serial console started.  To stop, type ESC (
  14.  
  15. xxxxsn1:&gt; <strong>show
  16. </strong>Properties:
  17. showcode = false
  18. showstack = false
  19. exitcoverage = false
  20. showmessage = true
  21. asserterrs = false
  22.  
  23. Children:
  24. configuration =&gt; Perform configuration actions
  25. maintenance =&gt; Perform maintenance actions
  26. raw =&gt; Make raw XML-RPC calls
  27. analytics =&gt; Manage appliance analytics
  28. status =&gt; View appliance status
  29. shares =&gt; Manage shares
  30.  
  31. xxxxsn1:&gt; <strong>maintenance</strong> <strong>system</strong> <strong>updates
  32. </strong>xxxxsn1:maintenance system updates&gt; <strong>show
  33. </strong>Updates:
  34.  
  35. UPDATE                           DATE                      STATUS
  36. ak-nas@2010.08.17.1.1,1-1.16     2010-11-1 12:46:16        previous
  37. ak-nas@2010.08.17.2.1,1-1.21     2011-3-10 23:49:47        previous
  38. <strong>ak-nas@2010.08.17.3.0,1-1.25     2011-4-29 15:48:52        current
  39. </strong> 
  40. Deferred updates:
  41.  
  42. The appliance is currently configured as part of a cluster. The cluster peer
  43. may have shared resources for which deferred updates are available. After all
  44. updates are completed, check both cluster peers for any deferred updates.

We can see the update history and the version we have now. We find that we have the same current version ak-nas@2010.08.17.3.0,1-1.25 on storage head 2 as well, as it should be (not shown here).
This can also be done via the ZFS 7320 storage webconsole, by going to the ‘Maintenance’ section and then to the ‘SYSTEM’ tab. I will use the webconsole gui later to check the update when it has been applied, see below.

Which version is actually good ? Further down in the README we find the version to check for:

17. Select the newly downloaded version

zfssn01:maintenance system updates> select ak-nas@2010.08.17.2.1,1-1.21
zfssn01:maintenance system updates ak-nas@2010.08.17.2.1,1-1.21>

So, apparently we are already on a somewhat higher version of the storage software as is required. We’re good here.

Checking the ILOM software versions:

Furthermore, we also have to check the ILOM versions. This can be done both via the ILOM gui and the CLI. The README.html states

Upgrading ILOM on the ZFS Storage Head

Check the current ILOM version:
-> version
SP firmware 3.0.9.27
SP firmware build number: 58107
SP firmware date: Tue Aug 17 01:35:56 EDT 2010
SP filesystem version: 0.1.23
If the firmware build number is less than 61398, you need to upgrade it.

Let’s check our system :

  1. xxxxxsn<strong>1</strong>:maintenance system updates&gt;
  2. Serial console stopped.
  3.  
  4. -&gt; <strong>version
  5. </strong>SP firmware <strong>3.0.14.11
  6. </strong>SP firmware build number: <strong>61398
  7. </strong>SP firmware date: Tue Dec 21 17:08:53 EST 2010
  8. SP filesystem version: 0.1.23
  9.  
  10. xxxxxsn<strong>2</strong>:&gt;
  11. Serial console stopped.
  12.  
  13. -&gt; version
  14. SP firmware <strong>3.0.14.11
  15. </strong>SP firmware build number: <strong>61398
  16. </strong>SP firmware date: Tue Dec 21 17:08:53 EST 2010
  17. SP filesystem version: 0.1.23

Via de ILOM webconsole you can find the version as well, via the ‘Overview’ subtab of ‘System Information’.

Looks like we are current here as well. So, as with the Gateway switches, we find that there are no updates to be applied at present because we already applied them with the october 2011 PSU 12634555. Instead I will again take the upgrade to version 2.0.0.0.0 (patch 13795376) as an example. Before we can start applying any patches however, we should first check which head is the active one, so we can do the other head first as suits our rolling upgrade process.

1.3     Checking which storage head is the active node

Login to the ILOM (if you’re not still there).

  1. <strong>Checking storage head 1 :
  2. </strong>xxxxxsn1:&gt; <strong>cd /
  3. </strong>xxxxxsn1:&gt; <strong>configuration cluster
  4. </strong>xxxxxsn1:configuration cluster&gt; <strong>show
  5. </strong>Properties:
  6. state = AKCS_OWNER
  7. <strong> description = Active (takeover completed)
  8. </strong>peer_asn = 9faf8ff1-c3a8-c090-8f4e-9871618a152e
  9. peer_hostname = xxxxxsn2
  10. peer_state = AKCS_STRIPPED
  11. peer_description = Ready (waiting for failback)
  12. Children:
  13. resources =&gt; Configure resources

So we see that storage head1 is the active head and we should patch storage head2 first. For consistency one should also check te other the other head, but I will not show it here.

We can also see this very nicely via the 7320 storage webconsole under ‘Configuration > CLUSTER’ (figure 1) :

Figure 1 : Cluster configuration and state of the ZFS 7320 as shown in the webconsole GUI on head1.

BTW: the  management webconsole can be found at https://<storage head ip>:215. The console for storage head2 (figure 2) actually shows the reverse of this view, mirrored left-to-right. Also note the FAILBACK and TAKEOVER buttons.

Figure 2 : Cluster configuration and state of the ZFS 7320 as shown in the webconsole GUI on head2.

1.4     Upgrading storage head 2

OK, since storage head 1 is actively in control, we should upgrade head 2 first. We will again take the 2.0.0.0.0 upgrade as an example, as there are actually some updates to be done. We take “Exalogic Elastic Cloud Software Upgrade Guide, Volume 1 Release 2.0.0.0.0” (file eecs_20_upg.pdf when unzipping the 2.0.0.0.0 patchfile p13795376_10020_Linux-x86-64.zip).

The upgrade guide states the following :

Upgrading the ZFS software to version 2011.1.1.0, and ILOM on each storage head involves the following steps:

  1. Upgrading ZFS Storage 7320 Software from Version Q3.1.1 to Version Q3.2.1
  2. Upgrading ILOM on ZFS Storage Heads
  3. Upgrading ZFS Storage 7320 Software to Version 2011.1.1.0

So again three upgrade steps to do.

Step 1: Upgrade to storage version 2010.Q3.2.1

Further on we read:

“This section describes how to upgrade the ZFS Storage 7320 software from version 2010.Q3.1.1 (2010.08.17.1.1,1-1.16 – the factory default for Exalogic 1.0) to version 2010.Q3.2.1 (2010.08.17.2.1,1-1.21). This version is a prerequisite for installing the 2011.1.1.0 version, which is the target software version in this upgrade kit for the Sun ZFS Storage 7320 appliance.”

As mentioned earlier in section 1.2, we are already at storage version ak-nas@2010.08.17.3.0,1-1.25, which is higher, so we only have to do steps 2 and 3, good.

Step 2: Upgrade ILOM on the ZFS Storage Heads

We have already gone through steps 1-6 of the guide, on to 7 it is:

7. Stop the system by running the following command:

  1. xxxxxsn2:configuration cluster&gt;
  2. Serial console stopped.
  3. -&gt; <strong>stop /SYS
  4. </strong>Are you sure you want to stop /SYS (y/n)? <strong>y
  5. </strong>Stopping /SYS

8. To load the new FW file, you require the FTP URL. The FTP URL is of the following format:ftp://root:<root_password>@<ethernet_address_of_active_ZFS_storage_head>/<path_ to_ZFS_ILOM_update_dir>/3.0.16.10_r65138/ILOM-3_0_16_10_r65138-Sun_Fire_ X4170M2_X4270M2.pkg

Ok let’s do that, using our patcher account for FTP which we defined via the ZFS storage console before:

  1. -&gt; <strong>load -source ftp://patcher:mypassword@&lt;storage VIP address&gt;//export/common/patches/todo/</strong>
  1. <strong>13795376/Infrastructure/2.0.0.0.0/ZFS_Storage_7320/ILOM/3.0.16.10_r65138/</strong>
  1. <strong>ILOM-3_0_16_10_r65138-Sun_Fire_X4170M2_X4270M2.pkg
  2. </strong>NOTE: An upgrade takes several minutes to complete. ILOM
  3. will enter a special mode to load new firmware. No
  4. other tasks can be performed in ILOM until the
  5. firmware upgrade is complete and ILOM is reset.
  6. You can choose to postpone the server BIOS upgrade until the
  7. next server poweroff. If you do not do that, you should
  8. perform a clean shutdown of the server before continuing.
  9. Are you sure you want to load the specified file (y/n)?<strong> y
  10. </strong>Preserve existing configuration (y/n)? <strong>y
  11. </strong>Delay BIOS upgrade until next server poweroff (y/n)? <strong>n
  12. </strong>..........................................................................................
  1. ...............
  2. Firmware update is complete.
  3. ILOM will now be restarted with the new firmware.
  4. -&gt; /sbin/reboot
  5. Connection to xxxxsn2-c closed by remote host.
  6. Connection to xxxxsn2-c closed.

OK that was the first reboot after upgrading… wait till it comes back online and login to the ILOM again:

12. Run the following command to verify the new ILOM version: -> version

Actually this is not needed as it already reports the version on login :

  1. % <strong>ssh root@xxxxsn2-c
  2. </strong>Password:
  3. Oracle(R) Integrated Lights Out Manager
  4. <strong>Version 3.0.16.10 r65138
  5. </strong>Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
  6. -&gt; version
  7. SP firmware 3.0.16.10
  8. SP firmware build number: 65138
  9. SP firmware date: Wed Jun 15 08:02:17 EDT 2011
  10. SP filesystem version: 0.1.23

Now it says we need to set the boot option to “ROM Enable” in the BIOS, I will not show this here. After doing so, it reboots once more, and we need to log back in once more to check the final status:

23. Run the following commands to check if the current head is Ready (waiting for failback), (that is, description = Ready (waiting for failback) ):

  1. xxxxsn2 console login: <strong>root
  2. </strong>Password:
  3. Last login: Tue Mar 20 13:05:42 on console
  4. xxxxsn2:&gt; <strong>cd /
  5. </strong>xxxxsn2:&gt; <strong>configuration cluster
  6. </strong>xxxxsn2:configuration cluster&gt; show
  7. Properties:
  8. state = AKCS_STRIPPED
  9. <strong> description = Ready (waiting for failback)
  10. </strong>peer_asn = a54b53a0-afba-eae1-a77a-b0013813b629
  11. peer_hostname = xxxxsn1
  12. peer_state = AKCS_OWNER
  13. peer_description = Active (takeover completed)
  14. Children:
  15. resources =&gt; Configure resources

OK, we have thus upgraded the ILOM software of storage head2 to version 3.0.16.10 r65138. We are now ready for step 3, upgrading the ZFS Storage 7320 Software to Version 2011.1.1.0.

1.5     Next time

In the next post, we continue where we left off to finish the upgrade of the ZFS storage by doing step 3 of the upgrade, then a switchover the active storage role to the newly upgraded head2, then upgrade head1 and finally check if eveything went OK. Of course in real life this should all be done without pause in a single excercise!

Publicatiedatum: 27 augustus 2012

Jos Nijhoff
About the author Jos Nijhoff

Jos Nijhoff is an experienced Application Infrastructure consultant at Qualogy. Currently he plays a key role as technical presales and hands-on implementation lead for Qualogy's exclusive Exalogic partnership with Oracle for the Benelux area. Thus he keeps in close contact with Oracle presales and partner services on new developments, but maintains an independent view. He gives technical guidance and designs, reviews, manages and updates the application infrastructure before, during and after the rollout of new and existing Oracle (Fusion) Applications & Fusion Middleware implementations. Jos is also familiar with subjects like high availability, disaster recovery scenarios, virtualization, performance analysis, data security, and identity management integration with respect to Oracle applications.

More posts by Jos Nijhoff
Comments
Reply