Patching Oracle Exalogic I
Published on: Author: Jos Nijhoff Category: OracleAfter some posts focussed on the administration and management aspects of the Oracle Exalogic engineered system in combination with Enterprise Manager 12c ‘Cloud Control’ and Ops Center 11g/12c, it’s time to tell you a little more about patching the Exalogic machine.
1 PATCHING ORACLE EXALOGIC
1.1 Introduction
We have been doing regular updates to our quarter rack system since it was delivered to us in august 2011 at version 1.0.0.0.0. Oracle delivers quarterly updates to the Exalogic engineered system, in the form of patch bundles. These ‘EECS’ (Exalogic Elastic Cloud Software) patch bundles deliver updates that cover everything down from firmware for the network switches and storage via OS updates up to patches for core products such as WebLogic 11g, Coherence. You can find information on Exalogic quarterly updates in Oracle Support note 1314535.1, called ‘Exalogic Patch Set Updates (PSU) Master Note’.
There has been considerable controversy about patches for the Exalogic, as Larry Ellison has stated that the Exalogic system could be updated with ‘a single patch file that updates everything’ or something of the sort. Oracle bashers, with IBM affiliated bloggers among the first of them, have been quick to critize and even ridicule this statement and will tell you have to collect all the pieces yourself, and do it all yourself. You might wonder who to believe and what’s true about this and what is not….
This blogpost is intented to give you more details and a balanced view, and to tell you the real deal with respect to Exalogic patching!
1.2 A bit of background on the Exalogic
To understand what the discussion is about, and to understand Exalogic patching, we need to look at what’s inside an Exalogic rack and have some idea of the hardware and software that make it such a neat proposition. The Exalogic is: a highly extensible ‘prefab datacenter in a box’. As such it is a high density and high performance system made up of some very nice network, storage and computing components: IB infiniband switches (Sun NM2-GW QDR), networked storage (a Sun ZFS 7320 storage appliance) and a number of Sun X4170 M2 blade servers that are called ‘compute nodes’.
A lot of attention has been put into hardware and system redundancy, which is important for patching as the upgrade procedure is intended for ‘rolling’ upgrades that keep downtime for production systems to a minimum, if any. Even the entry level machine (a one eigth rack) has two infiniband switches, two ZFS storage heads, four compute nodes and a reduntant network fabric connection.
The infiniband network fabric is the ‘special sauce’ that ties all these parts together and makes it quite unique (this is called the ‘Exabus’, more on that in a future blogpost). It is important to stress that we are not just looking at a bunch of high end components here, we are dealing with an engineered system. The hardware and software are tailored to each other and are engineered in step, which is another important aspect to write about in a future blogpost.
For the moment I refer to the following links if you want to know more, and continue on the software side of things and patching the Exalogic.
- 1. Exalogic system overview at oracle.com
- 2. Exalogic X2-2 hardware overview on oracle.com
- 3. Exalogic Software overview on oracle.com
- 4. Short Q&A for Oracle Exalogic Elastic Cloud X2-2
The software that drives all this hardware is collectively called ‘Elastic Cloud Software’ and is specially tailored to the machine, hence the adagium ‘hardware & software engineered together’. It spans the whole stack as it consists of firmware builds, OS builds (Linux & Solaris) with infiniband drivers, and core Oracle software components like Java (JRockit and Hotspot JVM’s), Weblogic, Coherence and Tuxedo that have been adapted to communicate over infiniband (henceforth shortened to ‘IB’) using the Sockets Direct Protocol (SDP) instead of TCP/IP and optimized to make the most of this advancement in networking technology. *
In addition there’s backward compatibility built in for (third party) applications and components that cannot yet do without TCP/IP networking in the form of Ethernet over and IP over infiniband (shorthand ‘EoIB’ and ‘IPoIB’). Finally there’s a set a mangement tools and scripts to facilitate administration and operation.
All this together builds a well defined and highly available, extremely reliable and performant computing platform for consolidation of middleware and applications that can be adequately operated and managed from a single source. Of course Oracle WebLogic based solutions such as Oracle SOA Suite & BPM, Webcenter, Content Management 11g and ATG etc. take center stage when it comes to boosting performance, but even non-Oracle software solutions that employ a lot internal communication between subcomponents and are certified for current Linux and Solaris versions could and would benefit enormously from running on Exalogic.
Giving some credit to the criticasters: sadly, a hypervisor (virtualization layer) has been missing from the mix thus far, shedding some doubt on the ‘cloud part’, but it (Oracle VM 3) is on the Exalogic roadmap and may well be with us in a few months time. If you are really curious about this, do a little ‘Googling’ on the term ‘Exalogic Control’. Also remember that virtualization is not a purpose onto itself, but a means to a end. Having said that, it would add another level of flexibility and ease of operation, especially when looking at provisioning test and development enviroments using Oracle virtual assemblies.
* Note that these adaptations have actually been put into the main codeline, for instance for WebLogic they have been in there since version 10.3.4. So if you are running on WebLogic 10.3.4+, you have them too. However you can only actually use them if you are running on an Exalogic.
1.3 Patching the Exalogic: what’s the deal?
Now back to patching. What is the story with the ‘single patch file’ that ‘updates everything’ ? Let’s take patchset update v1.0.0.4.0 shipped in January 2012 as an example, which for Linux brings us to downloading a single zipfile named p13113092_111140_Linux-x86-64_Jan2012.zip. Well that part is correct: it is a ‘single patch file’ indeed.
It’s quite a big file too, some 2.4Gb. When we read the documentation we see why it’s so big: the patch is cumulative, so it contains all the previous updates as well. That’s nice: we can apply it to any Exalogic 1.0 system without having to download and apply previous updates. Looking inside the file (or in the documentation on MOS (=My Oracle Support)), we find that it contains what appear to be separate updates for the infrastructure and for the middleware (being Coherence, JRockit, WebLogic).
[weblogic@xxxxcn1 PSUs_v1]$ unzip p13113092_111140_Linux-x86-64_Jan2012.zip
Archive: p13113092_111140_Linux-x86-64_Jan2012.zip
creating: 13113092/
inflating: 13113092/README.txt
creating: 13113092/Infrastructure/
creating: 13113092/Infrastructure/1.0.0.2.2/
inflating: 13113092/Infrastructure/1.0.0.2.2/README.txt
extracting: 13113092/Infrastructure/1.0.0.2.2/el_infrastructure_10022.zip
inflating: 13113092/Infrastructure/1.0.0.2.2/README.html
creating: 13113092/Middleware/
creating: 13113092/Middleware/Coherence/
creating: 13113092/Middleware/Coherence/3.6.1.6/
inflating: 13113092/Middleware/Coherence/3.6.1.6/README.txt
extracting: 13113092/Middleware/Coherence/3.6.1.6/coherence-java-3.6.1.6b27994.zip
creating: 13113092/Middleware/JRockit/
creating: 13113092/Middleware/JRockit/1.6.0.29/
inflating: 13113092/Middleware/JRockit/1.6.0.29/README.txt
extracting: 13113092/Middleware/JRockit/1.6.0.29/jrockit-jdk1.6.0_29-linux_x86_64.zip
creating: 13113092/Middleware/WebLogic/
creating: 13113092/Middleware/WebLogic/10.3.4.0.4/
inflating: 13113092/Middleware/WebLogic/10.3.4.0.4/README.txt
extracting: 13113092/Middleware/WebLogic/10.3.4.0.4/WebLogic.zip
inflating: 13113092/README.html
That’s quite logical: we get patches for the infrastructure (which we probably have to apply first) and patches for the middleware. So let’s look at the infrastructure part, which looks to be another zipfile and a readme. Let’s look at its contents:
[weblogic@xxxxcn1 1.0.0.2.2]$ unzip el_infrastructure_10022.zip
Archive: el_infrastructure_10022.zip
creating: BaseImage/
creating: BaseImage/1.0.0.2.2/
creating: BaseImage/1.0.0.2.2/Tools/
inflating: BaseImage/1.0.0.2.2/Tools/imageinfo
…
…
inflating: one-command/1.0.0.2.2/generate_factory_config.sh
inflating: one-command/1.0.0.2.2/solarisCNodeNetworkUpdateDriver.sh
inflating: one-command/1.0.0.2.2/el_configurator.ods
inflating: one-command/1.0.0.2.2/configure_node_ntp.sh
inflating: one-command/1.0.0.2.2/steps_list.sh
[weblogic@xxxxcn1 1.0.0.2.2]$ rm el_infrastructure_10022.zip ; ls
BaseImage NM2-36p NM2-GW one-command README.html README.txt ZFS_Storage_7320
Now let’s look at the middleware part:
[weblogic@xxxxcn1 13113092]$ cd Middleware ; ls
Coherence JRockit WebLogic
[weblogic@xxxxcn1 Middleware]$ cd WebLogic ; ls
10.3.4.0.4
So, have we been tricked after all? It looks to be all seperate fixes and patches to do. Well, yes and no… Of course Larry Ellison’s ‘single patch’ statement is an oversimplification and saying that it will patch ‘everything’ might make you expect you’ll have your Exalogic updated with just ‘push of a button’. But that’s marketing and oneliners, and ‘the big picture’ that concerns boards of directors. Anyone with a bit more technical background would know not to expect this in the first place. Nor even wish for it, as it would severely limit your options when planning and executing an upgrade to a mission critical production system.
However, there’s no need to believe the FUD that some of Oracle’s competitors like spreading either. Oracle has put together a coherent, sensible and reliable overall procedure to take us through the complete upgrade process. All steps from the lowest level firmware to the highest level middleware have been put into a single context, covering the stack from network, storage, to operating system, and on to Java and the application servers. The procedure leads us pretty much every step of the way, providing you pay attention to detail and put some concentrated effort into it.
Also it is important to note that it’s a ‘rolling’ upgrade process, meaning you can apply it to an online production system without significant downtime, if any, provided that you have setup your middleware/applications with clustering. There have been some issues and ambiguities in the first few patchset updates, but things have improved rapidly from there, and we can now say that it’s quite reliable and you can recover when making mistakes.
I can say this from direct personal experience. And the procedure is getting more consistent with each new quarter, Oracle is really working on improving the experience. Given that this is quite advanced technology made up of many complicated building blocks, I find this quite an achievement and a pleasant surprise.
1.4 Conclusion
The hardware and software in an Exalogic are tailored to each other and are engineered in step, which is key to the concept of engineered systems. The simple fact that you can now go and buy such a tightly ingrated system to run your middleware and applications instead of having to assemble and tweak all these bits yourself can save IT departements a truckload of work in assembling, setting up and maintaining the datacenter and thus frees up budget to focus on serving business needs.
This is even without considering the huge performance benefits that also translate into business advantages, e.g. by shortening processing times leading to faster billing, better optimized logistics and lower inventory or some other business advantage.
So let’s get back to the story of the ‘single patch file’ that ‘updates everything’ as Oracle’s marketing onliners would have it. Is it true? And what about the horror stories that it’s a jumbled mess of parts that you have to micromanage all by yourself ? As anyone with a little understanding of maintaining an advanced IT infrastructure might have guessed, neither statement is 100% true or false. The Exalogic cannot be patched without some thinking and planning, nor ‘with the push of a button’. However, saying that it’s just an incoherent bunch of components that you have to patch and maintain all by yourself is just as far from the truth. As most things in real life, it’s neither black or white. I’d say a pretty nice shade of grey that gets lighter and brighter as it matures rapidly.
I know, ‘nuance’ is not a popular thing these days neither in politics nor in business management at large, but I’ll stick to it anyway. It’s real life, it’s all we’ve got. Granted, we are not talking about one patch, we are talking about single patch releases that reliably and consistently upgrade all the necessary components in step, leading to a clear set of possible configurations for the Exalogic system. The same goes for the Exadata, for that matter.
Oracle has put together a coherent, sensible and reliable overall procedure to take you through the complete upgrade process for the Exalogic, keeping up with security and bugfixes through quarterly updates with little or no downtime. But it does take a bit of expertise, effort and attention to detail, as with everything delivering value, be it IT or other lines of business.
Consider this :
The most important point to my story here is this: patching an Exalogic is not a no-brainer, but it is still much, much better and easier to do than having to this for comparable stack of your average ‘best of breed’ datacenter components!!
That is: the custom, multivendor hard- and software sprawl in the datacenter that is the bread and butter of many IT departments in companies across the globe, be they small, medium, large or huge. Try keeping up-to-date, in-tune and online 24×7 instead! I am talking about that same ‘best of breed’ multivendor server and network sprawl that keeps those very IT departments busy so many nights and weekends running up downtime and overtime. I would prefer a set of standardized engineered systems like Exadata and Exalogic any day, saves you quite a bit of budget for smarter things to work on!
IBM has recognized this as well and has recently launched their PureSystems, changing their tune and proving the point of the engineered systems concept. The question is: can they execute as well as in real life as they do on paper? Oracle sure has some headstart for them to recover. We will have to wait and see. A fair bit of competition can sure speed things up and make everybody better.
The fact that there’s only a very narrow and well defined number of both hardware and software configurations accross the world make the Oracle engineered systems such as Exalogic and Exadata much better supportable and will drive innovations and optimizations and issue resolution for customers forward much faster. There may be more than a bit of influence from Steve Jobs here, a known friend of Larry Ellison, who’s first actions were to clean up and simplify the product line when he came back to rescue Apple in 1997, ultimately leading to the iPad and iPhone. Some very integrated hard- en software, indeed. Just look where it took them, where they are now….
So, who knows what the future holds: someday not so far removed, engineered systems may have ‘self-update’ functionality similar to what we see for the iPhone.
1.5 Next time
In one of my next post, I will go into more detail on how patching the Exalogic is actually done, taking the installation of the same patch set update 4 (i.e. patchset 13113092) on our quarter rack at Qualogy as an example.
Publicatiedatum: 8 juni 2012
