Setting up limited membership for VServers (part 2)
Published on: Author: Jos Nijhoff Category: IT development and operationsAfter setting up shop in part 1 of this post, it’s now time to get to business and change our vServers to limited membership on the IPoIB-default network.
In the previous post, I provided some background on Infiniband limited/full membership and why this is important from a security perspective on an Exalogic system. Because vServers are created as full members on the IPoIB-default network, you might have an open door for some unwelcome ‘ssh hopping’ in between vServers and even between vServers and compute nodes over this network.
Therefore, in part 1, we set up the Exalogic Control vServer with the tooling we need to set things straight on the IPoIB-default network if this is deemed necessary for proper network isolation of our application deployments. Thus, we installed the Infrastructure-as-a-Service API and CLI.
Changing vServers to limited membership
Now that everything is in place, I will demonstrate how changing vServers to limited membership is actually done. Ofcourse it would be best if Oracle changed the behaviour of Exalogic Control so it created the vServers as limited members instead of full members just as for the IPoIB-vserver-shared-storage and IPoIB-virt-admin, but for the moment there’s no EMOC patch for this and we only have a workaround in the form of a set of “toggle scripts”. You can download these from MOS note 1908498.1 and stage them in an appropriate place, in my case /u01/common/general/tools/toggle-scripts.
First however (there’s always more pre-reqs), we must give the Exalogic Control host (EC1) access to the Oracle VM repository, as the scripts need to update the vm.cfg files of the vServers we want to “toggle” membership for. Therefore, we need to mount the ExalogicRepo share in read/write mode.
Finally, you must set up passwordless ssh of the Exalogic Control vServer with the compute nodes in the vDC. This is described in the MOS note. After this you can access the compute nodes using the Distributed Command Line Interface (DCLI), which is also used by the “toggle scripts”:
The all-cnodes file contains the list of all our compute nodes, which is eight in our 1/4 rack.
Proper authentication
The toggle scripts will use the IaaS API and CLI, so they need the proper authentication, just like when you login to the Exalogic Control WebUI. First of all we need access to the vDC account in Exalogic Control that holds our vServers to be changed. For that, we need to know the vDC account’s ID. We can obtain this from EMOC via the IaaS akm-describe-accounts function:
The IaaS API is called over https so obviously this could also be done from a separate utility/provisioning Linux host or vServer instead of the EC1 vServer itself, you would just have install the IaaS rpms there. If you have one or more authorised admin users set up in EMOC instead of root, it would be best to use those, as in my example.
Find your account ID
The akm-describe-accounts function now asks you to accept its certificate and then provides you with a list of vDC ID’s with their names and descriptions. The account I want to work in for this demonstration is the QEL-TrainingDemo-project account.
Testing an ssh session via IPoIB-default
So the account ID I am looking for is ACC-428c9d8e-89dc-42b1-86cf-ea6fa02e8468. This account holds my vServers. It is a small demo project with four vServers. Two vServers of type SMALL for running a Traffic Director cluster, and two vServers of type LARGE for running a Weblogic cluster.
The two vServers for Weblogic called qnl-train1-wls-vs1 and -vs2 are connected to the IPoIB-default network (192.168.10.0/24) as they need to access a database, so I need to fix them by making them limited members of the IPoIB-default network.
Currently they are full members and I can setup an ssh session from one to the other, if I can get hold of the IP number and password, as demonstrated below:
This “backdoor” can be closed by making them limited members. For this, I need to know their vServer UUID’s so I can put those into a list file called vServers-2btoggled.lst. Unfortunately, the toggle scripts cannot work these out automatically yet. You can find the UUID’s via EMOC WebUI or from the vm.cfg files.
Let’s have a look at the IB partition keys that are currently used by my two vServers, e.g. for the first one:
Change the pkey
Infiniband membership is determined by the partion key. As we see, the Infiniband partition key (pkey) that is currently used for my vServers is 0xffff. The most significant bit determines the IB membership. If it is turned on, we have full membership, if it’s turned off, we have limited membership. So we need to change the pkey to 0x7fff for limited membership and restart the vServers for this to take effect.
We could do this change manually, but the toggle scripts will take care of this for us, and perform various checks. This is especially convenient if we have many vServers to change instead of just the two, as is the case for most of the customer projects I worked on where 20+ vServers had to be “toggled”.
UpdateVServers.sh and CheckVServers.sh
There are two main scripts : UpdateVServers.sh which makes the changes and CheckVServers.sh which can perform pre- and post change checks. Both scripts take a lot of arguments as input:
- n to provide the list of vServers to be changed
- c to tell it about all the OVS compute nodes in my machine
- url to specify the EMOC url to call the IaaS API
- r to tell it where my OVS repository is
- ak to specify my access key file
- a to specify the account I want to work on
- k to tell it which pkey value I want to toggle
- t to specify the timeout, how long to wait for a shutdown to complete
- log to specify a logfile name
Wrapper script
As most of these values do not change much, if at all, I made a wrapper script called ToggleVServers_TRAIN1.sh to make life a bit easier :
And I created the file with all my hypervisor compute nodes:
Now that we have set this up, we can finally go about changing the partition key. Let’s go and run our toggle script now. The pkey to be “toggled” has value 0xffff:
And remember..
Note that the vServers are fully stopped and then restarted in the process. This can be a problem (downtime for the users of the hosted applications) if there is no application high availability in place, and I have changed the scripts on occasion to delay the stop/restart to some later moment. However, remember that only when the VM is actually restarted does the desired network isolation kick in. By the way, you can also see your vServer going down in Exalogic Control as shown below.
Little problem
There seemed to be a problem with the fist vServer’s shutdown detection by the script, because it actually shutdown quite fast and the script did not pick it up and kept waiting for it until timeout. I suspect this may have something to do with my rather long vServer descriptions in EMOC, as it always works well with brief or empty description fields. When you have renamed a vServer in EMOC this may also occur, as the OVM_simple_name kept by the OVM Manager will have a “-renamed” postfix to it.
Check vm.cfg again
Let’s check our vm.cfg file again:
We see that the partion key has now been updated to specify limited membership for my vServer. Note that the script has also created a backup of the vm.cfg file. Ofcourse it’s OK to check on one vm.cfg file, but it’s a lot of work if you have “toggled” many vServers. For this we can use the CheckVServers.sh script, for which I have likewise created a wrapper script named CheckToggles_TRAIN1.sh:
Run the script and provide the same pkey value as you used for the toggle script. Note that your vServers need to be running for this check to do it’s work.
The script checks where your vServer is running (compute nodes 7 and 3 in my case) and then checks if it is running with the correct partition key for both the links in the bond (to GW1 and to GW2). It provides a lot of output, so for many vServers you would actually grep the output or grep the CheckVServerPKey.log file.
We check to see if our network isolation is now in place:
Thus, we can no longer go over the IPoIB-default network (192.168.10.0/24) to the other vServer, as both are now limited members.
Conclusion
The scripts provided by Oracle Support do a pretty good job if you have many vServers to fix, but it still is some work to set things up, run the procedure, deal with unwilling vServers manually and check the results. Also, you need to collect all the UUID strings of the vServers you want to fix and put them in a file. You cannot get the UUIDs readily from the IaaS CLI, so you need to get a little creative and do a search on the vm.cfg files with “find” through the OVS repository or do some other cleverness. However, if you treat your vServers as disposable and replacable (as is best practice), their UUID’s may have changed after over time and your list file needs to be updated as well.
Apart from the amount of effort, there’s also the downtime issue as every vServer needs to be stopped/restarted for this. It would have been better if we had “opt-out” security instead of “opt-in”, meaning that Exalogic Control should create our vServers as limited members on the IPoIB-default network to begin with, and then let us opt-out if we need a few vServers to be full members after all.
