Setting up limited membership for VServers (part 1)
Published on: Author: Jos Nijhoff Category: IT development and operationsThis blogpost discusses network security. I will demonstrate how vServers can be switched to limited membership on the IPoIB-default network, and why you probably need to do this.
In the last week of april of this year, my colleague Gerrit Boorsma and I were invited to participate in the Exalogic 12c beta event in Santa Clara. I found this to be a unique, wonderful and very stimulating experience. Oracle has quite some cool stuff in store for us with the 12c EECS stack, and I hope we will hear more anouncements at OpenWorld 2015, where Qualogy will also be participating, including my colleague Michel Schildmeijer.
The Exalogic 12c event was very well organized and I met many prominent product managers and product engineers in person that I only knew from e-mail contact, and learned their view on the future of Exalogic. Also, it was great to discuss my Exalogic experiences with peers working for customers around the globe. I got many positive remarks on my Exalogic blog in Santa Clara which encouraged me to pick up writing blog posts again, so here I am with a number of posts on security and Exalogic. The first post with be discussing a bit of network security.
Part 1: background and preparation
As most of you will know, Infiniband networks in Exalogic come with the concept of limited vs full membership with regard to Infiniband partitions, which are the analogues of VLANs in the Ethernet world. Infiniband partitions can have “limited” members and “full” members*:
- Limited members
Limited members of a given partition cannot communicate with other limited members. They can only connect to full members in the same partition (“one-to-all”).
- Full members
Full members can connect to both limited and full members in the same partition (“all-to-all”).
* Actually there’s also a third kind (“both”), but I’ll leave that one out for reasons of simplicity and lack of relevance.
Limited members can be viewed as network “clients” (e.g. the compute nodes) while full members can be viewed as “servers” (e.g. the ZFS storage and gateway switches). For more information, see Donald Forbes’ Exablurb article on this.
A virtualized Exalogic comes with it’s own “prefab” virtual datacenter, so it comes with a pre-built base network design having seperate physical and virtual storage networks, and several admin networks. These are all logical networks going over the unified Infiniband backbone that physically connects all the hardware components, and they rely on Infiniband partitions to enforce strict network seperation where it is needed. Naturally, this base design can be extended using Exalogic Control to fit your exact deployment needs.
3, 4 or 5 networks
Usually, when vServer are provisoned by Exalogic Control to run our middleware and applications, they come with at least three networks: an “uplink network” to connect to it (either public or private), a storage network, and an admin network. More often than not the middleware and applications you will deploy in it need to connect to a database, and you also need a network for that, bringing you up to four. If you want your solution to be scalable and support high(er) availabilty, you need clustering (Weblogic or otherwise), and you add in another fast private network for that, which could bring you up to five.
So commonly you have a set of private networks in your vServer, and some of these are “utility networks” for mounting storage, for management and metering by Exalogic Control, and for connecting to a database on Exadata.
- IPoIB-vserver-shared-storage: for mounting storage provisioned on the internal ZFS appliance.
- IPoIB-virt-admin: added in and used by Exalogic Control for all virtualization management.
- IPoIB-default: connect to other machines in the same IB fabric, mostly Exadata
As we view Exalogic vServers (Oracle Virtual Machines) as clients on these networks, they are provisioned as limited members on both the IPoIB-vserver-shared-storage and the IPoIB-virt-admin networks. If you try to open an ssh session from one vServer to another on these networks, you will not succeed eventhough the ssh daemon listens on this network.
Potential security concern
Curiously, Exalogic Control always provisions vServers to be full members on the IPoIB-default network. This is documented in MOS note 1908498.1. I can’t really see a rationale for this, as the IPoIB-default network is for connecting to databases mostly, and middleware and applications act as database clients, and are likely not intended as database servers. You may ask why this is important. This is important because it’s a potential security concern. Via the IPoIB-default network, vServers in one vDC account (say tenant X) can access the vServers that belong to another vDC account (say tenant Y). Thus, you can use this for “ssh hopping” to other nodes, which may not be desirable and could be a finding in your next security audit.
Good news and bad news
Luckily there is some good news: in the MOS note, Oracle provides some scripts to switch your vServers back to limited membership on the IPoIB-default network if needed, in fact they are nicknamed “toggle scripts”. So let’s dive in and see how it works.
Getting ready
First of all, you will need the Exalogic IaaS API and DCLI (Infrastructure-as-a-Service API and the Distributed Command Line Interface) in your Exalogic Control host as the scripts leverage its functionality. If it is not yet installed in your EC1 vServer, you will need to install it, as was the case for me:
Only the webservices appeared to be installed. Also, I found that the default Java version in our Exalogic Control was not up to par:
So first I downloaded Java 7 from the public yum repo and installed it:
Check again
Update : Don’t go and install a newer Java version like 1.7.0_85 as I did first, or you’ll get awful handshake errors when calling the IaaS API later on. The error you get is : Could not send HTTP request: Remote host closed connection during handshake (120122)
Took me quite some to figure it out, but after downgrading to Java version 1.7.0_65 it worked fine again.
Then I installed the IaaS packages, which can be found in the EC1 vServer itself:
Also, installed DCLI as it was not there yet (downloaded the zip from the MOS note):
So now that we have set up the necessary prerequisites, we can get down to business, which will be the subject of part 2 of this blogpost.
