Wednesday, January 6, 2010

Datacenter Turf Wars

One of the interesting new developments with the advent of large-scale virtualization in datacenters is a turf war between system administrators and network administrators. Whether implementing Intel-based virtual machines using a hypervisor such as VMware or Unix-based VM's using a hypervisor like IBM's Power Virtualization, there is no escaping one fact of life: the network is migrating from hardware into software. When you are connecting servers that are running virtually inside the same machine, there's no reason to go through a physical path out to a physical switch. Instead, servers talk to one another via a logical switch implemented in software.

The upshot of this is a migration of control over network configuration from the network administrator to the server administrator. Just as a network administrator would be loath to dole out userid's for their network routers and switches to server admins, server admins are not exactly eager to let network administrators get into their hypervisor configuration system. As a result, the job of configuring these software switches has become a bit of a hot potato.

I was recently a witness to one such bun fight when a senior architect presented a plan to adopt large-scale virtualization on an IBM AIX platform to his company's team of network administrators. The network admins had worked long and hard over the past few years to achieve a very hygenic network with all the application groups highly isolated from one another and now they could see all their physical pipes going into a single box with the server administrator controlling how they connected to one another in the software switch.

Needless to say, the network administrators were horrified that all their good works might be scrambled by a server admin who may not have been aware of the issues of separating church from state.

Fortunately, a senior IBM systems engineer was in attendance at the meeting and he'd seen all this before. He spoke up and outlined a solution that seemed to work for both parties:

  • The roles of the network administrators and systems administrators would be changed slightly to accommodate the change in the ecosystem.
  • Network administrators would retain the responsibility for network design, policy and governance.
  • A limited responsibility for implementation of the network design would shift to the systems administrator in the case where configuration of the software switch was integrated with the configuration of the hypervisor.
  • The systems administrators would be responsible for implementing network design and policy.
  • The server administrators would be responsible for auditing the work of the systems administrators to ensure that it conformed to design and policy.
In this way, the network guys retained control over what matters most: policy and governance. The systems guys retained their control over the hypervisor configuration as long as they followed network policy.

Everyone lived happily ever after.