How to Prepare a XenServer Host’s Network Interfaces

In Chapter 2 of “Citrix XenServer Design – Designing XenServer Network Configurations” Citrix describes how the physical network interfaces (PIF’s) of a XenServer host behave:

Impact of Pools on XenServer Networking

Networking is a pool-level feature in XenServer. When you change networking on the pool master, XenServer synchronizes all hosts in a pool to use the same network settings. As a result, for XenServer to operate correctly, you must ensure that network settings match across all hosts in the pool, including:

  • Which NICs are bonded
  • Which NICs are configured as the primary management interface
  • Which NICs connect to storage
The networks to which NIC’s connect must be the same on the corresponding NIC’s on each host in the pool.

Ideally, you should add all desired hosts to the pool before configuring any network settings. Pooling the hosts before configuring networking creates cleaner records in XenServer’s internal networking-configuration database.

After creating a new pool or joining a host to an existing pool, XenServer automatically replicates the network settings on the master to the joining hosts. When you use XenCenter to make networking changes, XenCenter changes the other hosts to match the newly modified host. When you use the CLI to change network settings, you must either:

  • Change each host manually to match the modified host’s settings
  • Make the change on the pool master and restart all the member hosts in the pool
XenServer requires network settings to match across the pool because of features that use live migration, such as XenMotion, High Availability, and Workload Balancing. These features enable the physical server hosting a VM to change at any time, and possibly automatically without your intervention. Therefore, the VMs must be able to access all of their target networks regardless of which host XenServer moves them on to.

For this reason, it is critical to have and maintain an identical physical cabling, NIC, and switch configuration for each host across the pool. Likewise, Citrix strongly recommends changing the physical configuration on all hosts in a pool before changing network settings on the pool.

Important: After joining the hosts to the pool, check the primary management interface on each member host to make sure that it has its own unique IP address and/or set the correct static IP address.

Chapter 2 of “Citrix XenServer Design – Designing XenServer Network Configurations
I’ve highlighted portions of the passage for emphasis but it’s clear – even without the highlighting – that the PIF’s of a XenServer host must be configured and connected identically across all members of the pool. What’s not obvious from the passage is the concept of interface names and how naming affects the way that XenServer uses the interfaces.

What does it mean that the PIF’s “must be configured and connected identically across all members of the pool”? To put it simply: If the interface named ‘eth0’ is configured as the Primary Management Interface (PMI) on one XenServer host; then the interface named ‘eth0’ must be configured as the PMI on all pool members. This same principle applies to all of the interfaces in the pool: If the interfaces named ‘eth3’ & ‘eth4’ are combined (i.e., bonded) and used to access network storage on one XenServer host, then the interfaces named ‘eth3’ & ‘eth4’ must be combined (i.e., bonded) and used to access network storage on all pool members.

How PIF’s are named determines how XenServer will configure and use those interfaces so controlling the naming process is critical when installing XenServer onto your host systems.

How does XenServer determine the way interfaces are named? Initially, the interfaces are named somewhat arbitrarily during installation. For example, this is how the XenServer installation script initially configured PXS Lab System #2:
[root@xs02-pdx0 ~]# ifconfig -a | grep Link eth0 Link encap:Ethernet HWaddr 00:1B:78:5C:74:34 eth1 Link encap:Ethernet HWaddr 00:1B:78:5C:74:35 eth2 Link encap:Ethernet HWaddr 00:26:55:D2:EE:50 eth3 Link encap:Ethernet HWaddr 00:26:55:D2:EE:51 eth4 Link encap:Ethernet HWaddr 00:1F:29:C4:9D:BA eth5 Link encap:Ethernet HWaddr 00:1F:29:C4:9D:C2
There's no opportunity during installation to actually influence the way in which XenServer names interfaces. XenServer administrators may only select the single interface that will serve as the host's PMI after installation is complete.

interfaces

How can you control the way interfaces are named? The way that interfaces are named can be controlled using the /etc/sysconfig/network-scripts/interface-rename-data/static-rules.conf configuration file. The format is quite simple. For example, the status-rules.conf file for PXS Lab System #2 looks like this:
eth0: mac = "00:1F:29:C4:9D:BA" eth1: mac = "00:1F:29:C4:9D:C2" eth2: mac = "00:26:55:D2:EE:50" eth3: mac = "00:26:55:D2:EE:51" eth4: mac = "00:1B:78:5C:74:35" eth5: mac = "00:1B:78:5C:74:34"
The static-rules.conf file may be created before, during, or after the installation of XenServer:

  1. Creating the static-rules.conf before installation:

    If you don't already have Linux installed on the host and you'd prefer to map the interfaces to the names before beginning the installation process, I recommend using a Linux Live CD to survey the host's interfaces, create the static-rules.conf file, and store the file to a USB flash drive for installation during or after the installation process.

  2. Creating/Installing the static-rules.conf during installation:

    During installation, administrators are able to interact with the system via an alternate Linux shell by pressing <Alt+F2> anytime during the installation process. By creating/installing the static-rules.conf file during the installation process, XenServer administrators may save valuable time by avoiding the costly step of installing the static-rules.conf file after XenServer installation and then rebooting a second time.

    • To create the static-rules.conf during installation:

      At the end of the installation process, the installer will pause and ask to reboot the host.

      installer
      1. Press <Alt+F2> to open the installer's other shell.
      2. Mount the host's root partition at /tmp/root:
        mount_root
      3. Execute the scripts below to create the static-rules.conf configuration file and to verify the ordering of the interfaces.
        create
      4. Copy the static-rules.conf file to it's proper location (/tmp/root/etc/sysconfig/network-scripts/interface-rename-data/):
        install
      5. Either...
        • Execute the `reboot` command to reboot the host, or
        • Press <Alt+F1> to return to the installer and press 'Enter' to reboot the host.
    • To install a static-rules.conf file that you've already created:

      At the end of the installation process, the installer will pause and ask to reboot the host.

      installer
    1. Press <Alt+F2> to open the installer's other shell.
    2. Mount the host's root partition at /tmp/root:
      mount_root
    3. Mount the USB flash drive at /tmp/mnt:
      mount_usb
    4. Copy the static-rules.conf file from the USB flash drive mount point (i.e., /tmp/mnt) to /tmp/root/etc/sysconfig/network-scripts/interface-rename-data/:
      copy
    5. Either...
      • Execute the `reboot` command to reboot the host, or
      • Press <Alt+F1> to return to the installer and press 'Enter' to reboot the host.
  3. Creating/Installing the static-rules.conf after installation:

    After installation is complete, the static-rules.conf file may be copied from a USB flash drive (if it's already been created and saved to a USB flash drive) or created using these steps.

The Scripts: The list of the host's interfaces can be added to the static-rules.conf and formatted all at once using this short script:
/sbin/ifconfig | /usr/bin/awk '/eth/ { print $1 ": mac = \"" $5 "\"" }' > /tmp/static-rules.conf
The ordering of the interfaces by the static-rules.conf file can be verified with this short script:
for x in `/usr/bin/awk '{ print $1 }' /tmp/static-rules.conf | /usr/bin/tr -d :` ; \ do echo -e "Now identifying interface $x" ; \ /usr/sbin/ethtool -p $x 3 ; \ done
(This command will physically identify each interface in sequential fashion by using ethtool to flash the interface's status light for 3 seconds. If the interfaces are not ordered correctly, you can modify the static-rules.conf file using vi or any other CLI-based editor and re-test until you are satisfied.)

After the file has been created and the ordering of the interfaces has been verified: Move the file to the /etc/sysconfig/network-scripts/interface-rename-data/ directory and reboot the host.

Once the static-rules.conf file has been created, I recommend saving it to a USB flash drive for future use. If you plan on installing XenServer on multiple machines and you're working in an environment with multiple hosts that you might need to re-install multiple times (e.g., a lab environment) you might consider creating the static-rules.conf file beforehand and storing it on a USB flash drive using the host's hardware asset tag number to name the file. (That's how we do it here in the PXS labs!) Otherwise, you're left to create the static-rules.conf and verify the ordering every time that you re-install XenServer!