Archive for September, 2010

Setting up Xen4 has been tricky, because of the move to the latest Linux kernels with paravirt_ops. Compiling it on CentOS, which uses 2.6.18 kernel, and old toolset is a bit more challenging. The following are my notes for setting up Xen4 + pvops 2.6.32.x on CentOS 5.5 on BL460c blades.

Important note: I couldn’t get Xen 4.0.1 release work on CentOS 5.5 on our blades. You have to get the latest Xen and pvops kernel available from http://xenbits.xen.org/xen-4.0-testing.hg and http://www.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git/ (4.0.2-rc1-pre and 2.6.32.21 as of this writing). This may or may not be a problem for you.

1. Install prerequisite packages

   yum -y groupinstall "Development Libraries"
   yum -y groupinstall "Development Tools"
   yum -y install transfig wget texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial glibc-devel
   

2. Install rpmforge repository. Follow the instructions on CentOS Wiki.

   wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm
   rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
   rpm -K rpmforge-release-0.5.1-1.el5.rf.*.rpm
   rpm -i rpmforge-release-0.5.1-1.el5.rf.*.rpm
   yum check-update
   

3. Install mercurial and git

   yum -y install mercurial git 

4. Compile Xen

   hg clone http://xenbits.xen.org/xen-4.0-testing.hg
   cd xen-4.0-testing.hg
   make -j16 xen
   make -j16 tools
   make -j16 stubdom
   

5. Compile pvops kernel

   make kernels
   make -j16 dist
   

   This is the most critical step, and if you don’t configure the kernel correctly, booting will fail badly. make kernels builds the kernel with the default .config. To modify it, change to build-xxx directory and run make menuconfig.

   Make sure to have the following in your config file

   # These are important for CentOS/RHEL
   CONFIG_SYSFS_DEPRECATED=y
   CONFIG_SYSFS_DEPRECATED_V2=y
   
   CONFIG_ACPI_PROCFS=y
   CONFIG_XEN=y
   CONFIG_XEN_MAX_DOMAIN_MEMORY=32
   CONFIG_XEN_SAVE_RESTORE=y
   CONFIG_XEN_PRIVILEGED_GUEST=y
   CONFIG_XEN_PCI=y
   CONFIG_PCI_XEN=y
   CONFIG_XEN_BLKDEV_FRONTEND=m
   CONFIG_NETXEN_NIC=m
   CONFIG_XEN_NETDEV_FRONTEND=m
   CONFIG_XEN_KBDDEV_FRONTEND=m
   CONFIG_HVC_XEN=y
   CONFIG_XEN_FBDEV_FRONTEND=m
   CONFIG_XEN_BALLOON=y
   CONFIG_XEN_SCRUB_PAGES=y
   CONFIG_XEN_DEV_EVTCHN=y
   CONFIG_XEN_BACKEND=y
   CONFIG_XEN_BLKDEV_BACKEND=y
   CONFIG_XEN_NETDEV_BACKEND=y
   CONFIG_XENFS=y
   CONFIG_XEN_COMPAT_XENFS=y
   CONFIG_XEN_XENBUS_FRONTEND=m
   CONFIG_XEN_PCIDEV_FRONTEND=y
   

   If you want to compile xen-evtchn, xen-netback, xen-blkback and xenfs as modules, use the following in your config.

   CONFIG_XEN_DEV_EVTCHN=m
   CONFIG_XEN_BACKEND=y
   CONFIG_XEN_BLKDEV_BACKEND=m
   CONFIG_XEN_NETDEV_BACKEND=m
   CONFIG_XENFS=m
   

6. Install Xen and kernel

   make install-xen
   make install-tools PYTHON_PREFIX_ARG=
   cd build-*
   make modules_install
   make install
   

   For make install-tools the PYTHON_PREFIX_ARG is REQUIRED. On Fedora systems, this is not required.

7. Create initrd (above make install usually takes care of creating an initrd matching your configuration, but just in case)

   depmod 2.6.32.21
   mkinitrd -v -f /boot/initrd-2.6.32.21.img 2.6.32.21
   

8. Update grub

   title Xen 4 + 2.6.32.21
       root   (hd0,0)
       kernel  /xen-4.0.gz dom0_mem=4096M loglvl=all guest_loglvl=all
       module  /vmlinuz-2.6.32.21 root=/dev/cciss/c0d0p1 ro nomodeset console=tty0 console=ttyS0,115200n8
       module  /initrd-2.6.32.21.img
   

9. Add the following to /etc/modprobe.conf to load all the Xen modules

   install xen /sbin/modprobe xen-evtchn; /sbin/modprobe xen-netback; /sbin/modprobe xenfs; /sbin/modprobe xen-blkback; /bin/true
   

   OR, add the following to /etc/rc.local (I had trouble making the modprobe work)

   if ( uname -r | grep "2.6.32.21") ; then
      for mod in xen-evtchn xen-netback xen-blkback xenfs; do
           /sbin/modprobe $mod
      done
      /etc/init.d/xend start
   fi
   

10. Add xenfs to /etc/fstab

        none /proc/xen xenfs defaults 0 0
    

11. Test it

    echo "savedefault --default=X --once" | grub --batch (where X is the entry for Xen)
    reboot
    /etc/init.d/xend start
    xm list
    

Troubleshooting

1. The biggest problem will be kernel not booting properly. There might be various reasons.
   • Make sure to use the right set of configuration options for your hardware
   • Make sure to have all Xen options correctly compiled
   • out_of_memory kernel OOPS
     • CONFIG_XEN_MAX_DOMAIN_MEMORY=32 # change this to a lesser number
     • Add more memory to dom0 with dom0_mem=4096M
   • Network card not appearing – make sure to have the right network module (bnx2x for BroadCom NetXtreme II card) inserted
2. Missing /dev/xen entries. Means that you haven’t loaded the xen-evtchn and xen-gntdev. Run

   modprobe xen-evtchn xen-gntdev
   

   See step 9 above to update /etc/modprobe.conf

3. If you see the following in /var/log/messages

   xenconsoled: Failed to contact xenstore (Connection refused)

   Do step 2 to insert all the appropriate modules, and verify that you can start xend

Installing VMs

1. First, verify that the initrd includes all the necessary Xen frontend modules. This
   is not necessary for hvm VMs, but for paravirtualization it’s crucial to have all
   the right netfront and blkfront modules installed. A sample mkinitrd for domU is
   below

   mkinitrd -v -f /boot/initrd-2.6.32.21-domU.img  --with=blktap \
           --with=xen-blkback --with=xenfs 2.6.32.21
   

2. Install virt-manager

   yum install virt-manager
   

3. Configure /etc/xen/xend-config.sxp to enable access through libvirt and virt-manager. Add the following lines

   (xend-http-server yes)
   (xend-unix-server yes)
   (xend-port        8000)
   (xend-relocation-port 8002)
   

4. Restart xend

   /etc/init.d/xend restart
   

5. Start virt-manager as root
6. Install a VM as usual

References

1. http://wiki.xensource.com/xenwiki/2.6.18-to-2.6.31-and-higher
2. http://wiki.xensource.com/xenwiki/XenParavirtOps. This is the most authoritative reference for compiling the pvops kernel.
3. http://wiki.xensource.com/xenwiki/Xen4.0
4. http://bderzhavets.wordpress.com/2010/04/24/set-up-ubuntu-10-04-server-pv-domu-at-xen-4-0-dom0-pvops-2-6-32-10-kernel-dom0-on-top-of-ubuntu-10-04-server/ (this is Ubuntu specific, but has some useful details)