Monthly Archives: September 2010

How to Setup Xen4 and pvops 2.6.32.x on CentOS 5.5

Setting up Xen4 has been tricky, because of the move to the latest Linux
kernels with paravirt_ops. Compiling it on CentOS, which uses 2.6.18 kernel, and old toolset is a bit more challenging. The following are my notes for setting up Xen4 + pvops 2.6.32.x on CentOS 5.5 on BL460c blades.

Important note: I couldn’t get Xen 4.0.1 release work
on CentOS 5.5 on our blades. You have to get the latest Xen and pvops kernel
available from http://xenbits.xen.org/xen-4.0-testing.hg and
http://www.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git/ (4.0.2-rc1-pre
and 2.6.32.21 as of this writing). This may or may not be a problem for you.

  1. Install prerequisite packages
    yum -y groupinstall "Development Libraries"
    yum -y groupinstall "Development Tools"
    yum -y install transfig wget texi2html libaio-devel dev86 glibc-devel
    e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs
    pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common
    qemu-img mercurial glibc-devel
  2. Install rpmforge repository. Follow the instructions on CentOS
    Wiki
    .

    wget
    
    http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm
    
    rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
    rpm -K rpmforge-release-0.5.1-1.el5.rf.*.rpm
    rpm -i rpmforge-release-0.5.1-1.el5.rf.*.rpm
    yum check-update
  3. Install mercurial and git
    yum -y install mercurial git
  4. Compile Xen
    hg clone http://xenbits.xen.org/xen-4.0-testing.hg
    cd xen-4.0-testing.hg
    make -j16 xen
    make -j16 tools
    make -j16 stubdom
  5. Compile pvops kernel
    make kernels
    make -j16 dist

    This is the most critical step, and if you don’t configure the
    kernel correctly, booting will fail badly. make kernels builds the
    kernel with the default .config. To modify it, change to build-xxx directory and
    run make menuconfig.

    Make sure to have the following in your config file

    # These are important for CentOS/RHEL
    CONFIG_SYSFS_DEPRECATED=y
    CONFIG_SYSFS_DEPRECATED_V2=y
    
    CONFIG_ACPI_PROCFS=y
    CONFIG_XEN=y
    CONFIG_XEN_MAX_DOMAIN_MEMORY=32
    CONFIG_XEN_SAVE_RESTORE=y
    CONFIG_XEN_PRIVILEGED_GUEST=y
    CONFIG_XEN_PCI=y
    CONFIG_PCI_XEN=y
    CONFIG_XEN_BLKDEV_FRONTEND=m
    CONFIG_NETXEN_NIC=m
    CONFIG_XEN_NETDEV_FRONTEND=m
    CONFIG_XEN_KBDDEV_FRONTEND=m
    CONFIG_HVC_XEN=y
    CONFIG_XEN_FBDEV_FRONTEND=m
    CONFIG_XEN_BALLOON=y
    CONFIG_XEN_SCRUB_PAGES=y
    CONFIG_XEN_DEV_EVTCHN=y
    CONFIG_XEN_BACKEND=y
    CONFIG_XEN_BLKDEV_BACKEND=y
    CONFIG_XEN_NETDEV_BACKEND=y
    CONFIG_XENFS=y
    CONFIG_XEN_COMPAT_XENFS=y
    CONFIG_XEN_XENBUS_FRONTEND=m
    CONFIG_XEN_PCIDEV_FRONTEND=y

    If you want to compile xen-evtchn, xen-netback, xen-blkback and xenfs as
    modules, use the following in your config.

    CONFIG_XEN_DEV_EVTCHN=m
    CONFIG_XEN_BACKEND=y
    CONFIG_XEN_BLKDEV_BACKEND=m
    CONFIG_XEN_NETDEV_BACKEND=m
    CONFIG_XENFS=m
  6. Install Xen and kernel
    make install-xen
    make install-tools PYTHON_PREFIX_ARG=
    cd build-*
    make modules_install
    make install

    For make install-tools the PYTHON_PREFIX_ARG is REQUIRED. On Fedora systems,
    this is not required.

  7. Create initrd (above make install usually takes care of creating an initrd
    matching your configuration, but just in case)

    depmod 2.6.32.21
    mkinitrd -v -f /boot/initrd-2.6.32.21.img 2.6.32.21
  8. Update grub
    title Xen 4 + 2.6.32.21
        root   (hd0,0)
        kernel  /xen-4.0.gz dom0_mem=4096M loglvl=all guest_loglvl=all
        module  /vmlinuz-2.6.32.21 root=/dev/cciss/c0d0p1 ro nomodeset console=tty0
    console=ttyS0,115200n8
        module  /initrd-2.6.32.21.img
  9. Add the following to /etc/modprobe.conf to load all the Xen modules
    install xen /sbin/modprobe xen-evtchn; /sbin/modprobe xen-netback;
    /sbin/modprobe xenfs; /sbin/modprobe xen-blkback; /bin/true

    OR, add the following to /etc/rc.local (I had trouble making the modprobe
    work)

    if ( uname -r | grep "2.6.32.21") ; then
       for mod in xen-evtchn xen-netback xen-blkback xenfs; do
            /sbin/modprobe $mod
       done
       /etc/init.d/xend start
    fi
  10. Add xenfs to /etc/fstab
        none /proc/xen xenfs defaults 0 0
  11. Test it
    echo "savedefault --default=X --once" | grub --batch (where X is the
    entry for Xen)
    reboot
    /etc/init.d/xend start
    xm list

Troubleshooting

  1. The biggest problem will be kernel not booting properly. There might be
    various reasons.

    • Make sure to use the right set of configuration options for your hardware
    • Make sure to have all Xen options correctly compiled
    • out_of_memory kernel OOPS
      • CONFIG_XEN_MAX_DOMAIN_MEMORY=32 # change this to a lesser number
      • Add more memory to dom0 with dom0_mem=4096M
    • Network card not appearing – make sure to have the right network
      module (bnx2x for BroadCom NetXtreme II card) inserted
  2. Missing /dev/xen entries. Means that you haven’t loaded the
    xen-evtchn and xen-gntdev. Run

    modprobe xen-evtchn xen-gntdev

    See step 9 above to update /etc/modprobe.conf

  3. If you see the following in /var/log/messages
    xenconsoled: Failed to contact xenstore (Connection refused)

    Do step 2 to insert all the appropriate modules, and verify that you can
    start xend

Installing VMs

  1. First, verify that the initrd includes all the necessary Xen frontend
    modules. Thisis not necessary for hvm VMs, but for paravirtualization it’s crucial
    to have allthe right netfront and blkfront modules installed. A sample mkinitrd for domU
    isbelow

    mkinitrd -v -f /boot/initrd-2.6.32.21-domU.img
    --with=blktap \
            --with=xen-blkback --with=xenfs 2.6.32.21
  2. Install virt-manager
    yum install virt-manager
  3. Configure /etc/xen/xend-config.sxp to enable access through libvirt and
    virt-manager. Add the following lines

    (xend-http-server yes)
    (xend-unix-server yes)
    (xend-port        8000)
    (xend-relocation-port 8002)
  4. Restart xend
    /etc/init.d/xend restart
  5. Start virt-manager as root
  6. Install a VM as usual

References

  1. http://wiki.xensource.com/xenwiki/2.6.18-to-2.6.31-and-higher
  2. http://wiki.xensource.com/xenwiki/XenParavirtOps. This is the most
    authoritative reference for compiling the pvops kernel.
  3. http://wiki.xensource.com/xenwiki/Xen4.0
  4. http://bderzhavets.wordpress.com/2010/04/24/set-up-ubuntu-10-04-server-pv-domu-at-xen-4-0-dom0-pvops-2-6-32-10-kernel-dom0-on-top-of-ubuntu-10-04-server/
    (this is Ubuntu specific, but has some useful details)

Using kpartx to Mount VM Disk Images

If you are using a full disk image for your VM, instead of using a file system, you cannot simply mount it directly using

mount -o loop d0.img

Use the following commands to see the partitions in your disk, and then mount them.

# kpartx -av d0.img
add map loop0p1 : 0 29333504 linear /dev/loop0 2048
add map loop0p5 : 0 1380352 linear /dev/loop0 29337600
# mount /dev/mapper/loop0p1 /mnt
# ls /mnt
bin   cdrom  etc   initrd.img      lib         media  opt   root  selinux  sys  usr  vmlinuz
boot  dev    home  initrd.img.old  lost+found  mnt    proc  sbin  srv      tmp  var
# unmount /mnt
# kpartx -d d0.img

The last two commands are to unmount and delete the partition mappings.

How to Setup Xen4 and pvops 2.6.32.x on CentOS 5.5 using GITCO repository

See my howto on setting up Xen4 + pvops 2.6.32.x first before following this.

  1. Do steps 1, 2 and 3 described in the howto
  2. Install Xen 4.0.1 from GITCO repository. The benefit of this is that all
    the required packages,Xen enabled Linux kernel (2.6.18) and libvirt are correctly installed and
    setup

       wget http://www.gitco.de/repo/GITCO-XEN4.0.1_testing_x86_64.repo
       yum -y install xen
  3. Get pvops kernel
       git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
       linux-2.6-xen
       cd linux-2.6-xen
       git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x
  4. Build the kernel to match your configuration
    make menuconfig

Follow the remaining steps in the howto
to setup grub, and test it.