All posts by admin

Challenges in Mobile (Smart Phone) Virtualization

We have virtualization on servers, desktops, laptops and what’s next? Naturally, the “smart phone”. Today’s mobile phones have become quite powerful that one can easily virtualize them and run multiple operating systems. But,

Why do you need smart phone virtualization?

Similar to desktop virtualization, the use-case is not that clear, but there are some benefits that can be gained.

  1. Reducing vendor lock-in. Can you run iPhone applications on an Android phone? Currently, no, but with virtualization this might be possible.
    There are issues with Apple (or other vendors)’s mobile OS licensing that make it illegal to do so, right now. You can only run iOS on Apple approved devices, but this may change in the future. Many old phones have legacy applications and libraries that can be run in virtualized sandboxes as well.
  2. Mobile cloud computing. You can clearly access the so-called “cloud” from a smart phone, but if smart phones are virtualized, mobile applications can be migrated to a cloud and back seamlessly. This is
    still a pipe-dream and a few research platforms (MAUI
    and CloneCloud) are proposed to make this a reality. The benefits of doing this are enormous: one can potentially access infinite compute and storage resources on the smart phone.
  3. Security. This is probably the strongest argument for mobile
    virtualization. The popularity of smart phones is making them a prime target for
    malicious hackers. MalwareViruses and other nasty stuff are making their way into smart phones, which currently don’t have much protection against them. It’s not easy to run Anti-virus software on a mobile phone, as the current AV software is very resource hungry making it unsuitable for smart phones.
  4. Isolation. This is related to security. It will be possible to have different user profiles with different levels of access (say work and home) with mobile virtualization. VMware recently demonstrated, how multiple user profiles can co-exist in the same phone at Mobile Congress 2011.
  5. Ease of development. This is an indirect benefit for mobile phone
    developers. Virtualization can make the development of mobile applications
    easier by removing the need for maintaining multiple versions of the same
    application for different operating systems.


Before you start seeing virtualized mobile phones, there are still plenty of
challenges that need to be solved.

  1. Limited resources. Though, the CPUs on mobile phones are getting more
    and more powerful, they still have limited battery and storage space.
    Virtualization usually adds more burden to the resources and is not necessarily
  2. Poor connectivity . Mobile cloud computing sounds like a great idea,
    but mobile phones often experience variable connectivity. It’s not easy to
    implement disconnected
    for storage and for seamlessly migrating computation.
  3. Poor performance. Virtualization adds performance overheads, which
    are still being researched in server virtualization environments. It’s not
    clear whether the overheads are worth the effort on a mobile phone.

These challenges can be overcome and it’s only a matter of time before
smart phones are virtualized. Some current players in the market are: VMware’s
mobile virtualizaiton platform (MVP)
and OKL4
. OKL4 claims to be the first open source mobile hypervisor, which
is an exciting development that can lead to advances in mobile

How to Access Your Web Server within Your Local Network

If you are hosting a web server at home and using your router’s port
forwarding to redirect requests from WAN port to WebServerHost:80, there’s
an annoying problem in some routers: you cannot access the web server from
inside the network
. There are multiple reasons why this might happen.

  1. The router assumes that all traffic from internal hosts to port 80 are for
    the router admin interface. This is annoying, as you just get to the
    router’s web admin page. Solution: you can port forward from
    another port (say 8000) to your WebServer’s port 80. Now, you can access
    the website with You are out of luck, if your router
    does’t allow port forwarding to a different port (say
    WAN:8000->WebServerHost80 instead of WAN:80->WebServerHost80).
  2. You have two or more routers with different subnets and the routing is
    messed up. This is a difficult problem to diagnose and often the router’s
    admin interfaces are not that amenable to debugging routing problems. I suggest
    changing your firmware to DD-WRT, if you can.
  3. Port forwarding doesn’t work for incoming traffic from local ports on
    the router. I think this is probably the most common reason. Without DD-WRT,
    there’s no easy way to fix this. Solution: One simple way to solve
    this is to set a hard-coded DNS entry in /etc/hosts (Linux) or
    c:\WINDOWS\system32\drivers\etc (Windows) as follows.


    192.168.x.x is the internal IP address of the WebServerHost

Best Resources to Learn about Linux Kernel Internals

The Source

The best resource is the kernel source.


Obviously, it’s not that easy to dive into thousands of lines of code.
I suggest starting with reading the books explaining Linux kernel in

Other resources

  • LWN’s kernel page has lots of great articles explaining kernel internals
  • TLDP’s TLK. Somewhat outdated, but useful.
  • Linux Journal’s KernelKorner has some excellent articles, most of
    which are freely available online.
  • The Linux Kernel Hackers’ Guide, compiled by Michael K. Johnson of Red Hat fame. Includes
    among otherdocuments selected Q/A’s from the linux-kernel mailing list.


Lists of links

Virtualization Bibliography

Virtualization is a hot topic and the field is growing at a rapid pace. There
are hundreds of papers on each sub-topic, and this list is intended to be a good
starting point for someone starting in virtualization research.


These are good starting points, if you are just learning about

  1. Survey of virtualization machine research. Robert P. Goldberg. IEEE
    Computer, June 1974, pp 34-45. [PDF]. One of the oldest surveys about virtualization research.
  2. A Survey on Virtualization Techniques. Susanta Nanda and Tzi-cker
    Chiueh [PDF].
    A more modern survey of virtualization techniques. The paper lists many of the
    techniques, but doesn’t really explain how they are different etc. Yet,
    this is a good read for budding virtualization researchers.
  3. Virtual Machines: Versatile Platforms for Systems and Processes .
    Jim Smith Ravi Nair. Amazon
    book link

Virtualization Overview

These are the papers that will help in understanding fundamentals and
concepts of virtualization.

  1. Xen and the art of virtualization. Barham This is the
    classic SOSP paper on para-virtualization. [PDF].
  2. When Virtual is Better Than Real. Peter Chen and Brian Noble. This
    is a great short article explaining the benefits of virtualization. ACM
  3. Disco: Running Commodity Operating Systems on Scalable Multiprocessors. E.
    Bugnion, S. Devine, and M. Rosenblum. This paper is considered to be the first
    paper that revived the virtualization concepts pioneered by IBM. ACM
  4. Running multiple operating systems concurrently on an IA32 PC using
    virtualization techniques. Kevin Lawton. This is a great article on the
    difficulties involved in virtualizing the x86 platform. link.
  5. Formal requirements for virtualizable third generation architectures. Popek
    and Goldberg. The classic paper explaining the requirements for virtualizing a
    specific ISA. ACM
    . If you want light reading, check the Wikipedia
    explanation of the requirements
  6. Scale and Performance in the Denali Isolation Kernel. Andrew Whitaker,
    Marianne Shaw, and Steven D. Gribble. Denali is another example of early
    virtualization papers that rejuvenated the research. Denali shows the containers
    (or vservers) concept, which is used in OpenVZ, KVM and VServers.
  7. The Exokernel Operating System Architecture. Dawson Engler’s thesis.
    This is more of OS research, but a great read for understanding some of the
    techniques (thin hypervisor, pass through etc.) that are used in modern
    virtualized systems.

CPU Virtualization

Virtualization of CPU is provided by a CPU scheduler that provides the
illusion of multiple CPUs (or VCPUs). Scheduling has a long and rich history.
Below are a few links in relation to virtualization.

  1. Xen’s
    credit scheduler
    . Xen’s credit scheduler is a proportional fair
    scheduler, which is pretty big improvement (especially in SMP environments) over
    the old SEDF
  2. Xen’s scheduler is partly inspired by Carl Waldspurger’s
    lottery scheduling and the paper: Lottery Scheduling: Flexible
    Proportional-Share Resource Management by Carl Waldspurger is a must read. [PDF].
  3. Comparison of the Three CPU Schedulers in Xen – Ludmila Cherkasova,
    Diwaker Gupta, Amin Vahdat – Perfomance Evaluation Review. Vol 35, Number
    2. Sept 3, 2007. This provides a great comparison of CPU schedulers and provides
    insight on how handling I/O is complicated. [PDF].

Network Virtualization

Network virtualization is gaining momentum in recent years. I would just
point you to the following survey as a starting point.

  1. Survey of network virtualization. N.M. Mosharaf Kabir Chowdhury and
    Raouf Boutaba. [PDF].
    This is pretty comprehensive and well-written survey on network virtualization.
    Mosharaf also has a
    nice network
    , listing lots of useful papers.

Device Virtualization

  1. Xen’s
    device driver virtualization
  2. Virtualizing I/O Devices on VMware Workstation’s Hosted Virtual
    Machine Monitor. Jeremy Sugerman, Ganesh Venkitachalam, and Beng-Hong Lim. As
    the name suggests, this paper explains VMware Workstation’s mechanisms for
    virtualizing devices. [PDF].
  3. GPU Virtualization on VMware’s Hosted I/O Architecture. Micah Dowty,
    Jeremy Sugerman. This paper by VMware folks explains how VMware implements GPU
    virtualization. Great starting point for GPU virtualization. [PDF].
    Andres’ paper on
    indepedent graphics acceleration
    is a good follow-up reading.

Memory Virtualization

  1. Memory Resource Management in VMware ESX Server. Carl A. Waldspurger. [PDF].
    Classic paper on memory virtualization. This paper introduces memory ballooning,
    which is a great technique used in commercial platforms as well.
  2. Difference engine: harnessing memory redundancy in virtual machines. Diwakar
    Gupta [PDF].
    A great follow up paper that talks about sharing memory across VMs.
    Incidentally, both these papers have received best paper awards.


  1. Live Migration of Virtual Machines. Clark One of the first academic
    papers discussing how to do live migration, which is implemented in Xen. VMware
    has supposedly implemented vmotion before this was published. [PDF]
  2. The design and implementation of Zap: a system for migrating computing
    environments. Osman Zap predates Xen paper and talks about sandboxing
    processes that can be migrated. This is more like container virtualization
    rather than full system virtualization. ACM
  3. SnowFlock: rapid virtual machine cloning for cloud computing. Andres
    Lagar-Cavilla Greatp paper to read to understanding cloning of virtual
    machines. Snowflock

Resource Management/Automation

  1. Carl Waldspurger’s PhD thesis is a great place to start for
    understanding resource management. MIT link.
  2. Automated control of multiple virtualized resources. Padala
    Disclaimer: this is my own paper :-) It’s a good place to start if you want to
    learn about automating resource management for virtual machines. The control
    theory aspects are a bit complex, but you can ignore them and focus on the
    issues in automating resource management. [PDF].
  3. Black-box and Gray-box Strategies for Virtual Machine Migration. Clark This is a follow up for live migration paper. The paper talks about
    strategies for automating the migration of VMs to meet specific goals. USENIX

How to Build Custom Xen Kernel Based on CentOS/RHEL Xen Kernel

The HOWTO on compiling CentOS kernel based on RHEL code base (currently 2.6.18) is overly complicated. Follow the following steps to build a custom Xen kernel, maybe with different kernel configuration.

  • Setup the rpmbuild directory, and install prerequisite packages.
    mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
    sudo yum -y install rpm-build redhat-rpm-config unifdef kernel-devel kernel-xen-devel
  • Grab the kernel source RPM, you want to modify and compile, and install it.
    wget -c`uname -r`.src.rpm
    rpm -i kernel-`uname -r`.src.rpm
  • Create the spec file for the downloaded kernel
     cd ~/rpmbuild/SPECS
     rpmbuild -bp --target=`uname -m` kernel-2.6.spec
  • Modify the config to your needs. The config file are stored in ~/rpmbuild/SOURCES.
  • Modify the buildid. This is a good practice, if you are making extensive changes.
     cd ~/rpmbuild/SPECS
     sed -ibak -e 's/#% define buildid/%define buildid .ppadala/' kernel-2.6.spec
  • Build the kernel
     cd ~/rpmbuild/SPECS
     sudo rpmbuild -bb --with xenonly --without debug --without debuginfo --target=`uname -m` kernel-2.6.spec
  • This will generate a RPM that you can install on other machines as well.

How to Setup Xen4 and pvops 2.6.32.x on CentOS 5.5

Setting up Xen4 has been tricky, because of the move to the latest Linux
kernels with paravirt_ops. Compiling it on CentOS, which uses 2.6.18 kernel, and old toolset is a bit more challenging. The following are my notes for setting up Xen4 + pvops 2.6.32.x on CentOS 5.5 on BL460c blades.

Important note: I couldn’t get Xen 4.0.1 release work
on CentOS 5.5 on our blades. You have to get the latest Xen and pvops kernel
available from and (4.0.2-rc1-pre
and as of this writing). This may or may not be a problem for you.

  1. Install prerequisite packages
    yum -y groupinstall "Development Libraries"
    yum -y groupinstall "Development Tools"
    yum -y install transfig wget texi2html libaio-devel dev86 glibc-devel
    e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs
    pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common
    qemu-img mercurial glibc-devel
  2. Install rpmforge repository. Follow the instructions on CentOS

    rpm --import
    rpm -K rpmforge-release-0.5.1-1.el5.rf.*.rpm
    rpm -i rpmforge-release-0.5.1-1.el5.rf.*.rpm
    yum check-update
  3. Install mercurial and git
    yum -y install mercurial git
  4. Compile Xen
    hg clone
    cd xen-4.0-testing.hg
    make -j16 xen
    make -j16 tools
    make -j16 stubdom
  5. Compile pvops kernel
    make kernels
    make -j16 dist

    This is the most critical step, and if you don’t configure the
    kernel correctly, booting will fail badly. make kernels builds the
    kernel with the default .config. To modify it, change to build-xxx directory and
    run make menuconfig.

    Make sure to have the following in your config file

    # These are important for CentOS/RHEL

    If you want to compile xen-evtchn, xen-netback, xen-blkback and xenfs as
    modules, use the following in your config.

  6. Install Xen and kernel
    make install-xen
    make install-tools PYTHON_PREFIX_ARG=
    cd build-*
    make modules_install
    make install

    For make install-tools the PYTHON_PREFIX_ARG is REQUIRED. On Fedora systems,
    this is not required.

  7. Create initrd (above make install usually takes care of creating an initrd
    matching your configuration, but just in case)

    mkinitrd -v -f /boot/initrd-
  8. Update grub
    title Xen 4 +
        root   (hd0,0)
        kernel  /xen-4.0.gz dom0_mem=4096M loglvl=all guest_loglvl=all
        module  /vmlinuz- root=/dev/cciss/c0d0p1 ro nomodeset console=tty0
        module  /initrd-
  9. Add the following to /etc/modprobe.conf to load all the Xen modules
    install xen /sbin/modprobe xen-evtchn; /sbin/modprobe xen-netback;
    /sbin/modprobe xenfs; /sbin/modprobe xen-blkback; /bin/true

    OR, add the following to /etc/rc.local (I had trouble making the modprobe

    if ( uname -r | grep "") ; then
       for mod in xen-evtchn xen-netback xen-blkback xenfs; do
            /sbin/modprobe $mod
       /etc/init.d/xend start
  10. Add xenfs to /etc/fstab
        none /proc/xen xenfs defaults 0 0
  11. Test it
    echo "savedefault --default=X --once" | grub --batch (where X is the
    entry for Xen)
    /etc/init.d/xend start
    xm list


  1. The biggest problem will be kernel not booting properly. There might be
    various reasons.

    • Make sure to use the right set of configuration options for your hardware
    • Make sure to have all Xen options correctly compiled
    • out_of_memory kernel OOPS
      • CONFIG_XEN_MAX_DOMAIN_MEMORY=32 # change this to a lesser number
      • Add more memory to dom0 with dom0_mem=4096M
    • Network card not appearing – make sure to have the right network
      module (bnx2x for BroadCom NetXtreme II card) inserted
  2. Missing /dev/xen entries. Means that you haven’t loaded the
    xen-evtchn and xen-gntdev. Run

    modprobe xen-evtchn xen-gntdev

    See step 9 above to update /etc/modprobe.conf

  3. If you see the following in /var/log/messages
    xenconsoled: Failed to contact xenstore (Connection refused)

    Do step 2 to insert all the appropriate modules, and verify that you can
    start xend

Installing VMs

  1. First, verify that the initrd includes all the necessary Xen frontend
    modules. Thisis not necessary for hvm VMs, but for paravirtualization it’s crucial
    to have allthe right netfront and blkfront modules installed. A sample mkinitrd for domU

    mkinitrd -v -f /boot/initrd-
    --with=blktap \
            --with=xen-blkback --with=xenfs
  2. Install virt-manager
    yum install virt-manager
  3. Configure /etc/xen/xend-config.sxp to enable access through libvirt and
    virt-manager. Add the following lines

    (xend-http-server yes)
    (xend-unix-server yes)
    (xend-port        8000)
    (xend-relocation-port 8002)
  4. Restart xend
    /etc/init.d/xend restart
  5. Start virt-manager as root
  6. Install a VM as usual


  2. This is the most
    authoritative reference for compiling the pvops kernel.
    (this is Ubuntu specific, but has some useful details)

Using kpartx to Mount VM Disk Images

If you are using a full disk image for your VM, instead of using a file system, you cannot simply mount it directly using

mount -o loop d0.img

Use the following commands to see the partitions in your disk, and then mount them.

# kpartx -av d0.img
add map loop0p1 : 0 29333504 linear /dev/loop0 2048
add map loop0p5 : 0 1380352 linear /dev/loop0 29337600
# mount /dev/mapper/loop0p1 /mnt
# ls /mnt
bin   cdrom  etc   initrd.img      lib         media  opt   root  selinux  sys  usr  vmlinuz
boot  dev    home  initrd.img.old  lost+found  mnt    proc  sbin  srv      tmp  var
# unmount /mnt
# kpartx -d d0.img

The last two commands are to unmount and delete the partition mappings.

How to Setup Xen4 and pvops 2.6.32.x on CentOS 5.5 using GITCO repository

See my howto on setting up Xen4 + pvops 2.6.32.x first before following this.

  1. Do steps 1, 2 and 3 described in the howto
  2. Install Xen 4.0.1 from GITCO repository. The benefit of this is that all
    the required packages,Xen enabled Linux kernel (2.6.18) and libvirt are correctly installed and

       yum -y install xen
  3. Get pvops kernel
       git clone git://
       cd linux-2.6-xen
       git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x
  4. Build the kernel to match your configuration
    make menuconfig

Follow the remaining steps in the howto
to setup grub, and test it.

Computer Systems Conference Rankings

Conference rankings are a contentious topic, and it is often difficult to
directly compare two conferences, because each conference has unique flavor,
community and history. In general, it’s easy to identify top 1 or 2
conferences in a field, but it gets murky as you go down. There are a bunch of
webpages (see below), where you can look at the rankings. About the field of
computer systems, defining it is another blog post. For now, read the Section
3 of white paper
, and Liviu’s
presentation on what constitutes good systems research.

This post is my view on the systems conference rankings. Disclaimer: these
are just my personal opinions. Quality of conferences varies over time, and
these are not set in stone. In my opinion, all the conferences I mentioned here
are reputed and great conferences to publish in.

For general systems (operating systems, distributed systems)

Conference name Acceptance rate Notes
11.8% – 21% These two conferences, which alternate every year are widely considered as
the top conferences in systems. SOSP has rich history, and is considered
slightly more prestigious. Both are highly selective and are very influential in
“real” systems.
Eurosys 16.9% – 21% Eurosys is still a new conference, but is quickly becoming a premier systems
conference. Though, it is hosted in Europe every year, published papers are from
all around the world, with majority of them coming from systems research groups
in the US.
Annual Technical Conference
16.3% – 30% USENIX ATC is another great conference, with lots of good systems work.
It’s quality has varied over time, and is slightly inconsistent.

For networked systems

Conference name Acceptance rate Notes
SIGCOMM 10% Widely considered as the most difficult conference (in networking) to
publish in, SIGCOMM has rich history, and great papers every year. Earlier
SIGCOMM conferences have accepted more papers, but the acceptance rates have
dropped to 10-12% in the past few years.
NSDI 19 – 21% Compared to SIGCOMM, NSDI has the interesting flavor of systems and
CoNEXT 19 – 21% CoNEXT is similar to Eurosys. It has a bit more European flavor to it, since
it is hosted in Europe in alternate years. It is also a new conference compared

Other systems-flavored conferences

  • For file systems, and storage, FAST
    is considered a top conference.
  • For programming languages, PLDI, POPL are
    considered top conferences.
  • For the combination of OS + architecture or OS + programming languages ASPLOS is
    considered the top conference. It usually has multi-disciplinary flavor to

Now, the fun part.

  1. Is INFOCOM a
    good networking conference for systems work? This is a tricky question, since
    INFOCOM traditionally has accepted more theory/algorithms/analysis/simulation
    oriented papers, with a bent on wireless systems. INFOCOM has low acceptance
    rates in the range of 16-20%, but due to large number of tracks, the quality
    sometimes is patchy. INFOCOM does accept systems papers.
  2. Is NSDI better than Eurosys for networking + systems work? Another
    interesting question to ponder during lunch breaks :-) I think, if a project has more networking flavor
    to it, one should consider submitting it to NSDI. On the other hand, if the
    project is purely systems work, then Eurosys is a better venue.
  3. What is the criterion for deciding the rankings of conferences? Should we
    use acceptrance rate, or citation count or industry influence or a combination
    of them?

Some useful links for conference rankings.

Virtualization Research Projects that Spawned Real Products

Virtualization is a hot topic for research, and with the rise of cloud
computing, it has gained even more attention. Great number of research papers
are being published in many major conferences. The following is an attempt to
list some of the research work, and how they influenced real products or
features. Some of the products have been major open source projects, before
becoming purely commercial products. These are roughly in reverse-chronological

Note: I will be updating this over time. Let me
know if you know of virtualization research papers that inspired or spawned real

Research Topic Research Paper Commercial Product/Feature (s)
Storage Resource Management PARDA:
Proportional Allocation of Resources for Distributed Storage Access
. Ajay
Gulati, Irfan Ahmad, Carl A. Waldspurger. Published in the USENIX Conference on
File and Storage Technologies (FAST ’09)
Video of
End-host Network Virtualization Crossbow
Virtual Wire: Network in a Box
. Sunay Tripathi, Nicolas Droux, Kais
Belgaied, and Shrikrishna Khare. Large
Installation System Administration Conference, 2009
in Solaris
Network Virtualization OpenFlow:
Enabling Innovation in Campus Networks
. Nick McKeown, Tom Anderson, Hari
Balakrishnan, Guru Parulkar, Larry Peterson, Jennifer Rexford, Scott Shenker,
and Jonathan Turn. CCR 2008
GPU virtualization GPU
Virtualization on VMware’s Hosted I/O Architecture
. Micah Dowty,
Jeremy Sugerman. Published in the USENIX Workshop on I/O Virtualization
VMware desktop virtualization
NetChannel2 Netchannel
2: Optimizing Network Performance
. J. Renato Santos, G. (John) Janakiraman,
Yoshio Turner, Ian Pratt. Presented at Xen Summit,
Apr 2007
NetChannel in Xen
IaaS platforms Eucalyptus research project started by Rich Wolski Eucalyptus
Performance profiling Enforcing
Performance Isolation Across Virtual Machines in Xen
– Diwaker Gupta,
Ludmila Cherkasova, Rob Gardner, and Amin Vahdat. Proceedings of the 7th
ACM/IFIP/USENIX Middleware Conference. Melbourne, Australia, Nov 2006
Live migration Live
Migration of Virtual Machines
– Christopher Clark, Keir Fraser, Steven
Hand, Jacob Gorm Hansen, Eric Jul, Christian Limpach, Ian Pratt, Andrew
Warfield. Published at NSDI
Xen Live Migration
Para-virtualization Xen
and the Art of Virtualization
– Paul Barham, Boris Dragovic, Keir
Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, Andrew
Warfield. Published at SOSP
Xen 1.x
Memory balooning
Memory Resource Management in VMware ESX Server
– Carl Waldspurger.
Published at OSDI ’02
ESX and Xen memory balooning
Virtual Machine Monitors Disco:
running commodity operating systems on scalable multiprocessors
. Edouard
Bugnion, Scott Devine, Kinshuk Govil, Mendel Rosenblum. SOSP 1997.
VMware workstation