Virtualization

mobile_hypervisor

Challenges in Mobile (Smart Phone) Virtualization

Feb 12th

Posted by Pradeep Padala in Cloud Computing

2 comments

We have virtualization on servers, desktops, laptops and what’s next? Naturally, the “smart phone”. Today’s mobile phones have become quite powerful that one can easily virtualize them and run multiple operating systems. But,

Why do you need smart phone virtualization?

Similar to desktop virtualization, the use-case is not that clear, but there are some benefits that can be gained.

  1. Reducing vendor lock-in. Can you run iPhone applications on an Android phone? Currently, no, but with virtualization this might be possible. There are issues with Apple (or other vendors)’s mobile OS licensing that make it illegal to do so, right now. You can only run iOS on Apple approved devices, but this may change in the future. Many old phones have legacy applications and libraries that can be run in virtualized sandboxes as well.
  2. Mobile cloud computing. You can clearly access the so-called “cloud” from a smart phone, but if smart phones are virtualized, mobile applications can be migrated to a cloud and back seamlessly. This is still a pipe-dream and a few research platforms (MAUI and CloneCloud) are proposed to make this a reality. The benefits of doing this are enormous: one can potentially access infinite compute and storage resources on the smart phone.
  3. Security. This is probably the strongest argument for mobile virtualization. The popularity of smart phones is making them a prime target for malicious hackers. Malware, Viruses and other nasty stuff are making their way into smart phones, which currently don’t have much protection against them. It’s not easy to run Anti-virus software on a mobile phone, as the current AV software is very resource hungry making it unsuitable for smart phones.
  4. Isolation. This is related to security. It will be possible to have different user profiles with different levels of access (say work and home) with mobile virtualization. VMware recently demonstrated, how multiple user profiles can co-exist in the same phone at Mobile Congress 2011.
  5. Ease of development. This is an indirect benefit for mobile phone developers. Virtualization can make the development of mobile applications easier by removing the need for maintaining multiple versions of the same application for different operating systems.

Challenges

Before you start seeing virtualized mobile phones, there are still plenty of challenges that need to be solved.

  1. Limited resources. Though, the CPUs on mobile phones are getting more and more powerful, they still have limited battery and storage space. Virtualization usually adds more burden to the resources and is not necessarily energy-aware.
  2. Poor connectivity . Mobile cloud computing sounds like a great idea, but mobile phones often experience variable connectivity. It’s not easy to implement disconnected operations for storage and for seamlessly migrating computation.
  3. Poor performance. Virtualization adds performance overheads, which are still being researched in server virtualization environments. It’s not clear whether the overheads are worth the effort on a mobile phone.

These challenges can be overcome and it’s only a matter of time before smart phones are virtualized. Some current players in the market are: VMware’s mobile virtualizaiton platform (MVP) and OKL4 MicroVisor. OKL4 claims to be the first open source mobile hypervisor, which is an exciting development that can lead to advances in mobile virtualization.

Share

Virtualization Bibliography

Feb 2nd

Posted by Pradeep Padala in Cloud Computing

2 comments

Virtualization is a hot topic and the field is growing at a rapid pace. There are hundreds of papers on each sub-topic, and this list is intended to be a good starting point for someone starting in virtualization research.

Surveys/Books

These are good starting points, if you are just learning about virtualization

  1. Survey of virtualization machine research. Robert P. Goldberg. IEEE Computer, June 1974, pp 34-45. [PDF]. One of the oldest surveys about virtualization research.
  2. A Survey on Virtualization Techniques. Susanta Nanda and Tzi-cker Chiueh [PDF]. A more modern survey of virtualization techniques. The paper lists many of the techniques, but doesn’t really explain how they are different etc. Yet, this is a good read for budding virtualization researchers.
  3. Virtual Machines: Versatile Platforms for Systems and Processes . Jim Smith Ravi Nair. Amazon book link

Virtualization Overview

These are the papers that will help in understanding fundamentals and concepts of virtualization.

  1. Xen and the art of virtualization. Barham et.al. This is the classic SOSP paper on para-virtualization. [PDF].
  2. When Virtual is Better Than Real. Peter Chen and Brian Noble. This is a great short article explaining the benefits of virtualization. ACM link.
  3. Disco: Running Commodity Operating Systems on Scalable Multiprocessors. E. Bugnion, S. Devine, and M. Rosenblum. This paper is considered to be the first paper that revived the virtualization concepts pioneered by IBM. ACM link.
  4. Running multiple operating systems concurrently on an IA32 PC using virtualization techniques. Kevin Lawton. This is a great article on the difficulties involved in virtualizing the x86 platform. link.
  5. Formal requirements for virtualizable third generation architectures. Popek and Goldberg. The classic paper explaining the requirements for virtualizing a specific ISA. ACM link. If you want light reading, check the Wikipedia explanation of the requirements.
  6. Scale and Performance in the Denali Isolation Kernel. Andrew Whitaker, Marianne Shaw, and Steven D. Gribble. Denali is another example of early virtualization papers that rejuvenated the research. Denali shows the containers (or vservers) concept, which is used in OpenVZ, KVM and VServers.
  7. The Exokernel Operating System Architecture. Dawson Engler’s thesis. [PS]. This is more of OS research, but a great read for understanding some of the techniques (thin hypervisor, pass through etc.) that are used in modern virtualized systems.

CPU Virtualization

Virtualization of CPU is provided by a CPU scheduler that provides the illusion of multiple CPUs (or VCPUs). Scheduling has a long and rich history. Below are a few links in relation to virtualization.

  1. Xen’s credit scheduler. Xen’s credit scheduler is a proportional fair scheduler, which is pretty big improvement (especially in SMP environments) over the old SEDF scheduler.
  2. Xen’s scheduler is partly inspired by Carl Waldspurger’s lottery scheduling and the paper: Lottery Scheduling: Flexible Proportional-Share Resource Management by Carl Waldspurger is a must read. [PDF].
  3. Comparison of the Three CPU Schedulers in Xen – Ludmila Cherkasova, Diwaker Gupta, Amin Vahdat – Perfomance Evaluation Review. Vol 35, Number 2. Sept 3, 2007. This provides a great comparison of CPU schedulers and provides insight on how handling I/O is complicated. [PDF].

Network Virtualization

Network virtualization is gaining momentum in recent years. I would just point you to the following survey as a starting point.

  1. Survey of network virtualization. N.M. Mosharaf Kabir Chowdhury and Raouf Boutaba. [PDF]. This is pretty comprehensive and well-written survey on network virtualization. Mosharaf also has a nice network virtualization, listing lots of useful papers.

Device Virtualization

  1. Xen’s device driver virtualization.
  2. Virtualizing I/O Devices on VMware Workstation’s Hosted Virtual Machine Monitor. Jeremy Sugerman, Ganesh Venkitachalam, and Beng-Hong Lim. As the name suggests, this paper explains VMware Workstation’s mechanisms for virtualizing devices. [PDF].
  3. GPU Virtualization on VMware’s Hosted I/O Architecture. Micah Dowty, Jeremy Sugerman. This paper by VMware folks explains how VMware implements GPU virtualization. Great starting point for GPU virtualization. [PDF]. Andres’ paper on VMM indepedent graphics acceleration is a good follow-up reading.

Memory Virtualization

  1. Memory Resource Management in VMware ESX Server. Carl A. Waldspurger. [PDF]. Classic paper on memory virtualization. This paper introduces memory ballooning, which is a great technique used in commercial platforms as well.
  2. Difference engine: harnessing memory redundancy in virtual machines. Diwakar Gupta et.al. [PDF]. A great follow up paper that talks about sharing memory across VMs. Incidentally, both these papers have received best paper awards.

Migration/Cloning

  1. Live Migration of Virtual Machines. Clark et.al. One of the first academic papers discussing how to do live migration, which is implemented in Xen. VMware has supposedly implemented vmotion before this was published. [PDF]
  2. The design and implementation of Zap: a system for migrating computing environments. Osman et.al. Zap predates Xen paper and talks about sandboxing processes that can be migrated. This is more like container virtualization rather than full system virtualization. ACM link
  3. SnowFlock: rapid virtual machine cloning for cloud computing. Andres Lagar-Cavilla et.al. Greatp paper to read to understanding cloning of virtual machines. Snowflock link.

Resource Management/Automation

  1. Carl Waldspurger’s PhD thesis is a great place to start for understanding resource management. MIT link.
  2. Automated control of multiple virtualized resources. Padala et.al. Disclaimer: this is my own paper :-) It’s a good place to start if you want to learn about automating resource management for virtual machines. The control theory aspects are a bit complex, but you can ignore them and focus on the issues in automating resource management. [PDF].
  3. Black-box and Gray-box Strategies for Virtual Machine Migration. Clark et.al. This is a follow up for live migration paper. The paper talks about strategies for automating the migration of VMs to meet specific goals. USENIX link.
Share
,

How to Setup Xen4 and pvops 2.6.32.x on CentOS 5.5 using GITCO repository

Sep 28th

Posted by Pradeep Padala in Tips

11 comments

See my howto on setting up Xen4 + pvops 2.6.32.x first before following this.

  1. Do steps 1, 2 and 3 described in the howto
  2. Install Xen 4.0.1 from GITCO repository. The benefit of this is that all the required packages,
    Xen enabled Linux kernel (2.6.18) and libvirt are correctly installed and setup

    wget http://www.gitco.de/repo/GITCO-XEN4.0.1_testing_x86_64.repo
yum -y install xen
  3. Get pvops kernel 
    git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
cd linux-2.6-xen
git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x
  4. Build the kernel to match your configuration 
    make menuconfig

Follow the remaining steps in the howto to setup grub, and test it.

Share
, , ,

How to Setup Xen4 and pvops 2.6.32.x on CentOS 5.5

Sep 28th

Posted by Pradeep Padala in Tips

30 comments

Setting up Xen4 has been tricky, because of the move to the latest Linux kernels with paravirt_ops. Compiling it on CentOS, which uses 2.6.18 kernel, and old toolset is a bit more challenging. The following are my notes for setting up Xen4 + pvops 2.6.32.x on CentOS 5.5 on BL460c blades.

Important note: I couldn’t get Xen 4.0.1 release work on CentOS 5.5 on our blades. You have to get the latest Xen and pvops kernel available from http://xenbits.xen.org/xen-4.0-testing.hg and http://www.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git/ (4.0.2-rc1-pre and 2.6.32.21 as of this writing). This may or may not be a problem for you.

  1. Install prerequisite packages 
    yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"
yum -y install transfig wget texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial glibc-devel
  2. Install rpmforge repository. Follow the instructions on CentOS Wiki. 
    wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm
rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
rpm -K rpmforge-release-0.5.1-1.el5.rf.*.rpm
rpm -i rpmforge-release-0.5.1-1.el5.rf.*.rpm
yum check-update
  3. Install mercurial and git 
    yum -y install mercurial git
  4. Compile Xen 
    hg clone http://xenbits.xen.org/xen-4.0-testing.hg
cd xen-4.0-testing.hg
make -j16 xen
make -j16 tools
make -j16 stubdom
  5. Compile pvops kernel 
    make kernels
make -j16 dist

    This is the most critical step, and if you don’t configure the kernel correctly, booting will fail badly. make kernels builds the kernel with the default .config. To modify it, change to build-xxx directory and run make menuconfig.

    Make sure to have the following in your config file

    # These are important for CentOS/RHEL
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y

CONFIG_ACPI_PROCFS=y
CONFIG_XEN=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=32
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PCI=y
CONFIG_PCI_XEN=y
CONFIG_XEN_BLKDEV_FRONTEND=m
CONFIG_NETXEN_NIC=m
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_XEN_KBDDEV_FRONTEND=m
CONFIG_HVC_XEN=y
CONFIG_XEN_FBDEV_FRONTEND=m
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_XENBUS_FRONTEND=m
CONFIG_XEN_PCIDEV_FRONTEND=y

    If you want to compile xen-evtchn, xen-netback, xen-blkback and xenfs as modules, use the following in your config.

    CONFIG_XEN_DEV_EVTCHN=m
CONFIG_XEN_BACKEND=y
CONFIG_XEN_BLKDEV_BACKEND=m
CONFIG_XEN_NETDEV_BACKEND=m
CONFIG_XENFS=m
  6. Install Xen and kernel 
    make install-xen
make install-tools PYTHON_PREFIX_ARG=
cd build-*
make modules_install
make install

    For make install-tools the PYTHON_PREFIX_ARG is REQUIRED. On Fedora systems, this is not required.

  7. Create initrd (above make install usually takes care of creating an initrd matching your configuration, but just in case) 
    depmod 2.6.32.21
mkinitrd -v -f /boot/initrd-2.6.32.21.img 2.6.32.21
  8. Update grub 
    title Xen 4 + 2.6.32.21
    root   (hd0,0)
    kernel  /xen-4.0.gz dom0_mem=4096M loglvl=all guest_loglvl=all
    module  /vmlinuz-2.6.32.21 root=/dev/cciss/c0d0p1 ro nomodeset console=tty0 console=ttyS0,115200n8
    module  /initrd-2.6.32.21.img
  9. Add the following to /etc/modprobe.conf to load all the Xen modules 
    install xen /sbin/modprobe xen-evtchn; /sbin/modprobe xen-netback; /sbin/modprobe xenfs; /sbin/modprobe xen-blkback; /bin/true

    OR, add the following to /etc/rc.local (I had trouble making the modprobe work)

    if ( uname -r | grep "2.6.32.21") ; then
   for mod in xen-evtchn xen-netback xen-blkback xenfs; do
        /sbin/modprobe $mod
   done
   /etc/init.d/xend start
fi
  10. Add xenfs to /etc/fstab 
        none /proc/xen xenfs defaults 0 0
  11. Test it 
    echo "savedefault --default=X --once" | grub --batch (where X is the entry for Xen)
reboot
/etc/init.d/xend start
xm list

Troubleshooting

  1. The biggest problem will be kernel not booting properly. There might be various reasons.
    • Make sure to use the right set of configuration options for your hardware
    • Make sure to have all Xen options correctly compiled
    • out_of_memory kernel OOPS
      • CONFIG_XEN_MAX_DOMAIN_MEMORY=32 # change this to a lesser number
      • Add more memory to dom0 with dom0_mem=4096M
    • Network card not appearing – make sure to have the right network module (bnx2x for BroadCom NetXtreme II card) inserted
  2. Missing /dev/xen entries. Means that you haven’t loaded the xen-evtchn and xen-gntdev. Run 
    modprobe xen-evtchn xen-gntdev

    See step 9 above to update /etc/modprobe.conf

  3. If you see the following in /var/log/messages 
    xenconsoled: Failed to contact xenstore (Connection refused)

    Do step 2 to insert all the appropriate modules, and verify that you can start xend

Installing VMs

  1. First, verify that the initrd includes all the necessary Xen frontend modules. This
    is not necessary for hvm VMs, but for paravirtualization it’s crucial to have all
    the right netfront and blkfront modules installed. A sample mkinitrd for domU is
    below

    mkinitrd -v -f /boot/initrd-2.6.32.21-domU.img  --with=blktap \
        --with=xen-blkback --with=xenfs 2.6.32.21
  2. Install virt-manager 
    yum install virt-manager
  3. Configure /etc/xen/xend-config.sxp to enable access through libvirt and virt-manager. Add the following lines 
    (xend-http-server yes)
(xend-unix-server yes)
(xend-port        8000)
(xend-relocation-port 8002)
  4. Restart xend 
    /etc/init.d/xend restart
  5. Start virt-manager as root
  6. Install a VM as usual

References

  1. http://wiki.xensource.com/xenwiki/2.6.18-to-2.6.31-and-higher
  2. http://wiki.xensource.com/xenwiki/XenParavirtOps. This is the most authoritative reference for compiling the pvops kernel.
  3. http://wiki.xensource.com/xenwiki/Xen4.0
  4. http://bderzhavets.wordpress.com/2010/04/24/set-up-ubuntu-10-04-server-pv-domu-at-xen-4-0-dom0-pvops-2-6-32-10-kernel-dom0-on-top-of-ubuntu-10-04-server/ (this is Ubuntu specific, but has some useful details)
Share
, , ,

Virtualization Research Projects that Spawned Real Products

Jul 7th

Posted by Pradeep Padala in Research

2 comments

Virtualization is a hot topic for research, and with the rise of cloud computing, it has gained even more attention. Great number of research papers are being published in many major conferences. The following is an attempt to list some of the research work, and how they influenced real products or features. Some of the products have been major open source projects, before becoming purely commercial products. These are roughly in reverse-chronological order.

Note: I will be updating this over time. Let me know if you know of virtualization research papers that inspired or spawned real products.

Research Topic Research Paper Commercial Product/Feature (s)
Storage Resource Management PARDA: Proportional Allocation of Resources for Distributed Storage Access. Ajay Gulati, Irfan Ahmad, Carl A. Waldspurger. Published in the USENIX Conference on File and Storage Technologies (FAST ’09) vSphere 4.1 Video of SIOC
End-host Network Virtualization Crossbow Virtual Wire: Network in a Box. Sunay Tripathi, Nicolas Droux, Kais Belgaied, and Shrikrishna Khare. Large Installation System Administration Conference, 2009. Crossbow in Solaris
Network Virtualization OpenFlow: Enabling Innovation in Campus Networks. Nick McKeown, Tom Anderson, Hari Balakrishnan, Guru Parulkar, Larry Peterson, Jennifer Rexford, Scott Shenker, and Jonathan Turn. CCR 2008 OpenFlow
GPU virtualization GPU Virtualization on VMware’s Hosted I/O Architecture. Micah Dowty, Jeremy Sugerman. Published in the USENIX Workshop on I/O Virtualization 2008 VMware desktop virtualization
NetChannel2 Netchannel 2: Optimizing Network Performance. J. Renato Santos, G. (John) Janakiraman, Yoshio Turner, Ian Pratt. Presented at Xen Summit, Apr 2007

NetChannel in Xen
IaaS platforms Eucalyptus research project started by Rich Wolski Eucalyptus
Performance profiling Enforcing Performance Isolation Across Virtual Machines in Xen – Diwaker Gupta, Ludmila Cherkasova, Rob Gardner, and Amin Vahdat. Proceedings of the 7th ACM/IFIP/USENIX Middleware Conference. Melbourne, Australia, Nov 2006 XenMon
Live migration Live Migration of Virtual Machines – Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hansen, Eric Jul, Christian Limpach, Ian Pratt, Andrew Warfield. Published at NSDI 2005 Xen Live Migration
Para-virtualization Xen and the Art of Virtualization – Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, Andrew Warfield. Published at SOSP 2003 Xen 1.x
Memory balooning Memory Resource Management in VMware ESX Server – Carl Waldspurger. Published at OSDI ’02 ESX and Xen memory balooning
Virtual Machine Monitors Disco: running commodity operating systems on scalable multiprocessors. Edouard Bugnion, Scott Devine, Kinshuk Govil, Mendel Rosenblum. SOSP 1997.

VMware workstation
Share
,

LiteGreen: Saving Energy in Networked Desktops Using Virtualization

Jun 21st

Posted by Pradeep Padala in Virtualization

1 comment

Update: LiteGreen wins best paper award!

I will be presenting LiteGreen this week at the USENIX Annual Technical Conference. LiteGreen saves desktop energy by migrating the idle desktops (running in VMs) to a central server. More details on the Project page. Slides and video of my presentation are available from talks page

LiteGreen Energy Savings

Share
, , ,

Client Hypervisors and GPU virtualization

Jun 15th

Posted by Pradeep Padala in Virtualization

No comments

What is a client hypervisor? Client hypervisors are hypervisors that run on a user’s (client’s) desktop. A nice video showing Xen Client Hypervisor should clarify its usage. Server hypervisors like VMware’s ESX server, and XenServer virtualize a physical server that allow you to run multiple virtual machines. This has many benefits including reduced costs due to consolidation, easy management, and for load balancing.

Question is: Why would you want to virtualize your desktop? This use-case is not that clear, considering that desktops usually run only one operating system like Windows. Some benefits are

  1. Easy management. If all desktops in an enterprise are running in VMs, the administrators can easily upgrade all the VMs easily, assuming that all VMs are derived from a single gold image. This is not as simple as it sounds, if you allow the desktop users to install software, since the operating systems running VMs will diverge from the gold image.
  2. Saving energy. When a desktop running a VM becomes idle, one can easily migrate it to a server, and put the physical machine to sleep. Though, this is conceptually simple, there are many issues including reducing user disruption and consolidation of desktops on the server. See my LiteGreen work for a thorough evaluation of such system and how to solve some of the problems.
  3. Security. Technically, desktops running in VMs are more secure, since they don’t have direct access to hardware and can be monitored by hypervisor to prevent malicious activities. Anti-virus, anit-malware software can be installed in hypervisor, host operating system or management stub VMs, which can monitor the desktop VMs. This is complicated by the fact that VMMs or hypervisors do not have complete state of the operating systems running in VMs. Virtual Machine Introspection (VMI) is an on-going research field pursued by many security/virtualization researchers (Jiang’s work) that tries to solve these problems.

    However, this is a two-edged sword, since hackers can build rootkits that run in a hypervisor too.

    Another interesting usage of client hypervisors is to run different desktop applications in light-weight VMs (Qubes project) for better application isolation.

  4. Mobile Devices. This may be surprising, since mobile devices have usually less powerful hardware (that’s changing, however). The benefit comes from the fact that mobile devices may be able to run proprietary applications in a light-weight VM providing more security and support for legacy applications. The VMs can be moved to a cloud, if needed.

The biggest disadvantage of client hypervisors is: performance overhead and reduced user experience due to not having direct access to display hardware. For example, running a game like Quake in a VM, is not what you are looking for when you are buying powerful graphics cards. GPU virtualization is picking up, but will take some time to become a mature technology. Researchers are working towards allowing VMs to take advantage of graphics acceleration. GPU vendors have to provide hardware support for virtualizing the graphics processors similar to Intel’s VT-x extensions.

Share
, , ,

Understanding Live Migration of Virtual Machines

Jun 13th

Posted by Pradeep Padala in Virtualization

10 comments

Introduction

In virtualization community, live migration of virtual machines is pretty much considered a “default” mature feature in any hypervisor product. All major vendors like VMware, XenSource/Citrix, and Microsoft have products that support live migration. I consider it a major success for virtualizaton research community, since the first academic paper on it was published not so long ago in NSDI 2005. VMware’s vmotion technology (I think) predates this paper, but the technical details were largely unknown.

This post is partly inspired by some of the questions I received during a recent LiteGreen talk. Non-virtualization folks seem to misunderstand some of the aspects of live migration, so in this post, I will explain live migration and some “gotchas” in using it.

Alright, What is live migration? Migration of a virtual machine is simply moving the VM running on a physical machine (let’s call it source node) to another physical machine (let’s call it target node). The trick is to do this, while the VM is running on the source node, and without disrupting any active network connections even after the VM is moved to the target node. It is considered “live”, since the original VM is running, while the migration is in progress. Huge benefit of doing the live migration is the very small downtime in the order of milli seconds.

How to achieve live migration

To move a VM from the source node to the target node, we need to consider moving its cpu state, memory content, storage content, and network connections.

  • Migrating CPU state has been extensively researched in the context of process migration. See Berkeley Lab Checkpoint/Restart (BLCR) for implementation details.
  • Migrating memory content is a bit tricky, considering that the VM on the source node is still running and making modifications to the memory state. The idea is to do iterative copying of the memory contents, and send only the “delta” changes to the target node. There is a point, when only a small “delta” memory that needs to be copied. At this stage, the VM on the source node is paused, the delta memory is copied, and VM is resumed on the target node. The brief pause is what causes the downtime.
  • Migrating storage content is similar to memory, but will require a lot more time, and the migration may take on the order of minutes. It may not be easy to guarantee small mill second downtime with storage migration. All current commercial products side-step this problem, by using centralized storage (e.g. NFS, iSCSI, Fibre Channel based SAN) that hosts VM images. Storage content doesn’t have to be migrated, if both source and target node are connected to the centralized storage. There are a few research solutions (1, 2, 3) that try to address this problem.
  • Migrating network connections is simple, if you assume that all of the nodes are in the same IP subnet. When the VM is migrated to the target node, the VM simply has to send an ARP broadcast saying that the IP address has moved to a new physical (MAC address) location. Since this happens at the connection between Layer 2 and Layer 3 of network stack, transport layer is transparent to this change, and TCP connections survive the migration. As a result, applications see no disruption in network connections. Clearly, this approach doesn’t work, if the VMs have to cross subnets. There are many ways (1, 2) to solve this problem, but all of them have huge performance implications

The success and popularity of live migration lies in the fact that it has very small downtime, and some of the limitations I mentioned above are inherent to achieving that goal.

Benefits of live migration

One of the primary use-cases for live migration is for resource management in cloud computing. For example, cloud computing providers like Amazon EC2 have thousands of VMs running in their data centers. To save energy, cost and for load balancing, they can move VMs using live migration, without disrupting their customer applications running in the VMs. How to do this efficiently (or optimally) is a big research question, and some solutions (1) are available.

Frequently asked questions about live migraiton

Before I conclude, some answers to frequently asked questions regarding live migration.

  1. What is the usual VM downtime? The original Xen paper reported downtimes as small as 60ms for specific workloads. This is dependent on the application running in the VM, and network heavy applications may see disruptions, even if the downtime is small.
  2. What is the total migration time? This is where most people get confused, since the total time it takes to perform live migration is different from the downtime. It is usually in the order of a few seconds (ranging from 10-120 seconds). This time depends on how heavily the VM on the source node is modifying its memory content.
  3. How is live migration different from suspend/resume? Suspend/resume is similar to migration, since you can suspend a VM to a disk (on a centralized storage), and then resume the VM on a different physical machine. This works, but it is not “live”. Clearly, active network connections cannot be resumed in this model.
  4. How is live migration different from migrating to the cloud? This is an unfortunate source of confusion, since migration may mean different things to different people. When people are talking about migrating to the cloud, they usually are talking about moving your computation and data to a cloud like Amazon EC2 or Windows Azure. This is not live, and it may not really be VM migration.
Share
, , ,

Our paper in CDC (Top tier control theory conference)

Jul 30th

Posted by Pradeep Padala in Research

No comments

Ok, finally the internship is over as I just wrap up my final presentation. It’s been extremely hectic, and I don’t even remember how the three months passed. This time around, I was involved in multiple projects, so the context switching some times hurt a little bit. Rachna and I also had a lot of fun meeting with lots of people, roaming around etc. I am too tired to write about all the stuff I did. Who cares any way? :-)

Today’s news is that one of our papers got accepted in IEEE Conference on Decision and Control, one of the oldest and most prestigious control theory conference out there. The paper that got accepted is about our MIMO (multiple input and multiple output) control and the math in this paper forms the basis for my work this summer. We have extended it quite a bit and the next paper is going to be a computer systems paper with the theory used in a real system and hopefully will be really exciting!

Stay tuned!

P.S. I am landing in Ann Arbor on Aug 4th morning. See you AAites soon.

Share

Europe, here I come !

Dec 5th

Posted by Pradeep Padala in Research

No comments

I am giddy, yes, giddy with excitement at seeing the mail from EuroSys chair that my paper to EuroSys 2007 is accepted. It’s one of the good systems conferences and is considered to be reaching the quality of OSDI/SOSP (the top systems conference). Also, another paper of mine co-authored with HP folks is accepted to IM 2007. It’s a sub-project originated from the main EuroSys paper.

It was a lot of hard work, as usual, but it’s worth it. Next stop is SOSP ! To put it in my HP colleague Mustafa’s words

… doing a SOSP paper with CPU+IO+network would be more important now that we have the genie out of the box

Go go go …

Share
,