Tips

How to Access Your Web Server within Your Local Network

Feb 9th

Posted by Pradeep Padala in Tips

No comments

If you are hosting a web server at home and using your router’s port forwarding to redirect requests from WAN port to WebServerHost:80, there’s an annoying problem in some routers: you cannot access the web server from inside the network. There are multiple reasons why this might happen.

  1. The router assumes that all traffic from internal hosts to port 80 are for the router admin interface. This is annoying, as you just get to the router’s web admin page. Solution: you can port forward from another port (say 8000) to your WebServer’s port 80. Now, you can access the website with http://my.site.net:8000. You are out of luck, if your router does’t allow port forwarding to a different port (say WAN:8000->WebServerHost80 instead of WAN:80->WebServerHost80).
  2. You have two or more routers with different subnets and the routing is messed up. This is a difficult problem to diagnose and often the router’s admin interfaces are not that amenable to debugging routing problems. I suggest changing your firmware to DD-WRT, if you can.
  3. Port forwarding doesn’t work for incoming traffic from local ports on the router. I think this is probably the most common reason. Without DD-WRT, there’s no easy way to fix this. Solution: One simple way to solve this is to set a hard-coded DNS entry in /etc/hosts (Linux) or c:\WINDOWS\system32\drivers\etc (Windows) as follows. 
    192.168.x.x        my.site.net

    192.168.x.x is the internal IP address of the WebServerHost

Share

Best Resources to Learn about Linux Kernel Internals

Feb 7th

Posted by Pradeep Padala in Open Source

No comments

The Source

The best resource is the kernel source.

Books

Obviously, it’s not that easy to dive into thousands of lines of code. I suggest starting with reading the books explaining Linux kernel in general.

  • Understanding Linux Kernel. This is one of the first books to provide in-depth explanations. Vastly improved over multiple editions, the current one is a very good read.
  • Robert Loves’ book. Love is a core developer, who implemented pre-emptive kernel and other features. I haven’t personally read this, but I have seen good reviews of this.
  • If you are specifically looking for networking aspects, this is an excellent book on understanding linux networking internals.
  • For device drivers, similarly, this book is very useful.

Other resources

  • LWN’s kernel page has lots of great articles explaining kernel internals
  • TLDP’s TLK. Somewhat outdated, but useful.
  • Linux Journal’s KernelKorner has some excellent articles, most of which are freely available online.
  • The Linux Kernel Hackers’ Guide, compiled by Michael K. Johnson of Red Hat fame. Includes among other
    documents selected Q/A’s from the linux-kernel mailing list.

HOWTOs

Lists of links

Share
img5

How to Convert PowerPoint Images to PDF

Jan 13th

Posted by Pradeep Padala in Tips

No comments

I am doing this a lot more often these days, want to convert the pretty images I created with PowerPoint into PDF, so that they are more amenable to Latex/PdfLatex. For fun, I prepared an illustrated HOWTO. Here you go, should be easy to follow.

Let’s say you have a slide with following picture.

Presentation slide with an image

First, you want to save the slide as PDF.

Don’t save it yet. Choose options to make sure that you are selecting only one slide.

As soon as you save it, Adobe Acrobat will open it. You may want to remove all the extra white space around it. To crop the image, you will need the Acrobat Writer (Reader is not enough to do the following task). Choose the crop tool, as shown below.

Choose the region you want to crop, and double click inside the region. You will see a dialog similar to the one shown below.

Press ok, and you are done.

Have fun!

P.S. On Mac, you don’t have any hassle. Just select the region you want to put in image. Right click and save as picture (choose PDF as the file format).

Share

Using kpartx to Mount VM Disk Images

Sep 29th

Posted by Pradeep Padala in Tips

3 comments

If you are using a full disk image for your VM, instead of using a file system, you cannot simply mount it directly using 

mount -o loop d0.img

Use the following commands to see the partitions in your disk, and then mount them.

# kpartx -av d0.img
add map loop0p1 : 0 29333504 linear /dev/loop0 2048
add map loop0p5 : 0 1380352 linear /dev/loop0 29337600
# mount /dev/mapper/loop0p1 /mnt
# ls /mnt
bin   cdrom  etc   initrd.img      lib         media  opt   root  selinux  sys  usr  vmlinuz
boot  dev    home  initrd.img.old  lost+found  mnt    proc  sbin  srv      tmp  var
# unmount /mnt
# kpartx -d d0.img

The last two commands are to unmount and delete the partition mappings.

Share
,

How to Setup Xen4 and pvops 2.6.32.x on CentOS 5.5 using GITCO repository

Sep 28th

Posted by Pradeep Padala in Tips

11 comments

See my howto on setting up Xen4 + pvops 2.6.32.x first before following this.

  1. Do steps 1, 2 and 3 described in the howto
  2. Install Xen 4.0.1 from GITCO repository. The benefit of this is that all the required packages,
    Xen enabled Linux kernel (2.6.18) and libvirt are correctly installed and setup

    wget http://www.gitco.de/repo/GITCO-XEN4.0.1_testing_x86_64.repo
yum -y install xen
  3. Get pvops kernel 
    git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
cd linux-2.6-xen
git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x
  4. Build the kernel to match your configuration 
    make menuconfig

Follow the remaining steps in the howto to setup grub, and test it.

Share
, , ,

How to Setup Xen4 and pvops 2.6.32.x on CentOS 5.5

Sep 28th

Posted by Pradeep Padala in Tips

30 comments

Setting up Xen4 has been tricky, because of the move to the latest Linux kernels with paravirt_ops. Compiling it on CentOS, which uses 2.6.18 kernel, and old toolset is a bit more challenging. The following are my notes for setting up Xen4 + pvops 2.6.32.x on CentOS 5.5 on BL460c blades.

Important note: I couldn’t get Xen 4.0.1 release work on CentOS 5.5 on our blades. You have to get the latest Xen and pvops kernel available from http://xenbits.xen.org/xen-4.0-testing.hg and http://www.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git/ (4.0.2-rc1-pre and 2.6.32.21 as of this writing). This may or may not be a problem for you.

  1. Install prerequisite packages 
    yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"
yum -y install transfig wget texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial glibc-devel
  2. Install rpmforge repository. Follow the instructions on CentOS Wiki. 
    wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm
rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
rpm -K rpmforge-release-0.5.1-1.el5.rf.*.rpm
rpm -i rpmforge-release-0.5.1-1.el5.rf.*.rpm
yum check-update
  3. Install mercurial and git 
    yum -y install mercurial git
  4. Compile Xen 
    hg clone http://xenbits.xen.org/xen-4.0-testing.hg
cd xen-4.0-testing.hg
make -j16 xen
make -j16 tools
make -j16 stubdom
  5. Compile pvops kernel 
    make kernels
make -j16 dist

    This is the most critical step, and if you don’t configure the kernel correctly, booting will fail badly. make kernels builds the kernel with the default .config. To modify it, change to build-xxx directory and run make menuconfig.

    Make sure to have the following in your config file

    # These are important for CentOS/RHEL
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y

CONFIG_ACPI_PROCFS=y
CONFIG_XEN=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=32
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PCI=y
CONFIG_PCI_XEN=y
CONFIG_XEN_BLKDEV_FRONTEND=m
CONFIG_NETXEN_NIC=m
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_XEN_KBDDEV_FRONTEND=m
CONFIG_HVC_XEN=y
CONFIG_XEN_FBDEV_FRONTEND=m
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_XENBUS_FRONTEND=m
CONFIG_XEN_PCIDEV_FRONTEND=y

    If you want to compile xen-evtchn, xen-netback, xen-blkback and xenfs as modules, use the following in your config.

    CONFIG_XEN_DEV_EVTCHN=m
CONFIG_XEN_BACKEND=y
CONFIG_XEN_BLKDEV_BACKEND=m
CONFIG_XEN_NETDEV_BACKEND=m
CONFIG_XENFS=m
  6. Install Xen and kernel 
    make install-xen
make install-tools PYTHON_PREFIX_ARG=
cd build-*
make modules_install
make install

    For make install-tools the PYTHON_PREFIX_ARG is REQUIRED. On Fedora systems, this is not required.

  7. Create initrd (above make install usually takes care of creating an initrd matching your configuration, but just in case) 
    depmod 2.6.32.21
mkinitrd -v -f /boot/initrd-2.6.32.21.img 2.6.32.21
  8. Update grub 
    title Xen 4 + 2.6.32.21
    root   (hd0,0)
    kernel  /xen-4.0.gz dom0_mem=4096M loglvl=all guest_loglvl=all
    module  /vmlinuz-2.6.32.21 root=/dev/cciss/c0d0p1 ro nomodeset console=tty0 console=ttyS0,115200n8
    module  /initrd-2.6.32.21.img
  9. Add the following to /etc/modprobe.conf to load all the Xen modules 
    install xen /sbin/modprobe xen-evtchn; /sbin/modprobe xen-netback; /sbin/modprobe xenfs; /sbin/modprobe xen-blkback; /bin/true

    OR, add the following to /etc/rc.local (I had trouble making the modprobe work)

    if ( uname -r | grep "2.6.32.21") ; then
   for mod in xen-evtchn xen-netback xen-blkback xenfs; do
        /sbin/modprobe $mod
   done
   /etc/init.d/xend start
fi
  10. Add xenfs to /etc/fstab 
        none /proc/xen xenfs defaults 0 0
  11. Test it 
    echo "savedefault --default=X --once" | grub --batch (where X is the entry for Xen)
reboot
/etc/init.d/xend start
xm list

Troubleshooting

  1. The biggest problem will be kernel not booting properly. There might be various reasons.
    • Make sure to use the right set of configuration options for your hardware
    • Make sure to have all Xen options correctly compiled
    • out_of_memory kernel OOPS
      • CONFIG_XEN_MAX_DOMAIN_MEMORY=32 # change this to a lesser number
      • Add more memory to dom0 with dom0_mem=4096M
    • Network card not appearing – make sure to have the right network module (bnx2x for BroadCom NetXtreme II card) inserted
  2. Missing /dev/xen entries. Means that you haven’t loaded the xen-evtchn and xen-gntdev. Run 
    modprobe xen-evtchn xen-gntdev

    See step 9 above to update /etc/modprobe.conf

  3. If you see the following in /var/log/messages 
    xenconsoled: Failed to contact xenstore (Connection refused)

    Do step 2 to insert all the appropriate modules, and verify that you can start xend

Installing VMs

  1. First, verify that the initrd includes all the necessary Xen frontend modules. This
    is not necessary for hvm VMs, but for paravirtualization it’s crucial to have all
    the right netfront and blkfront modules installed. A sample mkinitrd for domU is
    below

    mkinitrd -v -f /boot/initrd-2.6.32.21-domU.img  --with=blktap \
        --with=xen-blkback --with=xenfs 2.6.32.21
  2. Install virt-manager 
    yum install virt-manager
  3. Configure /etc/xen/xend-config.sxp to enable access through libvirt and virt-manager. Add the following lines 
    (xend-http-server yes)
(xend-unix-server yes)
(xend-port        8000)
(xend-relocation-port 8002)
  4. Restart xend 
    /etc/init.d/xend restart
  5. Start virt-manager as root
  6. Install a VM as usual

References

  1. http://wiki.xensource.com/xenwiki/2.6.18-to-2.6.31-and-higher
  2. http://wiki.xensource.com/xenwiki/XenParavirtOps. This is the most authoritative reference for compiling the pvops kernel.
  3. http://wiki.xensource.com/xenwiki/Xen4.0
  4. http://bderzhavets.wordpress.com/2010/04/24/set-up-ubuntu-10-04-server-pv-domu-at-xen-4-0-dom0-pvops-2-6-32-10-kernel-dom0-on-top-of-ubuntu-10-04-server/ (this is Ubuntu specific, but has some useful details)
Share
, , ,

How to Choose a CS Research Topic?

Jun 13th

Posted by Pradeep Padala in Research

4 comments

This post is motivated by questions from some of the new grad students looking for research topics. Here are a few things that can help you in this quest.

Follow your passion

Often, when you take an undergrad course, you wonder about a few things. Why is this system designed like this? Why can’t I use that sorting algorithm? etc. If you are more curious, you probably use Google to find more information, and you develop an interest. As you learn more, you realize that among the sea of questions, there are a few that you really want to go after. Go for it!

Learn state-of-the-art

State of the art is probably the most mis-used word in marketing, but in research it has a very concrete meaning. When some one says that they understand state of the art, that means they know every thing that’s been done so far, and the latest technology that probably is the best.

Good way to learn is to start reading latest publications from reputed conferences. For CS, journals are just treated as archival data, so it is not that important to read journals. Finding reputed conferences in a field is pretty easy. Good conferences can be found from Citeseer impact ratings, rankings maintained here and here.

The next step is to start reading publications. Reading papers is an art in itself and will require another long post. First, try to find papers that are of interest to you. One of the problems faced by newbies is that they don’t understand the paper fully, because some papers point to prior work that they are not aware of. You have to recursively read more papers pointing to prior/related work, and understand how people solved the problem. It’s a difficult and some times boring task, but you will get better at it.

In the end, you should be able to summarize the state of the art by listing the first work that tried to solve the problem to the latest. Often, there are solutions that have trade offs, and there may not be a single best solution. Your job is to figure out why some thing works and why some thing does not
work.

You may even find that there is no solution, and there you have your research topic.

Broad and narrow enough

The topic you choose should be broad enough to puruse for 2 or 3 years. It should still be narrow enough for you to solve it completely. For example, building the next generation search is a topic for multiple PhDs, but developing better clustering algorithms might be a good topic.

Follow your advisor

: Advisors vary in how they assign topics to students. Some advisors have certain research topic in mind, and they want you to pursue that specific path. Some (like my advisor Kang Shin) give you complete freedom to choose a topic. There are benefits and downsides to both approaches.

If your advisor gives you a specific research topic, and if you like it, you are all set. You can start reading papers related to that topic, and start formulating a research problem. Your advisor is probably aware of the state of the art and might also have thought of possible solutions. The downside is that you are stuck with it, and you may feel like you are forced in a certain direction.

Free-form topic choosing gives you great flexibility. Its benefits are obvious, as you can choose whatever you like within certain area (say software systems). Some people (like me :-) ) often have strong opinions on solving certain problems, and want to pursue them. However, you are bound to make some mistakes in the process. It is not always easy to judge state of the art, and it is not easy to find a concrete research problem in a big area. I originally wanted to work in grid scheduling, but I switched after reading a survey paper with 640 references. This surely will waste some of your time, but the process is highly rewarding.

Work with a fellow graduate student

If you are part of a research group (usually your adviser’s), then you will see senior graduate students who have already chosen a topic and/or close to graduation. They have done the hard work, and understand the pitfalls in research. Often, they can give you certain problem, which can be a PhD topic in itself. It is also a good way to learn more about an area that you are not familiar with. There might even be a continuation project to what your fellow student is doing. Downside is that the topic might be too narrow and may not be big enough for a PhD. Nevertheless, the experience is worth it.

Attend reading groups

Most universities have research students gathering in a group to discuss various research topics. Software systems reading group at UMich is a great place to mingle with other systems students. Usually, students, and some times guest speakers, professors present a paper and the group has a free flowing discussion about the work. You can even present a topic and gather opinions from your peers. You will also learn how to support your arguments, and how to criticize rationally. Also, free food is great :-)

If your university does not have one, you can start one. It doesn’t take much effort to gather a few students with similar interests and start reading papers. Every one has to go through the latest publications, why not do it together?

Don’t get too attached

After you decide on a topic and start spending a few months learning the state of the art, you may find out that some one already solved it or it is too narrow. Don’t get too attached to the topic. It is OK to choose a new topic, and many PhD students change their core focus.

If all else fails, you can try the systems topic generator or
href=”http://www-cse.ucsd.edu/users/mihir/crypto-topic-generator.html”>crypto topic generator or classic Douglas Comer’s topic generator :-)

Share
,

Why I switched to Outlook from Thunderbird?

Sep 7th

Posted by Pradeep Padala in Tips

No comments

I can hear the religious open source fanatics (I used to be one of them) screaming to burn me on the stake. Well, there’s a simple reason for the swtich: Outlook has more features and makes me more productive. Isn’t that what we should care about? I know people who still stick to Pine, because they thing it’s a geeky thing to do and they criticize people who use dumber GUIs. Guess what? The dumb people are doing more work than you, while you still search help to find the right key to press.

I have grown over the religious debates like “Is Windows better than Linux?”, “Pine vs. Thunderbird”, “Gnome vs. Kde”, “VIM vs. EMacs”. Most of them are just pointless fanaticism. Using and having seen both sides of all these debates, I have come to the conclusion that the only thing that matters to me is what makes me more productive.

If you are wondering what features I like in Outlook over Thunderbird, here they are

1. The very useful calendar feature. Yes, I have used Evolution and Google calendar, and they both suck.
2. The weekly, daily view in the message view pane. Such a simple feature, but very useful.
3. Ability to color up your messages. red, green, blue or whatever.

Share

Linux tips: mounting a disk image with multiple partitions

Jun 6th

Posted by Pradeep Padala in Tips

No comments

It’s been a while since I found a neat trick that I really wasn’t aware of :-) Usually, I am aware of the trick, but just don’t remember the exact details. But, this one is a gem.

How do you mount a disk image with multiple partitions? If an image contains a single partition, we can simply use a command like the following.

mount -o loop image_file /mnt

This will not work if the image_file is a complete disk image containing multiple partitions. I found a neat trick here that explains how to do it. The idea is to get the partition offset and then mount it as follows.

mount -o loop,offset=some_number image_file /mnt

The offset can be found by running fdisk -ul image_file. Detailed information is available here. Enjoy!

Share

HTML for my fellow non-cs bloggers

Nov 20th

Posted by Pradeep Padala in Tips

No comments

Ok folks, you guys (AB, K …) write some interesting stuff, but you need to learn a little bit of formatting to help people read it with out straining their eyes. No offense, just a few simple things to learn to beautify your posts.

First, you should start using the “Edit Html” tab in Blogger instead of the “Compose” tab. At worst, you don’t use any HTML and it will look exactly the same as you are used to seeing.

HTML is a markup language. Think of it as writing a blob of text and then marking it with different colors/styles. So you write some text and then use tags to mark it up. For example, to bold some text, you would use <b> your text </b> and it will look like your text. Note that every tag has an ending tag and it’s a good practice to end every tag even though some tags don’t have ending tags.

Some commonly used tags while blogging.

Paragraphs: <p> </p>
Line break: <br />
Style:
  bold: <b> </b>
  italic: <i> </i>
  underline: <u> </u>
  colors: <font color="red"> </font>
Preformatted text: Usually used for verbatim listing <pre> </pre>
Quoting: <blockquote> </blockquote> Try this one, it's nice.
Lists: these are slightly complicated, so use the menu available
in the compose button to create them.

If you are writing large amounts of text, you can create headings of different sizes with <h1> <h2> <h3> <h4> For example, H3 size heading looks like

A heading

That’s all for now, Blog away !!!

Share
12»