Saturday, November 26, 2011

Why I switched from Xen to LXC and other regrettable decisions

After playing around a lot with Asterisk I found my closet server was straining under the load. The issue wasn't the hardware, it's just I was running a fairly ancient version of Xen, and I was running that because I couldn't upgrade from Ubuntu 8.04, which was because later versions of Ubuntu didn't properly support Xen (as it was 8.04 barely supported it.)

But it was getting to be a problem. My wife was, understandably, getting upset that all the cool stuff I'd set up with the Asterisk server were somewhat undermined by the fact that after a few hours you couldn't make any outgoing calls. When you did, the entire VM hosting the Asterisk server would freeze for a few seconds, preventing anything from happening.

As an experiment, I made a new VM, and tried to upgrade it to 10.04 using the usual Ubuntu command line tools. This did not work.

I also upgraded the server's memory, but that didn't have any affect either.

So, finally, I bought a big ass drive for the server, and got ready to migrate all the VMs to something that Ubuntu 10.04 would be happy with. And, well, that meant no Xen.

But as it happened, this turned out to be more work than I'd hoped.

A boot up the rear

First problem. Now, I think we can all agree that the PC drive partitioning system sucks. It dates back to the original PC XT, and has nothing going for it. Several efforts have been made to fix the issues with it, or to make something better, including OpenFirmware and EFI, not to mention hacks built upon the existing system like Extended Partitions. The latest attempt to fix the problems is called GPT. It started as a part of EFI, but has started to be supported by more ordinary BIOSes recently because the standard PC system doesn't really do disks over 2T in size terribly well.

So, anyway, the geniuses at Canonical decided to make GPT the default under certain circumstances, including circumstances where it really isn't necessary. If you tell Ubuntu 10.04 to wipe a 2T drive (not a 2.5T drive, that would need it, but an ordinary 2T drive that currently fits within the 2T limit) and put on a fresh Ubuntu install, it will install a GPT partitioning system, and not install a standard partition system.

I found this out the hard way. Now, there is a way around it, for those of us with motherboards more than three years old (I know! We're so behind!) What you do is cat /dev/urandom > /dev/sda (because there's no command to wipe out the GPT partition table, and if you just try to install a normal one using fdisk, then it'll simply be ignored by Ubuntu's installed if there's a GPT thing there too); then you fire up fdisk from the command line, create a new partition table, and create three new partitions, one smallish one (a few gigs) at the beginning, one for your swap partition, and then one big one for the rest of your system.

You then fire up the Ubuntu installer, tell it to format #1 and #3 as ext4, as /boot and / respectively, making #1 bootable, and #2 your swap.

You'll note there's no GUI for this. If you tell Ubuntu to wipe the disk and start fresh, you can't tell it "Oh by the way, please make sure my computer is able to boot from this disk, please?


KVM is not Xen

What's Xen? Well, Xen is a hypervisor. Remember that User Mode Linux thing a long time ago that still exists but nobody uses it? It's a special Linux kernel that's been designed to behave itself so it can run within another operating system. Well, the Xen people went one better and said "Let's make all operating systems run like that, and we'll create a special operating system that's really lightweight in which they can all live." And, well, it works. It's a great idea. And it's what I had on my server.

But Xen isn't universal. Well, actually, it is, because the Xen people recognized early on that not every operating system vendor was going to modify their OSes to play well with others, so they created a special mode for such operating systems that made use of special CPU features, but for Linux, you didn't have to use it. But, nonetheless, people kinda assumed Xen wasn't the way to go because it encouraged rivals to work together, and KVM was born.

KVM simply runs other operating systems under Linux. It's not Xen because those other operating systems don't cooperate, instead KVM simply makes use of features in slightly higher end CPUs to keep the operating systems in line.

Canonical decided to go with KVM and avoided supporting Xen shortly after Ubuntu 8.04 (actually it didn't work very well under 8.04 either.) Unfortunately, however, the very fact KVM needs hardware support means it doesn't replace Xen even if you ignore the major differences in the way they work together.

Unfortunately the cheap CPU in my server doesn't support KVM.

Even if it was, I was already wondering what it would take to migrate my existing VMs. Apparently Redhat is working on a solution, but thus far the nearest I found to a "solution" was a tool called Xenner. I immediately hit a snag (before I realized that my CPU was a piece of crap) as it doesn't seem to take Xen disk images as is, you have to make them more "hard disk" like. Before I had a chance to look for solutions, I found the problem with the CPU.

Other options

At this point, the options were:
  • Try installing Xen server and find some way to make that work after all
  • Look at one of the simpler virtualization solutions like OpenVZ.
OpenVZ isn't directly supported by Ubuntu, but LXC - an OpenVZ spin-off - is. OpenVZ/LXC takes a third approach to "virtualization". Given a fairly common scenario (one that happens to be mine) is to simply run a large number of Linux-based operating systems on a single box, largely to keep different environments from standing on each other's toes and make it easier to experiment, OpenVZ simply runs a single operating system (single pool of processes, single file system, etc), but has the kernel hide this from running processes, who are presented with a sub-view of the running system, Each sub-view appears to each process and user as an independent operating system instance. And the kernel can use quotas and other security tools to limit processes running within a sub-view so they can't take over the entire computer.

Despite the fact Ubuntu nominally "supports" LXC, it's a little messed up under 10.04. To begin with, the 10.04 incarnation actually comes with a major bug that makes the system unusable if you have any major system partitions (such as *cough* /boot) separated from /. Installing from a PPA fixes that issue.

Another is that the documentation is pretty awful. Essentially you're pointed at other people's HOWTOs, that might cover Ubuntu, and which tends to gloss over important details like networking.

What I did was fairly simple:
  • I created directories for each VM under /home/lxc.
  • Each directory contains an lxc.conf file, fstab file, and "root" directory.
  • The root directory was the original VM file system. (I mounted it using losetup, and used cp -a to copy it. Nothing special.)
  • I moved /etc/init.d/udev* and /etc/init.d/check* out of the way as these would cause trouble in the new environment.
  • The lxc.conf and fstabs were cribbed from various blogs. I'd post them, but I'm not sure they're right yet
  • Finally, I set up bridged networking in /etc/network/interfaces. This essentially means everything you'd normally assign to eth0 gets assigned, instead, to br0, and you specify br0 is connected to eth0. 
This, surprisingly, worked. My old Xen 8.04 images are working under a modern kernel, using LXC instead, and actually the entire system feels rather smoother - probably in part because LXC is extremely lightweight.

As if to tell me not to stray too far, Xen gave me a final surprise while I was setting this up. I had to reboot multiple times, switching between the working "old" system, and the new system. At one point I had a shell open on one of my VMs. I then spent fifteen minutes in the new system, before rebooting in the old system so I could access the Internet for a bit.

And to my absolute amazement, the shell session was still alive. When I'd shut down the old system, it had saved the state of the VM, and restored it fifteen minutes later when I rebooted back into the old system.

I love Xen. I really do. Apparently Canonical is rethinking their lack of support for the system. It would be nice to switch back. Perhaps 12.04 will properly support Xen. It's about time they did.

Friday, November 25, 2011

Kerry, Gore, Obama

It would be good to have a time machine and to somehow, without being noticed, distort the results of the 2000 and 2004 elections to see if Gore and Kerry would, like Obama, have betrayed everything they claimed to stand for.

Why I can't recommend Hulu Plus

You have to feel a little sympathetic to Hulu. It's trying (or claiming to be trying) to do the right thing, make TV content legally available online in a way that satisfies the conflicting demands of content creators and people like you and me. To that end, it's produced a fairly pleasant Flash based website that streams a large chunk of what's on TV to any PC webbrowser, supported by relatively short ads, for free.

But there are limitations. Hulu has said it doesn't like them, for example, the fact that the free version of Hulu can't be accessed from mobile devices has to do with the arcane licensing issues Hulu has to deal with.

To that end, Hulu decided to create a premium version that supposedly deals with the licensing issues. People who subscribe to Hulu Plus get more content, and get the ability to see it on mobile devices.

Except... there are two major catches.

The first is we're paying for this service, and it's still infested with ads. Infact, my wife and I counted five breaks during a 22 minute long TV show. To make matters worse, most of the ads are the same, which becomes monotonous after a while. Now, I have no problem in principle with serving ads to free users. But I'm paying for this. Why is this acceptable?

And to those at Hulu who argue that I don't pay enough, who's fault is that? You set the subscription fee. It's not as if there was some negotiation where I said "I'm sorry, I'm willing to buy this, but not willing to spend more than $9 a month." You never even asked. Frankly, even at $9 a month, I don't want ads. But I've never said I'm unwilling to pay more than that.

Ads every four to five minutes is simply not acceptable. It makes the service physically unpleasant to use. The extra content is not desirable if I don't want to use the service to watch it.

The second major catch? The mobile devices thing is crap.

There's an obvious way to implement "Make it available on mobile devices" which is this: stop blocking them. If I'm using a mobile webbrowser with a full version of Flash, there's no reason to stick code in your app preventing me from using it to watch content if I'm a subscriber and if you claim that the entire point of Hulu Plus is to cover that whole "We can't do this due to licensing."

But that would interfere, I suspect, with Hulu's desire to see revenue from "exclusive deals" with tablet makers. So Hulu Plus is only available if a tablet or phone maker has paid Hulu. Again, there's no way around it. You can't even buy the Hulu Plus client. It's either available, if you have a tithed device, or it isn't.

You're asking me to pay what for what? Not going to do it. I pay less for Amazon Prime, get free two day shipping on other stuff I buy, get to borrow books on my Kindle, and don't get ads. The content isn't as full as I'd like, but on the other hand, I can easily chuck a few extra dollars Amazon's way for content that's outside of the free sandpit. And it's worth spending that money because, well, it's actually watchable content. Content I can watch anywhere, even on my mobile devices.

If you want me to buy or recommend Hulu Plus, you need to make it what a sane, reasonable, person would expect it to be. If that means charging a little more, or charging for the Hulu Plus app, then that's fine. That's what you should be doing. Providing half the product that people expect it to be for half the price isn't delivering a bargain, it's just being dishonest.