Amazon Ate My EBS Volume, Stole My CPU

(I apologize if the title of this post is a bit trollish for this blog, but I’m a pretty frustrated with AWS right now.)

I guess I’m just unlucky. Amazon claims that EBS volumes with 20G of data

… can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives.

Over the Thanksgiving weekend, the new admin server I recently moved to EC2 stopped responding. Turns out I have a corrupt EBS volume — it won’t boot, and the AWS admin console hangs trying to snapshot it. I started a thread about the issue on Amazon’s forums, but in the meantime I need my primary DNS server, so I went ahead and created a new instance and restored the server from my own backups.

All is well, right? Not exactly.

Continue reading

Exception Handling in Gearman PHP Extension

I’ve been using the gearman extension at work the last few days. (We’ve been discussing open sourcing the project, so I may have lots more to say about it in the near future.)

One thing I ran across yesterday was that exceptions don’t actually work in the latest 0.8.0 release. More specifically, calling GearmanJob::sendException() in a worker does not trigger the exception callback set by GearmanClient::setExceptionCallback() on the client. I spent about an hour trying to figure out what was wrong with my code before I looked to see if there was a bug in the extension. I found bug 59706 which is somewhat related, and a blog post by the patch’s author which was more informative. Since the post is dated August 2011 and the latest 0.8.0 release was in May, I’m assuming that the fixes described in the post aren’t in the release I’m using. Regardless, since triggering an exception will still trigger the fail callback afterward, I think I’ll stick with 0.8.0 and wait on using the new exception functionality.

The gearman client I’m writing is aggregating log messages from parallel workers, so for now I’m using GearmanJob::sendData() with an encoded log priority and log message in the data. If the worker runs into trouble, it calls sendData() with a log priority of ERR (the exception message is sent as the log message), then it calls sendFail() and exits.

// worker code
function addFunctionCallback(GearmanJob $job)
{
    try {
        // ... something that fails
    } catch (Exception $e) {
        $job->sendData(Logger::ERR . ' ' . $e->getMessage());
        $job->sendFail();
        return;
    }
}

This actually makes things a little simpler on the client — I only have to implement a single data callback and log the various log messages and priorities that come in. I don’t need to implement the exception callback and have special-case code to log an exception.

// client code
function dataCallback(GearmanTask $task)
{
    list($priority, $message) = explode(' ', $task->data(), 2);
    $this->logger->log($message, $priority);
}

Another weird thing I’m seeing is GearmanTask::returnCode() on the client always returns GEARMAN_SUCCESS, even in the fail callback where I’d expect it to return GEARMAN_WORK_FAIL. It’s not a huge deal — I can tell the worker failed because I’m in the fail callback, but it still strikes me as odd. I’m not sure if I’m just misinterpreting the documentation or if it’s an actual bug. I guess I’ll wait for the next release of the extension to come out and if it’s still that way, I’ll file a bug against the extension.

Moving WordPress Pages to the Domain Root

I wanted my About, Contact, and Résumé pages to live in the domain root (i.e., http://jontai.me/) instead of under /blog/. The following tiny WordPress plugin and .htaccess file placed in the document root does exactly that.

First, the plugin:

<?php
/*
Plugin Name: Pages in Root
Plugin URI: http://jontai.me/blog/2011/11/moving-wordpress-pages-to-the-domain-root
Description: Serve pages out of the domain root
Author: Jon Tai
Author URI: http://jontai.me/
Version: 0.8
*/

add_filter('page_link', 'pages_in_root_page_link');

function pages_in_root_page_link($url) {
	$parsed_url = parse_url(get_home_url());
	return str_replace($parsed_url['path'] . '/', '/', $url);
}

And the .htaccess file:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^about/?$ /blog/about/ [L]
RewriteRule ^contact/?$ /blog/contact/ [L]
RewriteRule ^resume/?$ /blog/resume/ [L]
</IfModule>

Configuring nginx to Serve Nagios CGI on Ubuntu 10.04

In Ubuntu 10.04, the nagios-common package ships an Apache config file in /usr/share/doc/nagios3-common/examples/apache2.conf, which the nagios-cgi package’s postinst script automatically installs if you happen to be running Apache. If you are running nginx, you will have to configure nginx yourself. This blog post shows the basic pieces involved, but vlad compiles Nagios from source. If you’re using the nagios3 package in Ubuntu 10.04, things are a bit simpler.

Besides installing the obvious nagios3 and nginx packages, you will need to add Brian Mercer’s fcgiwrap PPA and install the fcgiwrap package. (fcgiwrap didn’t make it into Ubuntu proper until 10.10.)

add-apt-repository ppa:brianmercer/fcgiwrap
apt-get update
apt-get install nagios3 nginx fcgiwrap

Configure Nagios and nginx the way you want. When you’re ready to set up the section for the Nagios CGI, make it look something like this:

server {
        listen 80 default;
        server_name localhost;

        location / {
                root /var/www/nginx-default;
                index index.html index.htm;
        }

        location /nagios3 {
                alias /usr/share/nagios3/htdocs;

                auth_basic "Nagios Access";
                auth_basic_user_file /etc/nagios3/htpasswd.users;

                expires max;
        }

        location /nagios3/stylesheets {
                alias /etc/nagios3/stylesheets;

                auth_basic "Nagios Access";
                auth_basic_user_file /etc/nagios3/htpasswd.users;

                expires max;
        }

        location /cgi-bin/nagios3 {
                root /usr/lib;

                auth_basic "Nagios Access";
                auth_basic_user_file /etc/nagios3/htpasswd.users;

                fastcgi_param AUTH_USER $remote_user;
                fastcgi_param REMOTE_USER $remote_user;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass unix:/var/run/fcgiwrap.socket;
        }
}

Restart nginx, add a nagiosadmin user to the /etc/nagios3/htpasswd.users file, and you’re all set!