Building fusecompress on Centos 5.5

Setting up a backup solution using rdiff-backup, I realized that we would quickly run out of space without compression. Enter fusecompress: a fuse module to transparently compress the files in the filesystem. There's no RPM available for CentOS, but the current version of the code should behave properly on CentOS 6.

First, there are some build dependencies to install. Since lzma compression in fusecompress is so breaky, I decided to compile without it. Development Tools is roughly the equivalent of Debian's build-essential.

yum groupinstall 'Development Tools'
yum install boost boost-devel boost141-iostreams
yum install fuse fuse-devel zlib-devel bzip2-devel lzo-devel

Now get the source from github and compile it; should be easy, right...

su - build
git clone git://github.com/tex/fusecompress.git
cd fusecompress
./configure --with-z --with-bz2 --with-lzo2 --without-lzma
make

Of course, there are complaints. CentOS 5.5 ships with a 2.6.18 kernel, and Boost v1.33.1. Fusecompress uses syscalls from the 2.6.22+ kernel, and Boost v1.44+.

The first step is to backport the Boost v1.44 issues:

git revert 9d5137d7d067151a9822b40e3687b0f645b33937

Then, I coded up implementations of futimens(3) and utimensat(2). These allow nanosecond control of file mtime and ctime attributes. The old versions are now deprecated, and "only" have microsecond precision. These implementations just use the fact that 1ns = 1/1000µs.

#include <fcntl.h>
#include <sys/time.h>

int futimens(int fd, const struct timespec times[2]) {
   struct timeval tv[2];
   tv[0].tv_sec  = times[0].tv_sec;
   tv[0].tv_usec = times[0].tv_nsec / 1000;
   tv[1].tv_sec  = times[1].tv_sec;
   tv[1].tv_usec = times[1].tv_nsec / 1000;

   return futimes(fd, tv);
}

int utimensat(int dirfd, const char *pathname, const struct timespec times[2], int flags) {
   struct timeval tv[2];
   tv[0].tv_sec  = times[0].tv_sec;
   tv[0].tv_usec = times[0].tv_nsec / 1000;
   tv[1].tv_sec  = times[1].tv_sec;
   tv[1].tv_usec = times[1].tv_nsec / 1000;

   return futimesat(dirfd, pathname, tv);
}

Then some hacking of the makefiles to include my new .hpp and .o files in the outputs, and I have a working fusecompress for CentOS 5.5.

:wq

Upcasting Failure: PHP and String Insanity

One of the most awesome bugs to bite me in PHP is how it handles comparisons. For any two strings, everything's fine. For any two ints, the comparison is what you expect. But, what happens when you compare a string to an int?

<?php
if ('astring' == 0) {
        echo 'insane';
}

This happly little script will print the fact that PHP is, in a word, insane. Clearly, 0 and 'astring' are not the same thing; not even close. 'astring' isn't even falsy.

What's happening here is that PHP is casting both to an integer, instead of to a string. And the rules in PHP say that, since string 'astring' has no leading numerals, it's converted to the number 0. Clearly not what anyone coming from a strongly typed language would expect. The "correct" way is, easily enough, to use === instead of ==.

I thought the insanity stopped there, but then I ran into another gem. We have a select box in one app that allows the user to choose hours, days, weeks, and months:

<select>
    <option value="1/24">Hourly</option>
    <option value="1">Daily</option>
    <option value="7">Weekly</option>
    <option value="31">Monthly</option>
</select>

In the app, we handled this with a switch. It worked beautifully until the time came to add in support for the 'Hourly' option. Javascript kept anyone from actually selecting 'Hourly' until we were done with the backend to support it, but the client wanted to see where the option would be in the final interface.

<?php

switch ($time) {
    case 31:
        // monthly processing
        break;

    case 7:
        // weekly processing
        break;

    case 1: default:
        // daily processing
        break;

    case '1/24':
        // @TODO: add support for this
        break;
}

PHP compares the case labels using == logic. This means everything was coming up daily instead of hourly. Instead, it's better to just leave everything in the switch as a string. Since $input comes in through the $_POST vars, it's a string regardless, and will compare correctly.

But really, why does 0 == 'astring' in the first place, PHP?

:wq

searching for a value in an array in javascript

Just a little thing I happened to whip up for work the other day. I can't believe that this wasn't around in quite this format on the internets, yet.

One of the guys at work has a fondness for ruby because of how easy it makes it to deal with collections and arrays. I agree that's one of the hallmarks of a good language; there's a reason nobody stays sane long doing arrays in bash.

Another strong feature of a language is its extensibility, especially with built-in types. Ruby and javascript have this down to an art. It seems to me like Java really missed the boat with its mix of Objects and built-ins (int, float, etc.).

It's trivial to check whether a key exists in an object or array in javascript. You can exploit this to great effect to make associative arrays.

'undefined' === typeof(arr[key])

But, for some reason, there's nothing inherent in the language to check for a value in an array. What I was really searching for was ruby's awesome Array.include? method.

So, I added one.

Just extend array.prototype and you can add methods to every array object. Add them to Number.prototype, and you can do crazy things like 5.times(function() { ... });.

Note the use of === for comparisons. Not only does using it religiously keep jslint happier, it means that this will work properly everywhere, even if the array contains 0 or ''. Unfortunately, I couldn't come up with any fanciness better than searching (up to) the whole array.

/**
 * Checks whether or not key is contained in this array.
 * This function compares elements with ===.
 *
 * @param Object key
 * @return bool
 */
array.prototype.include = function(key) {
    for (var i = 0; i < this.length; ++i) {
        if (this[i] === key) {
            return true;
        }
    }
    return false;
};

I only wish ? were valid in identifiers so I could have named the function correctly.

:wq

Restoring a System Image from Windows Backup

Previously, I mentioned that I had lost a hard drive, and the importance of backing up files.

If you're running a Windows 7 system, go do yourself a favor and turn on Windows Backup to an external CIFS server. I'll wait.

Everybody backing up regularly now? Good.

Yesterday, I finally threw a new hard drive in my dead system and flipped the switch again. As expected, the system complained that the hard drive was unbootable, and I started rooting around for my Windows 7 disc.

Rather than installing, there's a Repair your computer link. Tell it you want to restore a system image, let it connect to the network, select your system image, and go grab a book. This took me about 40min over a 100Mbit link, so it was comparable timewise to a full install.

That's it. The system will reboot automagically, at which point you probably want to pull out the disc.

That's it; log in and run Windows Update a few times and you're back in business.

:wq

Blogger Editing via ${EDITOR} - GoogleCL

Google CL allows the use of google tools over command line. This allows amazing things like editing google docs in vim (or emacs, if you're into that sort of thing). There's also blogger and picasa functionality, and you can dump your entire google calendar to stdout (in csv format, so start brushing up on your awk).

One of the things that's blatantly missing, however, is blogger integration with your favorite editor. Hopefully, this post will fix that, as it's doubling as my test of google-cl blogger integration and my blogger script.

#!/bin/sh

GOOGLE=/usr/bin/google
BLOGFILE=`mktemp`

ed=${VISUAL-${EDITOR-/usr/bin/vim}}

timestamp=`date -r "${BLOGFILE}" +%s`

${ed} "${BLOGFILE}"

newtime=`date -r "${BLOGFILE}" +%s`

if [ $? -eq 0 -a "${newtime}" -gt "${timestamp}" ]; then
 title=`head -1 "${BLOGFILE}"`
 /bin/sed -i '1d' "${BLOGFILE}"
 echo 'Posting to blog "fmt >/dev/internets"'
 ${GOOGLE} blogger post --title "${title}" "${@}" "${BLOGFILE}"
fi

rm "${BLOGFILE}"

File inclusion courtesy of :r. Now that makes me happy.

:wq

Get the Keys of a Hash in Javascript

I found myself today needing to do something that should be really easy in nearly any language: get a key of a given hash. (In the actual problem, any random key would do.)

After trying all kinds of different jQuery and array hacks, I finally found the solution; sometimes, you spend so much time looking in libraries you forget about the actual language:

var key = (function() { for(foo in obj) { return foo; })();

And that's exactly what I used; since any key is good enough, the first key is the easiest one to grab. Making it an anonymous function means I don't have to muck with break; or the like, and running it immediately means I can save the return value into a variable directly.

You can also use the for(key in Object) construct to get all the keys of any given Object. This is mostly academic, since you could always just use for directly (and without the overhead of a function). It may come in handy for debugging routines, however.

/**
 * Returns the keys of a given object.
 *
 * @param Object obj
 * @return Array
 */
function keys(obj) {
  var ks;
  for (k in obj) { ks.push(k) }
  return ks;
}

:wq

Git Trick: Multiple remotes

git is a wonderful little vcs. It took a while for me to warm up to it, but now I could never go back to svn or cvs. RCS still holds a special place in my heart, and is so damn useful I can't give it up completely. But these days, my vcs world is all git-based.

I'm sure there's a way to do this multiple-remote trick in bzr and hg, but I don't use them, so haven't bothered to figure it out. If I need hg, there's always hg-git, which may earn its own entry soon.

git makes it easy to track multiple remote branches, which is ostensibly used for sharing work between colleagues. i.e. I can 'merge steve/master' to bring all Steve's unpublished changes to master into my current local branch. This is great for hacking in small groups or pairs without needing a central server. However, the true power of this, I think, is that any git url can be a remote.

So, for example, I'm building a site based on dabl, and I want to keep it up to date with the latest changes. Now, up until last week, that meant I would check out dabl, check out my code, and then copy the files in and commit. Something like this:

git clone dabl
git clone my-project
cp -R dabl/{libraries,helpers} my-project/
cd my-project
git commit -a -v -m 'Updated dabl'
git push origin master

Now, of course that `git commit -a` wipes out any local changes. That means that, for example, anything in helpers/ that gets modified in both my-project and dabl will be replaced by the dabl version. Usually, that's fine because I've pushed the changes from my-project upstream. Where there would logically be a conflict, it is happily ignored in favor of the upstream version. Of course, upstream is the same procedure but in the reverse direction, and has the same pitfalls.

The solution is to recognize that these are both git repositories in their own right. Just add the dabl repository to my-project. Updating is as easy as:

git fetch dabl
git merge dabl/master

Now, I have the latest version of my upstream code, with all the benefits of merging it in. This is what I really wanted in the first place. I love cp, but it's never going to give me an octopus merge.

But, that's only one half of the problem. Is there any way pushing upstream could be aided by this?

Okay, it would be pretty anti-climactic if the answer to that were no. As it turns out, pushing upstream is the reverse of pulling from upstream (like with the cp "solution"); you just have to be a little more careful. Lets say I need to push HEAD on master upstream:

git checkout dabl/master
git cherry-pick master
git push dabl HEAD:master

Yes, it really is that easy. "master" is a valid commit-ish, so you can cherry-pick it. You can even do exciting things like master^ and master~3. And of course, you can name a commit by its hash or tag if you want to be really safe. In practice, it generally helps to be really safe.

A word of caution: DO NOT make your other remote call its refs "origin." This will give you and everyone you work with headaches as you will switch repositories instead of merging. And I mean "switch" in the svn sense of the word (i.e. "rebase"). Name it something sane and unique, like the name of the project.

:wq