fsdb_ufs - The UFS File System Debugger

Previously, I mentioned that I had run across fsdb and planned to spend some time playing around with it. I had some time last week, so here's a few simple notes on it.

There are 2 automagic variables in fsdb: ";" and ".". ";" represents the current i-node, and "." is the current location on disk. Between the two of them, you can do things like "go to i-node X and read the first 16 bytes." Then, since "." has been updated, you can readily read the next 16 bytes from the file, and so on until you hit the end of the file or disk.

All commands (except "shell escape") must begin with an i-node specification. ":" is the current i-node, but you can specify another i-node with "#:inode". The third part of the command tells fsdb how to update the value of ":". "inode mode" updates the value to the given inode, as long as it's already allocated. "dir mode" updates : to the given directory entry, ordered by inode (this is not the same order as the directory listing is in). This makes it much easier to modify directory entries than trying to fill over the correct bytes in the directory.

"?" is the "formatted output" command. It is similar in feel to gdb's "print" command, but there are file-system specific formats, such as "inode" and "dir." ("/" is unformatted output).

Some examples:

/dev/lofi/1 > 2:inode?ino
i#: 2              md: d---rwxr-xr-x  uid: 29b7          gid: 12c     
ln: 6              bs: 2              sz : c_flags : 0           200                 

db#0: 170          
        accessed: Mon Nov 30 16:31:46 2009
        modified: Mon Nov 30 16:26:25 2009
        created : Mon Nov 30 16:26:25 2009

inode 2 is special in UFS; it is always the root of the filesystem (inode 3 seems to always be lost+found, as well). The output format looks curiously like the output from stat, except that it doesn't actually stat() the file.

/dev/lofi/1 > 2:
/dev/lofi/1 > :ls -l
/:
i#: 2           ./
i#: 2           ../
i#: 4           bin/
i#: d           etc/
i#: 3           lost+found/
i#: 5           sbin/

:ls is a shorthand for listing the current directory or inode. There is also :cd which is easier than listing the current directory, finding the entry you want, and then updating the value of : to it.

Note that all the inodes are in hex. There is a :base command to change both the input and output bases (:base=0xa is decimal; :base=0x8 is octal, etc).

Updating a directory entry is done with N:dir=M. This sets directory slot N (starting from 0) to inode M. For example:

/dev/lofi/1 > :cd bin
/dev/lofi/1 > :ls -l
/bin:
i#: 4           ./
i#: 2           ../
i#: 9           alink?
i#: a           blink?
i#: b           clink@
i#: 6           cp*
i#: 8           ls*
i#: 7           mv*
/dev/lofi/1 > 6:dir:nm,10/c
   58454:       b   l   i   n   k   \0  \0  \0  \0  \0  \0  \?  \?  \?  \0  \?
/dev/lofi/1 > 6:dir=0x09
i#: 9           blink
/dev/lofi/1 > :ls -l
/bin:
i#: 4           ./
i#: 2           ../
i#: 9           alink?
i#: 9           blink?
i#: b           clink@
i#: 6           cp*
i#: 8           ls*
i#: 7           mv*

So, what have we done here? Since blink and alink now share the same inode number, they are hard links of the same symlink. Running 9:inode?ino reveals that fsdb updated the link count for the inode automatically, so we don't have to worry about file system corruption if we unlink one of these later.

N:dir:nm,10/c outputs the first 16 characters of the name of directory slot N (directory slot N's name to ". plus 0x10", unformatted output, characters). The man page seems to point to outputting the name of a file being possible by N:dir:nm,*/c, but that reads to the end of the block, not to the first terminating null.

Renaming a file is done by writing to the "nm" value. This won't allocate any extra space for the filename, so the filename is truncated if it is longer than the already allocated space. Here is the hard way to re-name those nasty files named "\008\008\008\008...".

/dev/lofi/1 > :ls -l
/bin:
i#: 4           ./
i#: 2           ../
i#: 9           alink?
i#: 9           blink?
i#: b           clink@
i#: 6           cp*
i#: 8           ls*
i#: 7           mv*
/dev/lofi/1 > 7:dir:nm,10/c
   58464:       c   l   i   n   k   \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
/dev/lofi/1 > 7:dir:nm="dlink"
i#: b           dlink
/dev/lofi/1 > :ls -l
/bin:
i#: 4           ./
i#: 2           ../
i#: 9           alink?
i#: 9           blink?
i#: 6           cp*
i#: b           dlink@
i#: 8           ls*
i#: 7           mv*

Now you can impress all your sysadmin friends by creating hard links and renaming files.

But what about something you can't do with normal shell tools?

/dev/lofi/1 > :cd
/dev/lofi/1 > :cd etc
/dev/lofi/1 > :ls -l
/etc:
i#: d           ./
i#: 2           ../
i#: e           etc@
/dev/lofi/1 > 2:dir:nm,10/c
  178c20:       e   t   c   \0  t   \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
/dev/lofi/1 > 2:dir=0xd
i#: d           etc
/dev/lofi/1 > :ls -l
/etc:
i#: d           ./
i#: 2           ../
i#: d           etc/

Exit fsdb and remount the file system to see changes (yes, this is required; I suspected caching, but this occurs even with files that haven't been looked at yet).

$ ls -lain etc
total 6
        13 drwxr-xr-x   2 10679    300          512 Nov 30 16:26 .
         2 drwxr-xr-x   6 10679    300          512 Nov 30 16:26 ..
        13 drwxr-xr-x   2 10679    300          512 Nov 30 16:26 etc

And there you have it, a hard-linked directory. ln has been lying to you this entire time.

:wq

GraphViz for Fun and Profit

A long time ago, in a data center far, far away, we had a NIS server. It was a good little NIS server. It knew of usernames, and passwords, and automounter maps, and netgroups. Ah, the joys of NIS netgroups. There is one basic form of netgroup that we used:

([computer],[user],-)

Specifies a computer, user, or computer/user combination ("-" indicates "any user"). In addition, netgroups can contain other netgroups, so you get a nice graph of which netgroups are part of other netgroups.

Along with this wonderful little NIS server, there was an old Python script which would run through and generate a set of webpages to graphically display netgroup relations. If netgroup A contains netgroups B and C, and C contains D, we could check the generated graph to see:

    A
   / \
  B   C
      |
      D

Fast forward a decade or so to an insanely large NIS tree, with netgroups for things we've long since thrown away (e.g. line-printer-access). When we upgraded from NIS to LDAP as the backend directory service, we kept the netgroups around, though. Still in their NIS format. Of course, our wonderful little Python app broke because of the transition to LDAP, but nobody wanted to admit to knowing enough Python to fix it.

So in working on the replacement for our Python app, I got to learn a whole new set of tools. Graphviz is a suite of tools for displaying graphs and trees. A friend of mine showed me his Perl program to generate "dot" diagrams for Tic-tac-toe (turns out the only winning move really is not to play), and I realized that just redoing the generator in ruby or Perl would be much easier than trying to tackle Python's LDAP bindings (not to mention debugging Python).

The "dot" language spec, while exhaustive, isn't very helpful for learning the language. The User Manual, though slightly dated, serves as a much gentler introduction. The syntax allows for a lot of room for error, once you get the basics down: semi-colons are (nearly) optional; whitespace isn't counted for or against you; and nodes that aren't pre-declared are created without complaint.

The most basic directed graph is something like:

digraph G {
    foo -> bar;
    bar -> blort;
    foo -> quux;
}

Generating that from a list of netgroups in ruby is also quite simple:

puts "digraph G {"

netgroups.each do |netgroup|
    netgroup.children.each do |child|
        puts "\"#{netgroup.name}\" -> \"#{child.name}\";"
    end
end

puts "}"

Which can then be used to generate a nice image by:

./generator.rb | dot -Tpng -o image.png

This is exactly what we used to have (except in dia) for our Python/NIS version, except that that had a page for the subgraph at every node. So, clicking on C (using dot-generated HTML image maps) should pop up a page with:

    C
    |
    D

The original script had all the busy work to figure out which nodes to include on subgraphs done in native Python, and tons of generated dia files that got passed on to generate the images and image maps. Graphviz, however, includes "gvpr," an awk-like tool for graphs written in "dot."

BEGIN{} and END{} blocks function exactly the same as in awk.
BEG_G{} and END_G{} blocks are called at the beginning and end of each graph to be processed.
N{} blocks are called for each node, and
E{} blocks are called for each edge.

Just like awk, there is an optional test parameter (inside [], not //) to specify which "things" to operate on. For example:

N [ color == "blue" ] { color = "red" }
E [ color == "red" ] { color = "blue" }

turns all the blue nodes red and all the red edges blue. There is a whole API built in to work with graphs, nodes, and edges within gvpr, which includes a rich set of tools for "subgraphs."

Say we want to generate a graph from a couple nodes in the original graph we were given (known as "root" in gvpr).

graph_t g = subg( root, "SubGraph" ); /* create a new subgraph */
subnode( g, node( root, "C" ) );  /* add node "C" to the subgraph */
subnode( g, node( root, "D" ) );  /* add node "D" to the subgraph */

Now, this is just the nodes. To copy the edges, we could loop through all the edges from "C" (there are for and while in gvpr), or, we can "induce" the edge relations:

induce( g );

Which the man page states "extends g to its node-induced subgraph extension in its root graph." Roughly translated to English, this means "draw all the edges as they are in the original graph."

Add in some writeG() calls to output to files, and several Python files are now around 15 lines of gvpr, 40 lines of ruby, and a shell script to fire it all off.

:wq

Let's Do the Time Warp Again...

I'm reposting the work related stuff from my old blog, in reverse order. Now I can use the personal blog for personal stuff and this one for sysadmin/work stuff, and the two don't have to intermix.

This means that rather than posts getting pushed onto the stack, they're being enqueued, but the all-powerful google will still index them and make them searchable. Also, I'm making sure that the "new" old posts are tagged.

:wq

fsdb, or Thank God for Snapshots

Scanning through the logs as they went flying by today, two of the scariest lines I've seen in a long time went flying past:

fileserver: 2009-Nov-25 17:29:08 ufs:
        [ID 879645 kern.notice] NOTICE: /usr: unexpected free inode 1147, run fsck(1M) -o f
fileserver: 2009-Nov-25 17:29:13 ufs:
        [ID 879645 kern.notice] NOTICE: /usr: unexpected free inode 1145, run fsck(1M) -o f

Now, /usr is important for any system, but a primary file server isn't something we can just take down for an hour to run fsck a couple times. Especially the day before a long weekend. So, how to fix this? Sun helpfully recommends rebooting into single user mode from alternate media (actually, `boot net -s` from the openboot prom, but good luck getting that to happen on a thumper), so the Sun docs are out.

Or are they? Squirreled away in the See Also section of the fsck(1M) man page is clri(1M). This little utility should clear out the data with zeroes and release the inode back into the free list. This is (almost) exactly right, as it will just delete the offending files. But...there are important things in /usr, so what are those inodes supposed to be?

There's an app a utility for that. ncheck(1M) will look at the disk and generate a list of pathnames from inode numbers or inode numbers from pathnames. I had already generated a list of suspects from the errors running `find` on /usr, but it's good to have confirmation.

fileserver# ncheck -i 1145,1147 /dev/md/rdsk/d20
/dev/md/rdsk/d20:
1145    /bin/hd
1147    /bin/hdadm

`ls -i` confirms that these indeed do map to these inode numbers, but ncheck is more complete in that it shows all the hard links to each file. So, now that I know the names of what I'm missing...what am I missing? These are symlinks into /opt/SUNWhd/hd/bin for the thumper and thor hard drive utilities hd and hdadm. The SUNWhd package installs them, so whenever triage is done, it's probably best if they are put back. Thankfully, they're just symlinks.

So, I don't need the data. The only problem with clri is that it won't muck with the directory listing for the file. That's not so much a failing of clri as a bonus (do one thing: clear a file), but if the directory isn't cleaned up, then fsck will fail just as hard. It may even try to undo all my hard work, and we can't have that. In a fit of insanity, I tried pulling up the directory in ed to see if I could do anything with it (it's just a file like everything else, right?), but that way lies even more madness. There must be a tool somewhere to do this (and be less prone to hunan error).

Back to the man pages, and I ran across fsdb(1M). It looks fairly useful, as it lets you debug the file system. If this is anything like gdb (and it slightly resembles it), this includes displaying various values, as well as editing them. But, the man page is very vague and doesn't give any indication of how to actually use it.

Enter the google. fsdb is a wrapper around the file system specific debugger, so the actual relevant man page is fsdb_ufs(1M). The syntax is based on something called adb, which looks like a horrible thing to program in. And I've written brainfuck before (editor's note: don't ever write anything in brainfuck).

So, the next step is to grab a test box and learn how to use this debugging tool. I hear tales on the intarblag that there is a zfs version, too. But for that, I could just have snapshots instead of a tarball in case I explode the real file system. And given some of the things you can do ("fill an area of disk with pattern p"), I fully expect to blow away a few test filesystems learning it.

:wq

Test Blog Post with BlogWriter Lite

This is a test blog post from BlogWriter Lite.

Though I haven't yet used it, BlogPress' free version has a camera button built in; I assume this works in conjunction with the integrated Picaso client to allow posting of images stored on the iPhone or taking images with the webcam and posting them immediately.

BlogWriter doesn't have any image support in the free version, but supposedly the full version does offer HTML and image support.

Neither program allows for turning the phone sideways and using the landscape keyboard, which would be really nice.

:wq

Blogging From the iPhone

I just downloaded a couple mobile blogging apps for iPhone that support the Blogger API. BlogWriter and BlogPress both have free trial versions, so I figured my first attempt to get things working would be with them. If I can't connect at all to blogspot, then it's no money out of my pocket for the experiment.

A word from the not-so-wise: look up your username and password when setting up something on the iPhone. I could have posted this 3 hours earlier if I hadn't been entering the wrong password to my gmail account. Thank you, Firefox, for letting me forget.

After the 3 hours of learning about XML-RPC, the nasty guts of the original Blogger API, and how to integrate with Drupal/Joomla, I finally have two working mobile blogging apps.

BlogWriter seems to be aimed at a larger community of bloggers. It supports Blogger, WordPress, and anything that can speak the Blogger API at a custom URI.

BlogPress, despite being a portmanteau of Blogger and WordPress, only supports Blogger. The configuration is simple (provided you know your password), but it does require both Blogger/BlogSpot and Picaso credentials. They're the same now, though, and thankfully the app pre-filled my Picaso login information.

BlogPress also appears to be able to work with drafts; I don't yet know I'd they're only local to the iPhone or if it's integrated into the main Blogger web interface.

This just in: BlogPress doesn't turn sideways. Really a bummer because the iPhone keyboard is much easier to use in landscape orientation. Spellcheck and autocorrect can only do so much.

That's really all I have to compare until I actually use BlogWriter. So far, though, it looks like blogging on the iPhone is definitely doable, even without Google's SMS and e-mail hacks.

:wq

Making uzbl a little more vim-like

In the process of tweaking my uzbl config to be a little more vi(m)-like, I've made several modifications:

<<, >> (scroll to begin, end) are replaced with the more correct gg and G
b, m (next, previous page) are replaced with vimperator's H and L
 
t opens a new tab in uzbl-tabbed, so I removed it from the config
 
B (add bookmark) replaced with the more correct m
u (load bookmark) replaced with the more correct `
 
y2url is just long; y is the original vi(m) for it anyway
:q should also exit; unfortunately, this happens immediately and I haven't figured out how to require pressing as part of the command

Swapped case on F, fl, and FL to match vimperator better

And finally, since google.com automagically moves the focus to its input area, I commented out the "home page." gh still goes to google, though.

Uzbl-tabbed modifications:

I used to have ^W,h and ^W,l switch between tabs, but haven't gotten this to work correctly in the latest version of uzbl yet. Instead, Ctrl+Tab and Ctrl+Shift+Tab move between tabs.

And, of course, the colors have been modified. I find the color scheme I've got helps improve readability. It also color codes based on SSL/non-SSL, even with tabs that aren't selected.

My current complete config file can be viewed here.

:wq

Vim Spell Checking

Just found out that vim has spell check. If you use vim, this is all kinds of awesome. For the non-gvim among us :se spell is all you really need to know. Since I use vim with mutt to compose email, this is all kinds of awesome since now I don't have to worry about misspellinks in my email.

:wq

Porting Perl+curses to Solaris

While porting a Curses app to Solaris (u6) the other day, I ran into a problem with the Curses::UI library. MakeMaker couldn't find the use_default_colors() function (which should be there somewhere) in the installed ncurses library headers, which Curses::UI uses by default.

It would seem that passing Curses::UI->new() an arg for -default_colors => 0 will make the library not call the use_default_colors() function, which should avoid the error. Apparently this isn't the case; I had to create my GUI with compatibility mode by using -compat => 1.

Additionally, my ncurses installation doesn't support the KEY_RESIZE constant. Fixing this was a simple tweak to the code to ignore it (the library ignores the key anyway, so it's not like I'm losing any functionality to it).

index 9ab97ae..3b39d89 100644
--- a/lib/solaris/Curses/UI.pm
+++ b/lib/solaris/Curses/UI.pm
@@ -383,9 +383,9 @@ sub do_one_event(;$)
     # ncurses sends KEY_RESIZE() key on resize. Ignore this key.
     # TODO: Try to redraw and layout everything anew
     # KEY_RESIZE doesn't seem to work right;
-    if (Curses->can("KEY_RESIZE")) {
-        $key = '-1' if $key eq KEY_RESIZE();
-    }
+#    if (Curses->can("KEY_RESIZE")) {
+#        $key = '-1' if $key eq KEY_RESIZE();
+#    }
     my ($cols,$lines) = GetTerminalSize;
     if ( ($ENV{COLS} != $cols) || ( $ENV{LINES} != $lines )) {
        $self->layout();

A quick wrapper around the original to do OS detection and pick the Curses::UI library at runtime from it, and I now have an app that'll run in curses mode on both Linux and Solaris.

:wq

IRC Wars: weechat vs irssi

The other week, I switched from irssi to weechat for my console IRC client of choice. The box these run on is a FreeBSD 6.2 system, so getting everything working like I wanted with weechat was a bit of a chore.

weechat comes with a couple improvements over vanilla irssi. First, the nicklist "plugin" from irssi is included by default. I thought this was a cool feature the few times that I've seen it used on people's clients, but didn't really realize how much it helps until I started using weechat. I don't think I've done a /names in a week.

Secondly, color support is vastly improved. /set reveals a whole slew of color and styling options, without having to muck around with a separate style config and the correct magic incantations of /save, /load, and /savestyle.

Finally, weechat has a much richer filtering interface. The IRC smart filter automatically collapses people who flap into a single set of join/part messages. It also keeps the text that it actually received, so unlike irssi, you can get the data back after filtering it out.

Now, onto compiling...

I would much rather have used the port of weechat than compile my own version, but that's only 0.2.6 (current is 0.3.0), and running old versions is somewhat suboptimal.

One of the main issues was that weechat has multiple front-ends and the configure script failed to find any of them (the box is headless, so no surprises about missing gtk+/qt libs). I'm only interested in the ncurses front-end, but the default ncurses library on the system doesn't work with wide-characters. Linux-compatibility ncurses does, but only has the lib and not the headers. For most users, this probably isn't a problem, but since I use IRC in languages other than English (specifically, Japanese without wide-characters is painful), it's a must. I actually had my irssi session around still just for this.

So, I had to compile my own ncurses library (--with-widec --with-ext-colors). Unfortunately, weechat only picks up on the wide characters and doesn't have 256-color support. Not really a big deal, but I like my shinies when I can get them. Ncurses 5.7 compiled like a charm the first time, and getting weechat to recognize the new lib is just a matter of correcting CFLAGS.

Now I'm off and running full time with weechat. I guess if irssi is the client of the future, the present isn't doing too shabby, either.

:wq

Browser Wars, Part 1 of ?

After Firefox 3.5 crashed for the 10th time in an hour or something like that, I finally cracked down and started on getting uzbl (pronounced "usable") working.

Install and launch uzbl, and you're likely to not see how nifty this little browser is. There's no title bar, no navigation, no keybindings; you get a blank white screen. What's the big deal? I can do that in something like 20 lines of c using the OpenGL library (it's a 3-D white screen, even).

The first thing to realize is that there is no default configuration. If you want to go to a page, you must launch the program with something like: `uzbl --uri www.uzbl.org`. Now, this isn't very usable, since you have to open a new browser every time you want to go to a new url (like I used to do in lynx before I found the 'g' command).

Thankfully, there is a very easy way to get a basic configuration. If you're used to vimperator (and you should be, if you still use Firefox), then you'll be all set by copying the sample configuration (likely under /usr/share/uzbl/examples) into your XDG_*_HOME locations. The color scheme is a bit ugly, but it gets the basic information across, and you can use all your favorite vim keybindings, except the ones that are bound incorrectly. I reconfigured mine so that top of page is gg, and bottom of page is G, instead of the defaults of << and >>, for instance. If I could indent and dedent text on the fly, those would be the commands for that.

The biggest difference between the uzbl and vimperator keybindings, however, is that uzbl starts in command mode, not insert mode. I've typed my username into the command bar dozens of times already, because I'm used to clicking in a text field and having it steal my input focus. Not so with uzbl! You must hit i to go to insert mode, just like in a real editor. I hear there are ways to make uzbl perform like emacs, but matters of religion aside, it's not that hard to get used to vim-like behavior.

Since uzbl subscribes to the UNIX philosophy, there's also no notion of tabs. This makes it very amenable to being embedded into gtk+ applications, since those can either implement tabbed browsing or not as appropriate. This is a little reminescent of the Microsoft way, with IE browsing-only windows able to be embedded into anything. You know, if you like having all your apps that want to render HTML be vulnerable to .NET-based attacks or something.

In order to get tabbed browsing, there is a Python wrapper called uzbl_tabbed.py to implement it. The source is pretty straightforward; it opens a new uzbl process every time you create a new tab. This can be a great way to test your configuration, since new tabs use the latest configuration, and old tabs stay the same. Of course, if you blow up uzbl_tabbed impressively enough, it won't clear out the fifos and sockets in /tmp and you'll be left with a problem resembling Firefox's "already opened" message. Except with no error message, unless you start uzbl_tabbed.py from a terminal. For me, CADIE is just about perfect for this.

Having now figured out tabbed browsing and how to make the keyboard bindings more sane, I figured the only thing that I needed to keep Firefox around for was flash. Every once in a while, there's a purpose for it, like my bank's online banking login. They still have a "deprecated" HTML version, but I'm not about to be caught locked out with a check waiting to bounce if I can avoid it. To my great surprise, uzbl supported flash right out of the box. I don't know if this is just because of my distro's package, or if it's actually in the version of webkit that uzbl is built on or what, but it means I can do online banking, and the occasional youtube video. Oh, those silly, silly people shaking babies...

So far uzbl can do everything I use Firefox for. What could there possibly be that's bad?

No (or very limited) incremental rendering. This means that clicking on a link sometimes offers absolutely no immediate feedback that something is happening. But, one of the mottos of UNIX tools is that "no news is good news." Unfortunately, the internet being what it is, this is more like a GNU tool where no news is, well, no news. Pages eventually do load, but seem to take forever, since the entire page has to load and render before anything shows up. Since it doesn't clear the old page until the new one is ready, it's easy to click a link a few times without realizing that uzbl is trying to get the page for you.

Downloading files is a little weird. GTK+ already has support for downloading files, so why build it into uzbl? This makes a lot of sense, as it makes the browser more lightweight and doesn't duplicate functionality, but even w3m has a download manager.

uzbl uses webkit. I really don't like webkit as much as Gecko; it isn't as widely-recognized as a "real" rendering engine, so sites aren't designed or tested with webkit browsers. Maybe the iPhone and Chrome are changing this, but Safari seems to have done very little to be taken seriously, so it's an uphill climb. On the other hand, at least it's not IE.

(editor's note: wow, was I right about Chrome making webkit a standard browser)

uzbl_tabbed is pretty bare-bones. Tabs aren't shortened when there isn't enough room; there's just an ellipsis to show that more tabs exist. How many more? Good question. What tabs are over there past number 3? An equally good question. I'm sure this can be fixed by hacking on the script a little, but I haven't started on that yet. There is an option for gtk+ styled tabs, which is (thankfully) turned off by default. I'm guessing that the gtk+ notebook tabs handle this much more sanely than the text tabs. Of course, they also take up more screen real estate, and I've only got 600px from top to bottom.

Bookmarks. What are those? uzbl has no concept of a bookmark. There are scripts to implement this functionality, but one of the coolest extensions I use for Firefox is Xmarks (formerly FoxMarks, formerly Google Browser Sync). Automagically, I can bookmark a page on a computer at school and it shows up on my laptop. Delete a bookmark on my laptop and it's gone from the systems at work. UNIX Firefox doesn't show it; Linux Firefox doesn't show it; Windows Firefox doesn't show it. Pretty freakin' cool. Setting the same thing up in uzbl will likely require rsync and cron and a few hours of mucking with scripts. Or learning git.

Japanese text. This is really an issue I have with gtk+ in general. Kanji are roughly half again as large as kana and extend above and below the baseline. WHY?!? It's nice that this makes the kanji bigger and therefore a little easier to read (pack 29 strokes into 16px high and it's more a Rorschach Test than reading), but it looks like a little kid's handwriting with letters all different heights. Or, since they're printed, maybe like a ransom note cut out of magazines. "If you ever want to see all your characters the same height again..."

All in all, uzbl is pretty neat so far. I dislike being dependent on Python in general (run a Python 2.4 script in a 2.6 interpreter to see why), so instead of hacking on the tabbed browsing script, I may just write my own. Then again, that would mean I'd have to restart my uzbl session, and that's been going for longer than I've gotten out of Firefox since upgrading. Plus, Firefox seems to have amnesia lately and won't restore my tabs from last time, except when it crashes (which, yes, is embarrasing, but leave the witty commentary to Chrome).

If I ever get Chromium working on Linux (I haven't tried in months, so it could be a fun project for next weekend), expect Browser Wars to continue for some time; with more than just Firefox and lynx, I actually get a choice now for my Linux browser. It's about time we caught up to Windows on this one.

:wq

SSH over LPR

A couple weeks ago, somebody asked if it were possible to get printers from school to work through a local cups client (ala, just putting the print server in client.conf). Some google-fu landed me this script for making a cups backend that prints via ssh. Of course, like most things on the intarwebs, it almost does what I wanted it to do.

A little tweaking and I got the script to run on my laptop, using ssh-agent so that I don't have to have passwordless root keys in order to print anywhere (still need them for my fuse-sshfs/autofs hack, though). Since archlinux runs cups as daemon instead of root, the script needs an suid wrapper in order to work properly (daemon can't su to $user).

The instructions are basically the same as for the original script. Install (or install suid wrapper) to /usr/lib/cups/backend . Then restart cups and add the printer. Cups will not show the remote and local users, but they are stored and will work correctly.

Exit codes can be modified to make the "printer" respond differently based on various conditions. For example, `exit 3` causes the queue to be stopped if the print server is not responding.

#!/bin/sh

# SSH-LPR Backend

# The purpose of the back end is to send print jobs to a remote system
# through ssh and lpr on the remote side. It requires that the user who
# wishes to print have a key-based authentication set up for the server in
# question. The printer URI is sshlpr://[ruser[:luser]@]server/queue where
# ruser is the name of remote user if it is different from the local use
# name, luser is the local user account with the SSH key, if different
# from the user who actually printed, server is the host and queue is the
# print queue name as it would get passed to lpr -P queue.

# With no parameters, we need to tell CUPS what we are.
if [ $# -le 0 ]
then
        echo "network sshlpr \"Unknown\" \"LPR thorugh SSH\""
        exit 0
fi

HOME='/root'
export HOME

# If we get the correct number of arguments, as per:
#       $1=job-id $2=user $3=title $4=copies $5=options $6=[file]
[ $# -lt 5 -o $# -gt 6 ] && exit 1

# Parse URL
user=$2
localuser=$2

[ -z "$DEVICE_URI" ] && exit 1

server="`echo $DEVICE_URI | cut -f 3 -d /`"
printer="`echo $DEVICE_URI | cut -f 4 -d /`"
if echo $server | grep '@' > /dev/null 2> /dev/null
then
        user="`echo $server | cut -f 1 -d '@' `"
        server="`echo $server | cut -f 2 -d '@' `"
fi

if echo $user | grep ':' > /dev/null 2> /dev/null
then
        localuser="`echo $user | cut -f 1 -d ':' `"
        user="`echo $user | cut -f 2 -d ':' `"
fi

# sanitize input for title
title=`echo $3 | sed -e "s/'//g" -e 's/"//g'`

# find a running SSH_AGENT
for i in /tmp/ssh-*/agent.* ; do
        if [ -r $i ] ; then
                ssh_auth_old=$SSH_AUTH_SOCK
                SSH_AUTH_SOCK=$i
                export SSH_AUTH_SOCK
                lines=`ssh-add -l | wc -l`
                if [ $lines -eq 0 ] ; then
                        SSH_AUTH_SOCK=$ssh_auth_old
                        export SSH_AUTH_SOCK
                fi
        fi
done

echo "Using SSH_AUTH_SOCK of $SSH_AUTH_SOCK" >&2

LPR="/usr/bin/lpr -U ${user} -P ${printer} -J '${title}' -# $4"

/bin/ping -c 1 -W 1 $server >/dev/null 2>&1
if [ $? -ne 0 ] ; then
        echo "Host is down: $server"
        exit 3
fi

echo "Doing cat $6 | su $localuser -c \"ssh $user@$server \\\"$LPR\\\"\"" >&2

# Has CUPS given us a file in $6?
if [ -r "$6" ] ; then
        /bin/cat $6 | su $localuser -c "/usr/bin/ssh -q -o BatchMode=yes ${user}@${server} \"${LPR}\""
        if [ $? -eq 0 ] ; then exit 0; fi
        exit 3
else
        echo "Cannot read file: $6" >&2
        exit 2
fi

:wq

@ IN SOA localdomain. aaron.localdomain.

So, I finally got my networking set up at home a couple months ago. There's a little web interface to add computers to the network, which automatically updates DHCP and DNS and allows them on wireless.

I also have an IPv6 tunnel and auto-configuration for my /48 (rDNS is annoyingly difficult, thanks to comcast).

So, of course, this weekend I got to break it all!

I've been using dnsmasq on my linux server to do DHCP and DNS. Basically, it looks through /etc/hosts and /etc/ethers and figures out the answers to queries from there. It also has support for built-in TFTP and can send the right options to be able to PXE boot computers. Neat.

But, my router is running FreeBSD, and I want to reload my linux box to OpenSolaris. The linux machine is basically a fileserver, so reloading it means I can use ZFS. Here's the overall plan:

1. reload fileserver
2. ???
3. profit

Step 1 requires that I move the infrastructure (DHCP, DNS, TFTP, etc.) onto the router, which I've been meaning to do anyway. I've got linux compatibility mode turned on on the router, so I could probably run dnsmasq on there, but that's not very impressive. Bind9 and ISC dhcpd is a much cooler option.

Enter `svn branch`.

Today, I spent my time rewriting my networking automagic (the scripts, not the daemon) to generate DNS zone files from my networking database. Of course, the cron will still run on the db server, since I don't want to be firing up make and perl on the router every 5 minutes. Sure, I could, but the db would be hella slow. Now I've got DNS running on 2 boxes, and as soon as I figure out all the merging and branching and moving to make it work correctly, I'll be able to shut down the DNS server on linux.

DHCP will, of course, require me to learn yet another config file format, but it looks similar, and I should just have to tweak the scripts a little bit to make them output the new format. The only hard part of dnsmasq -> named was the conversion from /etc/hosts to the IPv6 PTR records. About 30 lines of perl run through the whole shooting match for that; IPv4 is just an extra sed script.

:wq

ZFS over iSCSI over ZFS

Some stuff I was hacking away on at work today; this is mostly a note to self:

The tl;dr version:

1. double-check commands you type when playing with disks
2. triple-check commands you type when playing with disks
3. see #1 and #2
4. the way to "refresh" the list of iscsi targets on the initiator is `iscsiadm modify discovery -t enable` (yes, I already had it enabled; it works anyway)
5. it's `zpool iostat [pool] [interval]`, not `zpool iostat [interval] [pool]`

So...I set out to find out what happens when we resize a disk that's shared out over iSCSI.

And, just for my own sanity that this will work with real data, I copied over some D&D stuff to the client:

> time find dnd -type f -exec cat {} \; | md5sum
b5f1fc52d09baaf9f3db34408ce9c184  -

real    4m6.008s

I started with a mirror of 2 disks, cleverly named foo and bar, both 10G. Resizing bar to 15G caused the initiator to disconnect, and the pool faulted. Then I removed the faulty disk and added the new one in, and viola! 25G of space instead of my 9G mirror.

...crap; I wanted to mirror that. Backing out changes is a PITA, so I'll create a new disk (cleverly named zot) to be a mirror of foo during the upgrade to 15G.

Before adding zot (removed the targets that I'm not playing with):

fileserver:~# iscsitadm list target
Target: idle/robin/foo
    iSCSI Name: iqn.1986-03.com.sun:02:0fb34688-bd5c-6f54-ba25-ca3d972187f6
    Connections: 1
Target: idle/robin/bar
    iSCSI Name: iqn.1986-03.com.sun:02:2e83e95d-3841-44d7-c97c-c8de9fe7629c
    Connections: 1



client:/# iscsiadm list target
Target: iqn.1986-03.com.sun:02:2e83e95d-3841-44d7-c97c-c8de9fe7629c
        Alias: idle/robin/bar
        TPGT: 1
        ISID: 4000002a0000
        Connections: 0
Target: iqn.1986-03.com.sun:02:0fb34688-bd5c-6f54-ba25-ca3d972187f6
        Alias: idle/robin/foo
        TPGT: 1
        ISID: 4000002a0000
        Connections: 1

um...this is odd, since bar is part of my zpool on the client.

And then I added the new zfs iSCSI target on the fileserver:
mkiscitarget.sh is a wrapper around all the steps:

1. zfs create -s -V ${SIZE} ${TARGET}
2. (optional) iscsitadm modify -l ${INITIATOR} ${TARGET}
3. (optional) iscsitadm modify -p ${TPGT} ${TARGET}

fileserver:~# ./mkiscsitarget.sh idle/robin/zot 10G client 208
fileserver:~# iscsitadm list target
Target: idle/robin/foo
    iSCSI Name: iqn.1986-03.com.sun:02:0fb34688-bd5c-6f54-ba25-ca3d972187f6
    Connections: 1
Target: idle/robin/bar
    iSCSI Name: iqn.1986-03.com.sun:02:2e83e95d-3841-44d7-c97c-c8de9fe7629c
    Connections: 1
Target: idle/robin/zot
    iSCSI Name: iqn.1986-03.com.sun:02:e0fe1c0d-634e-e96e-f8ad-dbc3c304a6e5
    Connections: 1



client:/# iscsiadm modify discovery -t enable
client:/# iscsiadm list target
Target: iqn.1986-03.com.sun:02:0fb34688-bd5c-6f54-ba25-ca3d972187f6
        Alias: idle/robin/foo
        TPGT: 208
        ISID: 4000002a0000
        Connections: 1
Target: iqn.1986-03.com.sun:02:2e83e95d-3841-44d7-c97c-c8de9fe7629c
        Alias: idle/robin/bar
        TPGT: 208
        ISID: 4000002a0000
        Connections: 1
Target: iqn.1986-03.com.sun:02:e0fe1c0d-634e-e96e-f8ad-dbc3c304a6e5
        Alias: idle/robin/zot
        TPGT: 208
        ISID: 4000002a0000
        Connections: 1
Target: iqn.1986-03.com.sun:02:2e83e95d-3841-44d7-c97c-c8de9fe7629c
        Alias: idle/robin/bar
        TPGT: 1
        ISID: 4000002a0000
        Connections: 0
Target: iqn.1986-03.com.sun:02:0fb34688-bd5c-6f54-ba25-ca3d972187f6
        Alias: idle/robin/foo
        TPGT: 1
        ISID: 4000002a0000
        Connections: 0
client:/# format
Searching for disks...done

c1t010000144FF2985500002A004A31B8A6d0: configured with capacity of 10.00GB


AVAILABLE DISK SELECTIONS:
       0. c0t0d0 
          /pci@1f,0/ide@d/dad@0,0
       1. c1t010000144FF2985500002A004A31A1F7d0 
          /scsi_vhci/ssd@g010000144ff2985500002a004a31a1f7
       2. c1t010000144FF2985500002A004A31B8A6d0 
          /scsi_vhci/ssd@g010000144ff2985500002a004a31b8a6
       3. c1t010000144FF2985500002A004A304D18d0 
          /scsi_vhci/ssd@g010000144ff2985500002a004a304d18
Specify disk (enter its number): ^C

an aside: Ben Rockwood used ^D to get out of format in an article I ran across. Since that works, it would stand to reason that `format <&-` would work in bash; so much for reason. `format </dev/null` or `echo | format` work fine, but who wants to type all that?</aside>

Oh...and I changed the tpgt for zot, bar, and foo - 208 is the box's nge0; 1 is actually tpgt 0 on the targets, but it comes across to the initiator as 1; no idea why that is. Evidently, iscsiadm has decided the disks are different (despite matching GUIDs) because the TPGT entries are different.

That's just silly; I'll move them back so it sees its targets again. Besides, my `zpool status` on the client is hanging and unkillable, so maybe the disk is gone and solaris hasn't quite figured it out (40min seek time? sure, that makes sense).

fileserver:~# iscsitadm delete target -p 208 idle/robin/foo
fileserver:~# iscsitadm delete target -p 208 idle/robin/bar


client:/# iscsiadm modify discovery -t enable
client:/# iscsiadm list target
Target: iqn.1986-03.com.sun:02:0fb34688-bd5c-6f54-ba25-ca3d972187f6
        Alias: idle/robin/foo
        TPGT: 208
        ISID: 4000002a0000
        Connections: 1
Target: iqn.1986-03.com.sun:02:2e83e95d-3841-44d7-c97c-c8de9fe7629c
        Alias: idle/robin/bar
        TPGT: 208
        ISID: 4000002a0000
        Connections: 0
Target: iqn.1986-03.com.sun:02:e0fe1c0d-634e-e96e-f8ad-dbc3c304a6e5
        Alias: idle/robin/zot
        TPGT: 208
        ISID: 4000002a0000
        Connections: 1
Target: iqn.1986-03.com.sun:02:2e83e95d-3841-44d7-c97c-c8de9fe7629c
        Alias: idle/robin/bar
        TPGT: 1
        ISID: 4000002a0000
        Connections: 1
Target: iqn.1986-03.com.sun:02:0fb34688-bd5c-6f54-ba25-ca3d972187f6
        Alias: idle/robin/foo
        TPGT: 1
        ISID: 4000002a0000
        Connections: 0

wait...what?!? I tried moving foo back to 208, since that's where the client says it's connected to the disk, but nothing changed. Time to turn off the iSCSI sharing and see if ZFS can figure it out when things come back.

fileserver:~# zfs set shareiscsi=off idle/robin/foo
fileserver:~# zfs set shareiscsi=off idle/robin/bar
fileserver:~# zfs set shareiscsi=off idle/robin/zot
fileserver:~# iscsitadm list target
fileserver:~#



client:/# iscsiadm modify discovery -t enable
client:/# iscsiadm list target
Target: iqn.1986-03.com.sun:02:0fb34688-bd5c-6f54-ba25-ca3d972187f6
        Alias: idle/robin/foo
        TPGT: 208
        ISID: 4000002a0000
        Connections: 0
Target: iqn.1986-03.com.sun:02:2e83e95d-3841-44d7-c97c-c8de9fe7629c
        Alias: idle/robin/bar
        TPGT: 208
        ISID: 4000002a0000
        Connections: 0
Target: iqn.1986-03.com.sun:02:e0fe1c0d-634e-e96e-f8ad-dbc3c304a6e5
        Alias: idle/robin/zot
        TPGT: 208
        ISID: 4000002a0000
        Connections: 0
Target: iqn.1986-03.com.sun:02:2e83e95d-3841-44d7-c97c-c8de9fe7629c
        Alias: idle/robin/bar
        TPGT: 1
        ISID: 4000002a0000
        Connections: 0
Target: iqn.1986-03.com.sun:02:0fb34688-bd5c-6f54-ba25-ca3d972187f6
        Alias: idle/robin/foo
        TPGT: 1
        ISID: 4000002a0000
        Connections: 0
client:/# iscsiadm modify discovery -t disable
iscsiadm: logical unit in use
iscsiadm: Unable to complete operation

Weird; and the file system is still mounted. But I can't `zpool status` (going on an hour now), so it shouldn't still work, right?

> ls dnd/
2e                                   cerat
3.5e                                 dX_skills.ods
4e                                   dnd
...

Yes, I know I have a dnd/dnd; I really need to clean it up.

Okay...looks like things are still hosed; time to pull the plug on the processes:

client:/# kill 726 2531 17260
client:/# kill -9 726 2531 17260
client:/# kill -CONT 726 2531 17260
client:/# for i in `/pkgs/gnu/bin/seq 1 48`; do kill -$i 726 2531 17260; done
client:/#

Now, the box is just mocking me...I think this is why the default is to panic in such a situation.

Sharing all the stuff back out to client so it can come up "clean":

fileserver:~# zfs set shareiscsi=on idle/robin/foo
fileserver:~# iscsitadm modify target -l xitomatl idle/robin/foo
fileserver:~# iscsitadm modify target -p 208 idle/robin/foo

... ditto for bar and zot ...

fileserver:~# iscsitadm list target
Target: idle/robin/foo
    iSCSI Name: iqn.1986-03.com.sun:02:0fb34688-bd5c-6f54-ba25-ca3d972187f6
    Connections: 0
Target: idle/robin/bar
    iSCSI Name: iqn.1986-03.com.sun:02:2e83e95d-3841-44d7-c97c-c8de9fe7629c
    Connections: 0
Target: idle/robin/zot
    iSCSI Name: iqn.1986-03.com.sun:02:e0fe1c0d-634e-e96e-f8ad-dbc3c304a6e5
    Connections: 0
client:/# reboot
Connection to client closed by remote host.
Connection to client closed.

fileserver:~# iscsitadm list target
Target: idle/robin/foo
    iSCSI Name: iqn.1986-03.com.sun:02:0fb34688-bd5c-6f54-ba25-ca3d972187f6
    Connections: 1
Target: idle/robin/bar
    iSCSI Name: iqn.1986-03.com.sun:02:2e83e95d-3841-44d7-c97c-c8de9fe7629c
    Connections: 1
Target: idle/robin/zot
    iSCSI Name: iqn.1986-03.com.sun:02:e0fe1c0d-634e-e96e-f8ad-dbc3c304a6e5
    Connections: 1

Yes, it connected just after I told it to reboot and SSH kicked me out.

And, back to the client, now sitting at a white screen of unlife:

stop+a
Type 'go' to resume
> boot

On reboot:

NOTICE: iscsi session(12) iqn.1986-03.com.sun:02:0fb34688-bd5c-6f54-ba25-ca3d972187f6 online
NOTICE: iscsi session(9) iqn.1986-03.com.sun:02:2e83e95d-3841-44d7-c97c-c8de9fe7629c online
NOTICE: iscsi session(6) iqn.1986-03.com.sun:02:e0fe1c0d-634e-e96e-f8ad-dbc3c304a6e5 online

You'll notice these are the iqn's for foo, bar, and zot, respectively. After cde-login comes up, the box helpfully tells me that my pool has faulted.

After logging in and running a `zpool status`, it says that bar is offline. Waiting a little makes bar come back. Weird, but at least it recovered. It's not a mirror, but I still only seem to have 10G out of my 2 10G disks. I hate using the Windows approach in UNIX, though.

client:/# zpool status trump
  pool: trump
 state: ONLINE
 scrub: none requested
config:

        NAME                                     STATE     READ WRITE CKSUM
        trump                                    ONLINE       0     0     0
          c1t010000144FF2985500002A004A31A1F7d0  ONLINE       0     0     0
          c1t010000144FF2985500002A004A304D18d0  ONLINE       0     0     0

errors: No known data errors
client:/# format </dev/null
Searching for disks...done

c1t010000144FF2985500002A004A31B8A6d0: configured with capacity of 10.00GB


AVAILABLE DISK SELECTIONS:
       0. c0t0d0 
          /pci@1f,0/ide@d/dad@0,0
       1. c1t010000144FF2985500002A004A31A1F7d0 
          /scsi_vhci/ssd@g010000144ff2985500002a004a31a1f7
       2. c1t010000144FF2985500002A004A31B8A6d0 
          /scsi_vhci/ssd@g010000144ff2985500002a004a31b8a6
       3. c1t010000144FF2985500002A004A304D18d0 
          /scsi_vhci/ssd@g010000144ff2985500002a004a304d18
Specify disk (enter its number):
client:/# zfs list trump
NAME    USED  AVAIL  REFER  MOUNTPOINT
trump  1.59G  8.19G  1.59G  /disk/trump
client:/# zpool iostat trump 30
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
trump       1.59G  23.2G     13      0  1.59M      0

Weird...looks like it still thinks the devices are 10G and 15G.

fileserver:~# zfs get volsize idle/robin/foo
NAME            PROPERTY  VALUE           SOURCE
idle/robin/foo  volsize   10G             -
fileserver:~# zfs get volsize idle/robin/bar
NAME            PROPERTY  VALUE           SOURCE
idle/robin/bar  volsize   10G             -
fileserver:~# zfs get volsize idle/robin/zot
NAME            PROPERTY  VALUE           SOURCE
idle/robin/zot  volsize   10G             -

And a quick data integrity check:

> time find dnd -type f -exec cat {} \; | md5sum
b5f1fc52d09baaf9f3db34408ce9c184  -

real    4m26.241s

Time to attach zot to the mirror and wait for it to resilver.

client:/# zpool attach trump c1t010000144FF2985500002A004A31A1F7d0 c1t010000144FF2985500002A004A31B8A6d0
client:/# zpool status trump
  pool: trump
 state: FAULTED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scrub: resilver in progress for 0h0m, 6.39% done, 0h3m to go
config:

        NAME                                       STATE     READ WRITE CKSUM
        trump                                      FAULTED      0     6     0  insufficient replicas
          mirror                                   ONLINE       0     0     0
            c1t010000144FF2985500002A004A31A1F7d0  ONLINE       0     0     0
            c1t010000144FF2985500002A004A31B8A6d0  ONLINE       0     0     0
          c1t010000144FF2985500002A004A304D18d0    UNAVAIL      0     6     0  cannot open

errors: No known data errors

0h3m became 0h6m; 0h6m became 0h10m; 0h10m became 0h30m...I went to get food. I'll work on this more later.

:wq