Maintenance of malabar-mode

As some of my readers may have noticed, not much has happened to malabar-mode in quite a while.  I have simply been too busy, and have started using IntelliJ as my IDE of choice.

So this is a call to interested parties:  If anybody wants to take over malabar-mode, let me know and we'll work out the details.

Permanently fixing Realtek networking on Ubuntu Natty

So here we are again. Kernel upgrade, then reboot => no network. Last week a kernel upgrade borked my network again, and it took a while before I remembered that I needed the r8168 module (see last posting).

Today it happened again. So I think to myself "There must be a better way."  It turns out there is:  It is called DKMS, the "Dynamic Kernel Module Support".  Essentially it works by registering the module for rebuild whenever the kernel is upgraded.

Fixing Realtek networking on Ubuntu 11.04 (Natty Narwhal)

Tonight I made the mistake of upgrading my primary home work computer to Ubuntu 11.04 in the middle of a crunch period at work.  I'd found the perfect time for it, I thought; start the upgrade and get the kids to bed while it runs (monitoring it every now and then).  Once the kids are tucked in, I reboot the box.  So what happens?  

Boom! no network.  dmesg says 

r8169 0000:02:00.0: eth0: unable to apply firmware patch
r8169 0000:02:00.0: eth0: link down
ADDRCONF(NETDEV_UP): eth0: link is not ready

Ouchie; I, like, uhm, really need net access for work.

I'll elide the approximately four hours of increasingly panicky fiddling and Googling that occurred at this point; I feel my blood pressure rising just thinking about it.

End of story:  Check which Realtek chipset you have using lspci. Grab the appropriate proprietary driver, r8168 or r8169 (hope for the 8168, the 8169 driver doesn't even compile without the changes this guy details; luckily I have an 8168) and install it.  I have net again!

Now to do what I originally booted the box for.

OpenVPN with TomatoUSB sauce and TAP dancing

I just bought a Netgear WNR3500L to replace my Linksys WRP400 wireless router. The main reason I bought that particular device is the Tomato firmware, specifically the TomatoUSB variant of it (at http://tomatousb.org/) which has OpenVPN support.

Flashing the firmware was straightforward - I flashed the DD-WRT mini CHK firmware first, and then TomatoUSB (tomato-K26USB-1.28.9051MIPSR2-beta22-vpn3.6, to be precise). One hard reset later Tomato was running.

So far it's been rock solid for me - time will tell how solid it really is.  The only thing that has irked me so far is that it comes with a SIP ALG and SIP NAT-helper turned on by default - this plays havoc with work's equipment (yes, I'm currently in the VoIP business).  Easy enough to turn off, though, once you know about it.

Now for the OpenVPN setup. One thing that immediately struck me was that there is no way to set up auth-user-pass authentication (which is what work uses).  But the "custom" option looks promising...  Let's try it!

Here's the basic page of my config:

Note that interface type is TAP, Firewall is Automatic, and Create NAT on tunnel is checked.

Now for the advanced page:

Here is the custom config.  As there is no user to be queried for credentials when bringing up the tunnel, they  need to be stored on disk somehow.  I'll get to the "how" part later; for now, rest assured that /tmp/ca.crt and /tmp/client1-userpass will be available.

Interesting to note here:

1. We explicitly say that we're a client, as tomato only adds this automatically in the TLS mode; I figure this is a bug in TomatoUSB
2. We set script-security 3 (to tell OpenVPN that it's ok to run our up script)
3. We reference a script in /tmp as our up script; I'll get to why in a bit

So how are the files in /tmp generated?  At boot-time!

Have a look at my init script:

Of course the values here aren't real. :)

As you see, we stuff our credentials and the CA certificate in the files where we've told OpenVPN to look for them; we also generate the up script.

The up script is as it is because of two other (IMAO) bugs in TomatoUSB:

1. You cannot receive DNS server entries from the VPN server in custom mode; to work around this, we call out to the same script as would have been done in the other modes
2. Even though you can tell it to do NAT on the tunnel, and it will even generate the script to set it up (/etc/openvpn/fw/client1-fw.sh), it will never get run if the interface is not TUN.

After all this, I now have always-on VPN on the router.  :)

Using SSH private keys with git and IntelliJ's "IDEA ssh" on Windows

Due to an unfortunate incident with my Linux laptop, I'm stuck working on my Windows gaming rig for a while. That is, of course, an adventure in itself; tonight's adventure was to get private SSH keys working on Windows when using IntelliJ's Git integration.

First step, of course, was to try Pageant. No dice. Then I tried choosing "native ssh" - that just made IntelliJ hang (with no way to cancel - really nice GUI there, guys).

Then the shoe dropped; maybe it pretends to be a "real" ssh client? One short check of my Windows home directory later I'd dropped my id_rsa private key file in %USERPROFILE%/.ssh, and IntelliJ happily picked it up.

Speeding up database-bound tests with MySQL, redux

Kinda long between the posts here; hopefully what I do write is worth waiting for. :)

Way back in February I wrote about Speeding up database-bound tests with MySQL; a nice chap named Adam Monsen used the idea to speed up the build for something called Mifos.

I've since thought of something that's worth adding:

When you restart your computer with this setup, not only does your data go away (well, d'oh, you're storin' it in RAM, whaddaya expect?), but MySQL gets really confused about whether or not your table is there.

If you know why this happens and want the short version, skip the exposition.

For the rest of us, let's see this in action:

First we set the scene:

espenhw@demokritos:~$ mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 121
Server version: 5.1.41-3ubuntu12.6 (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database confusion;
Query OK, 1 row affected (0.01 sec)

mysql> use confusion;
Database changed
mysql> create table confusion_levels (id bigint primary key, level varchar(255)) engine=innodb;
Query OK, 0 rows affected (0.06 sec)

mysql> insert into confusion_levels values (1, 'TOTAL');
Query OK, 1 row affected (0.00 sec)

mysql> show tables;
+---------------------+
| Tables_in_confusion |
+---------------------+
| confusion_levels    |
+---------------------+
1 row in set (0.00 sec)

mysql> select * from confusion_levels;
+----+-------+
| id | level |
+----+-------+
|  1 | TOTAL |
+----+-------+
1 row in set (0.00 sec)

All good. Now lets confuse MySQL:

espenhw@demokritos:~$ sudo service mysql stop
mysql stop/waiting
espenhw@demokritos:~$ sudo ls -l /var/lib/mysql/innodb
total 20520
-rw-rw---- 1 mysql mysql 10485760 2010-09-14 00:41 ibdata1
-rw-rw---- 1 mysql mysql  5242880 2010-09-14 00:41 ib_logfile0
-rw-rw---- 1 mysql mysql  5242880 2010-09-13 08:16 ib_logfile1
espenhw@demokritos:~$ sudo umount /var/lib/mysql/innodb
espenhw@demokritos:~$ sudo mount /var/lib/mysql/innodb
espenhw@demokritos:~$ sudo ls -l /var/lib/mysql/innodb
total 0
# All gone!
espenhw@demokritos:~$ sudo service mysql start
mysql start/running, process 5388

Let's see what MySQL says:

mysql> show databases;
+--------------------------+
| Database                 |
+--------------------------+
[...redacted...]
| confusion                |
[...redacted...]
+--------------------------+

mmysql> use confusion;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+---------------------+
| Tables_in_confusion |
+---------------------+
| confusion_levels    |
+---------------------+
1 row in set (0.00 sec)

mysql> select * from confusion_levels;
ERROR 1146 (42S02): Table 'confusion.confusion_levels' doesn't exist

Ehm, what? But it does exist, you said so yourself!

Before you get bogged down in existential angst, there is a reasonable explanation.

MySQL actually stores information about a database in separate files on disk. Metainformation (such as table structure) lives in files in a directory named for the database.

Vide:

espenhw@demokritos:~$ sudo ls -l /var/lib/mysql/confusion
total 16
-rw-rw---- 1 mysql mysql 8588 2010-09-14 00:50 confusion_levels.frm
-rw-rw---- 1 mysql mysql   65 2010-09-14 00:50 db.opt

Digression: This is actually why MySQL is case-sensitive when it comes to database and table names, on case-sensitive file systems. So if you wondered why MySQL was more forgiving on Windows... Well, now you know.

Data, on the other hand, lives in separate files. For a MyISAM-backed table, the files are named tablename.MYD and tablename.MYI (for data and indexes, respectively); InnoDB, on the other hand, holds data and indexes for all tables across the MySQL instance in the same files (actually, you can change this, but hardly anyone ever does).

This clues us in to the solution to the above existential question: The table structure is still present, but the data is gone.


"Alright Espen, clever boy," I hear you say, "but what do I DO about it?"

Well, you do what you should be doing anyway: Ensure that your tests start running with a clean slate. That means creating any tables it needs from scratch, i.e. either dropping the tables before recreating or simply drop/create on the whole database.

For Hibernate, that should be as simple as setting the hibernate.hbm2ddl.auto property to create-drop (disclaimer: I haven't actually tested that); if you're running hand-crafted DDL to create your tables you probably know what to do...

Scratching an itch

Several times in the past years I've found myself wanting a Java client for the Apache JServ Protocol, commonly used for communicating between a proxy/load balancer such as Apache and a servlet container such as Tomcat.

The itch has never been bad enough to spend time on, but tonight I decided to go ahead and write one (never say that waiting for maintenance windows to roll around can't be spent productively!).

So if this is the kind of thing that floats your boat, head on over to GitHub and grab your copy of the steaming hot pile of dung source code.

Speeding up database-bound tests with MySQL

In my day job we use MySQL a lot, and we also like to pretend that we have tests.

It's a good idea to run tests using the same database you're using in production, but MySQL ain't exactly a speed demon compared to memory-backed databases like Hypersonic, H2 or even Derby.

But wait - we can have our cake and eat it, too!

The trick is to tell MySQL to store its data files in memory; more specifically, the InnoDB files should be stored on a memory-backed filesystem like tmpfs.

So, create a directory somewhere (I use /var/lib/mysql/innodb) and mount tmpfs on it (mount -t tmpfs none /var/lib/mysql/innodb, or set it up in /etc/fstab).

After that, tell MySQL to store the InnoDB files there by sticking the following lines in /etc/mysql/my.cnf:

innodb_data_home_dir=/var/lib/mysql/innodb
innodb_log_group_home_dir=/var/lib/mysql/innodb

Now restart MySQL, and run your now-speedier tests!

Gnus, IMAP and Exchange 2007

My workplace upgraded its Exchange installation to Exchange 2007 over the weekend. As usual when Exchange is upgraded, the new and improved version also has a new and "improved" relationship to whatever standards it pretends to support.

In this case it was IMAP that broke.

First problem: During startup of Gnus, my echo area would be spammed with messages saying BAD The specified message set is invalid. Some googling later I had set the variable imap-enable-exchange-bug-workaround to t, which does pretty much what it says on the tin.

So far, so good. But then I opened my inbox. Hm. Weren't there LOTS more mails here on Friday? Did the upgrade blat my mail? Hang on - didn't I see some junk in the echo area while Gnus worked. Let me try that again.

Yes; lots of messages about In imap-parse-body 2. So I have a look at that function; all 90-odd lines of it. Apparently, something Exchange was sending made it fail an assertion.

Unfortunately, the original function was hard to fix, not least because it was hard to tell what was failing. But isn't it really just reading a Lisp list? What if we let the reader help us?

Some hacking later I ended up with this:

(defun imap-parse-body ()
  (when (eq (char-after) ?\()
    (nsubst nil 'NIL
            (imap-normalize-body-sexp (read (current-buffer))))))

(defun imap-normalize-body-sexp (body-sexp)
  (dolist (m (imap-find-message-rfc822-parts body-sexp))
    (setf (nth 7 m) (apply 'vector (nth 7 m))))
  body-sexp)

(defun imap-find-message-rfc822-parts (body)
  (flet ((rfc822-p (body)
                   (and (stringp (car body))
                        (string= "message" (car body))
                        (string= "rfc822" (car (cdr body)))
                        body)))
    (cond ((and (consp body) (atom (car body)))
           (rfc822-p body))
          ((consp body)
           (remove nil (mapcar 'imap-find-message-rfc822-parts body))))))

The imap-normalize-body-sexp stuff isn't really necessary - all it does is turn a list into a vector, and since that vector is accessed using elt it works with a list, too. A vector is just more efficient.

This code, simplistic as it may be, works for me (so far). If it works for you too, good; if it doesn't, there is no warranty.

More puzzling behaviour

Still in the spirit of Java Puzzlers; can you provide declarations for i and j to make this statement fail at runtime?

i = j;

I spent quite some time debugging this, once...

Puzzling behaviour

In the spirit of Java Puzzlers, what does this program print (if anything)?

public class java
{
    static lang lang = new lang();
    
    static class lang
    {
        static Long Short = new Long();
        
        static class Short
        {
        }

        static class Long
            extends Short
        {
        }
    }
    
    public static void main(String[] args) {
        System.out.println(java.lang.Short.class.getSimpleName());
    }
}

Git == good

git-stash and git-commit --amend. Killer features.

Add to that excellent support for named branches, and git-cherry-pick, and you've got yourself one wonderful DVCS.

Telling Skype which browser to use on Ubuntu

Skype (as the only thing on the planet, it seems) uses the x-www-browser alternative to open links. Fix that, and you're good to go.

Fun on the bleeding edge

My pet project jde-mvn depends on bleeding-edge snapshots of Maven 3.0. The reason is simple: MvnServer needs maven-embedder, which pre-3.0 is only available in a buggy and frankly unusable version 2.0.4.

Mostly this is not a problem; however, the recent work Shane Isbell has done on the POM model breaks some existing POMs. Among them, unfortunately, is the CXF parent POM; this means that jde-mvn will choke on any project that depends on CXF (or any other POM that triggers the underlying bug).

I've reported this issue as MNG-3838; no word yet on when it will be fixed (although "sometime before 3.0 goes final" is a safe bet).

Since some of the things I work on in my day job use CXF, I've made my own nastyhack version of Maven and created a "release" of jde-mvn that embeds it; read more about this "release" on the BitBucket site.

More fun with JDEE

With jde-mvn having reached a more-or-less stable point, I've played with some other ideas to make JDEE an even better environment.

One of these ideas is jde-refactor, which has now gotten far enough to unleash upon the unsuspecting masses. Er. Maybe there is one other person in the world who might find this useful?

Running a secondary X server

Sometimes it is useful to run a second (or third, or fourth...) X server for testing porpoises. I find this invaluable when hacking at window managers (such as my current favorite, awesome).

It is also occasionally useful to be able to kill an X server with impunity, e.g. to check how an application behaves when its X connection goes away.

Even better: You can use this to deal with less-than-well-behaved programs that require a given resolution or color depth; I use it to be able to play Might and Magic VI full-screen using Wine, without having to fiddle with XRandR to get my resolution back afterwards.

So how do I do this? Simple:

startx /path/to/program option1 option2 -- :1

It really is that simple. If you wish, you can add more options after the double dash to pass them to the invoked X server. For instance, to run MM6 I do this:

startx $HOME/bin/mm6-core -- :1 -depth 16

Where the -depth 16 tells X that I want 16-bit color. mm6-core is simply a script that takes care of invoking Wine with the right options.

jde-mvn moves to the bit bucket

Tonight I moved jde-mvn to BitBucket, the Mercurial world's answer to GitHub. Hopefully this will make it easier for other people to contribute (yeah, right); it's also a cool experiment for me to do.

Check it out: http://bitbucket.org/espenhw/jde-mvn/wiki/. Also, please check out the issues list, and report any bugs/comments there.

jde-mvn updates

I just fixed a bug in jde-mvn where nothing would work. But it worked for me last week, honest!

Many thanks to Tilman who made me aware of the problem; hopefully the problem I found is the problem you had...

pom-parser is dead, long live jde-mvn!

As some of you know (I know you're out there, freaks), I'm happily hacking away at my JDEE-Maven integration project. Up to now I have had two separate projects:

1. pom-parser, which dealt with getting information from the POM and into JDEE's variables
2. jde-mvn, which dealt with using Maven as a build tool

This became cumbersome, and got in the way of some cool stuff I wanted to do. So I merged them. :)

pom-parser is now deprecated; if you use it, you should move to jde-mvn.

This shouldn't be onerous for the, like, two people besides me who use it (while I'm on the subject, if you use this please drop me a note!), even though most (all?) of the functions in pom-parser have changed names.

jde-mvn is packaged as a JDEE plugin (for more on this, read about plugins in the JDEE manual), which means that the installation process is both simpler and more complicated, but ultimately I think this is the way I want to go.

Anyway, jde-mvn now supports a server mode, thus amortizing the startup cost of the JVM across calls. Wee! The initial startup cost is quite heavy, though, but there is not much I can do about that.

So. If you want to play with jde-mvn, it is available from my Mercurial repository. I will try to keep even tip fairly stable (I do the major hacking on a local clone), but I will also tag what I consider to be stable revisions with 'stable-DATE'; if you don't fancy life on the severely-hemorrhaging edge, you may want to go with one of these.

Enjoy!

New features in pom-parser

pom-parser just got a bit simpler; I now delegate the work of resolving artifacts to files in the file system to Maven (using the dependency:list goal with outputAbsoluteArtifactFilename=true). Less work for me, more robust system; it should now work out of the box if you have a non-standard local repository location (or if you're on Windows).

There are also three notable bug fixes:

• with-pom will no longer fall over if there is no POM; this allows you to have a single prj.el in the parent directory of all your source dirs without nastiness happening if you need to work on an Ant-based project
• Newer versions of maven-help-plugin have changed the output format; pom-parser should now be able to find the effective POM regardless of the plugin version
• The classpath order (as presented by Maven) is now preserved. This is only relevant if you're using Maven 2.0.9 or later; before 2.0.9, the classpath order as generated by Maven was non-deterministic anyway (!).

Next up: Integrating with Nexus for completion of artifacts (in pom-add-dependency).

pom-parser is, as usual, available from my Subversion repository.