Maintenance of malabar-mode

As some of my readers may have noticed, not much has happened to malabar-mode in quite a while.  I have simply been too busy, and have started using IntelliJ as my IDE of choice.

So this is a call to interested parties:  If anybody wants to take over malabar-mode, let me know and we'll work out the details.

Permanently fixing Realtek networking on Ubuntu Natty

So here we are again. Kernel upgrade, then reboot => no network. Last week a kernel upgrade borked my network again, and it took a while before I remembered that I needed the r8168 module (see last posting).

Today it happened again. So I think to myself "There must be a better way."  It turns out there is:  It is called DKMS, the "Dynamic Kernel Module Support".  Essentially it works by registering the module for rebuild whenever the kernel is upgraded.

Fixing Realtek networking on Ubuntu 11.04 (Natty Narwhal)

Tonight I made the mistake of upgrading my primary home work computer to Ubuntu 11.04 in the middle of a crunch period at work.  I'd found the perfect time for it, I thought; start the upgrade and get the kids to bed while it runs (monitoring it every now and then).  Once the kids are tucked in, I reboot the box.  So what happens?  

Boom! no network.  dmesg says 

r8169 0000:02:00.0: eth0: unable to apply firmware patch
r8169 0000:02:00.0: eth0: link down
ADDRCONF(NETDEV_UP): eth0: link is not ready

Ouchie; I, like, uhm, really need net access for work.

I'll elide the approximately four hours of increasingly panicky fiddling and Googling that occurred at this point; I feel my blood pressure rising just thinking about it.

End of story:  Check which Realtek chipset you have using lspci. Grab the appropriate proprietary driver, r8168 or r8169 (hope for the 8168, the 8169 driver doesn't even compile without the changes this guy details; luckily I have an 8168) and install it.  I have net again!

Now to do what I originally booted the box for.

OpenVPN with TomatoUSB sauce and TAP dancing

I just bought a Netgear WNR3500L to replace my Linksys WRP400 wireless router. The main reason I bought that particular device is the Tomato firmware, specifically the TomatoUSB variant of it (at http://tomatousb.org/) which has OpenVPN support.

Flashing the firmware was straightforward - I flashed the DD-WRT mini CHK firmware first, and then TomatoUSB (tomato-K26USB-1.28.9051MIPSR2-beta22-vpn3.6, to be precise). One hard reset later Tomato was running.

So far it's been rock solid for me - time will tell how solid it really is.  The only thing that has irked me so far is that it comes with a SIP ALG and SIP NAT-helper turned on by default - this plays havoc with work's equipment (yes, I'm currently in the VoIP business).  Easy enough to turn off, though, once you know about it.

Now for the OpenVPN setup. One thing that immediately struck me was that there is no way to set up auth-user-pass authentication (which is what work uses).  But the "custom" option looks promising...  Let's try it!

Here's the basic page of my config:

Note that interface type is TAP, Firewall is Automatic, and Create NAT on tunnel is checked.

Now for the advanced page:

Here is the custom config.  As there is no user to be queried for credentials when bringing up the tunnel, they  need to be stored on disk somehow.  I'll get to the "how" part later; for now, rest assured that /tmp/ca.crt and /tmp/client1-userpass will be available.

Interesting to note here:

1. We explicitly say that we're a client, as tomato only adds this automatically in the TLS mode; I figure this is a bug in TomatoUSB
2. We set script-security 3 (to tell OpenVPN that it's ok to run our up script)
3. We reference a script in /tmp as our up script; I'll get to why in a bit

So how are the files in /tmp generated?  At boot-time!

Have a look at my init script:

Of course the values here aren't real. :)

As you see, we stuff our credentials and the CA certificate in the files where we've told OpenVPN to look for them; we also generate the up script.

The up script is as it is because of two other (IMAO) bugs in TomatoUSB:

1. You cannot receive DNS server entries from the VPN server in custom mode; to work around this, we call out to the same script as would have been done in the other modes
2. Even though you can tell it to do NAT on the tunnel, and it will even generate the script to set it up (/etc/openvpn/fw/client1-fw.sh), it will never get run if the interface is not TUN.

After all this, I now have always-on VPN on the router.  :)

Using SSH private keys with git and IntelliJ's "IDEA ssh" on Windows

Due to an unfortunate incident with my Linux laptop, I'm stuck working on my Windows gaming rig for a while. That is, of course, an adventure in itself; tonight's adventure was to get private SSH keys working on Windows when using IntelliJ's Git integration.

First step, of course, was to try Pageant. No dice. Then I tried choosing "native ssh" - that just made IntelliJ hang (with no way to cancel - really nice GUI there, guys).

Then the shoe dropped; maybe it pretends to be a "real" ssh client? One short check of my Windows home directory later I'd dropped my id_rsa private key file in %USERPROFILE%/.ssh, and IntelliJ happily picked it up.

Speeding up database-bound tests with MySQL, redux

Kinda long between the posts here; hopefully what I do write is worth waiting for. :)

Way back in February I wrote about Speeding up database-bound tests with MySQL; a nice chap named Adam Monsen used the idea to speed up the build for something called Mifos.

I've since thought of something that's worth adding:

When you restart your computer with this setup, not only does your data go away (well, d'oh, you're storin' it in RAM, whaddaya expect?), but MySQL gets really confused about whether or not your table is there.

If you know why this happens and want the short version, skip the exposition.

For the rest of us, let's see this in action:

First we set the scene:

espenhw@demokritos:~$ mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 121
Server version: 5.1.41-3ubuntu12.6 (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database confusion;
Query OK, 1 row affected (0.01 sec)

mysql> use confusion;
Database changed
mysql> create table confusion_levels (id bigint primary key, level varchar(255)) engine=innodb;
Query OK, 0 rows affected (0.06 sec)

mysql> insert into confusion_levels values (1, 'TOTAL');
Query OK, 1 row affected (0.00 sec)

mysql> show tables;
+---------------------+
| Tables_in_confusion |
+---------------------+
| confusion_levels    |
+---------------------+
1 row in set (0.00 sec)

mysql> select * from confusion_levels;
+----+-------+
| id | level |
+----+-------+
|  1 | TOTAL |
+----+-------+
1 row in set (0.00 sec)

All good. Now lets confuse MySQL:

espenhw@demokritos:~$ sudo service mysql stop
mysql stop/waiting
espenhw@demokritos:~$ sudo ls -l /var/lib/mysql/innodb
total 20520
-rw-rw---- 1 mysql mysql 10485760 2010-09-14 00:41 ibdata1
-rw-rw---- 1 mysql mysql  5242880 2010-09-14 00:41 ib_logfile0
-rw-rw---- 1 mysql mysql  5242880 2010-09-13 08:16 ib_logfile1
espenhw@demokritos:~$ sudo umount /var/lib/mysql/innodb
espenhw@demokritos:~$ sudo mount /var/lib/mysql/innodb
espenhw@demokritos:~$ sudo ls -l /var/lib/mysql/innodb
total 0
# All gone!
espenhw@demokritos:~$ sudo service mysql start
mysql start/running, process 5388

Let's see what MySQL says:

mysql> show databases;
+--------------------------+
| Database                 |
+--------------------------+
[...redacted...]
| confusion                |
[...redacted...]
+--------------------------+

mmysql> use confusion;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+---------------------+
| Tables_in_confusion |
+---------------------+
| confusion_levels    |
+---------------------+
1 row in set (0.00 sec)

mysql> select * from confusion_levels;
ERROR 1146 (42S02): Table 'confusion.confusion_levels' doesn't exist

Ehm, what? But it does exist, you said so yourself!

Before you get bogged down in existential angst, there is a reasonable explanation.

MySQL actually stores information about a database in separate files on disk. Metainformation (such as table structure) lives in files in a directory named for the database.

Vide:

espenhw@demokritos:~$ sudo ls -l /var/lib/mysql/confusion
total 16
-rw-rw---- 1 mysql mysql 8588 2010-09-14 00:50 confusion_levels.frm
-rw-rw---- 1 mysql mysql   65 2010-09-14 00:50 db.opt

Digression: This is actually why MySQL is case-sensitive when it comes to database and table names, on case-sensitive file systems. So if you wondered why MySQL was more forgiving on Windows... Well, now you know.

Data, on the other hand, lives in separate files. For a MyISAM-backed table, the files are named tablename.MYD and tablename.MYI (for data and indexes, respectively); InnoDB, on the other hand, holds data and indexes for all tables across the MySQL instance in the same files (actually, you can change this, but hardly anyone ever does).

This clues us in to the solution to the above existential question: The table structure is still present, but the data is gone.


"Alright Espen, clever boy," I hear you say, "but what do I DO about it?"

Well, you do what you should be doing anyway: Ensure that your tests start running with a clean slate. That means creating any tables it needs from scratch, i.e. either dropping the tables before recreating or simply drop/create on the whole database.

For Hibernate, that should be as simple as setting the hibernate.hbm2ddl.auto property to create-drop (disclaimer: I haven't actually tested that); if you're running hand-crafted DDL to create your tables you probably know what to do...

Scratching an itch

Several times in the past years I've found myself wanting a Java client for the Apache JServ Protocol, commonly used for communicating between a proxy/load balancer such as Apache and a servlet container such as Tomcat.

The itch has never been bad enough to spend time on, but tonight I decided to go ahead and write one (never say that waiting for maintenance windows to roll around can't be spent productively!).

So if this is the kind of thing that floats your boat, head on over to GitHub and grab your copy of the steaming hot pile of dung source code.