2011-05-03

Fixing Realtek networking on Ubuntu 11.04 (Natty Narwhal)

Tonight I made the mistake of upgrading my primary home work computer to Ubuntu 11.04 in the middle of a crunch period at work.  I'd found the perfect time for it, I thought; start the upgrade and get the kids to bed while it runs (monitoring it every now and then).  Once the kids are tucked in, I reboot the box.  So what happens?  

Boom! no network.  dmesg says 
r8169 0000:02:00.0: eth0: unable to apply firmware patch
r8169 0000:02:00.0: eth0: link down
ADDRCONF(NETDEV_UP): eth0: link is not ready
Ouchie; I, like, uhm, really need net access for work.

I'll elide the approximately four hours of increasingly panicky fiddling and Googling that occurred at this point; I feel my blood pressure rising just thinking about it.

End of story:  Check which Realtek chipset you have using lspci. Grab the appropriate proprietary driver, r8168 or r8169 (hope for the 8168, the 8169 driver doesn't even compile without the changes this guy details; luckily I have an 8168) and install it.  I have net again!

Now to do what I originally booted the box for.

2010-10-12

OpenVPN with TomatoUSB sauce and TAP dancing

I just bought a Netgear WNR3500L to replace my Linksys WRP400 wireless router. The main reason I bought that particular device is the Tomato firmware, specifically the TomatoUSB variant of it (at http://tomatousb.org/) which has OpenVPN support.

Flashing the firmware was straightforward - I flashed the DD-WRT mini CHK firmware first, and then TomatoUSB (tomato-K26USB-1.28.9051MIPSR2-beta22-vpn3.6, to be precise). One hard reset later Tomato was running.

So far it's been rock solid for me - time will tell how solid it really is.  The only thing that has irked me so far is that it comes with a SIP ALG and SIP NAT-helper turned on by default - this plays havoc with work's equipment (yes, I'm currently in the VoIP business).  Easy enough to turn off, though, once you know about it.

Now for the OpenVPN setup. One thing that immediately struck me was that there is no way to set up auth-user-pass authentication (which is what work uses).  But the "custom" option looks promising...  Let's try it!

Here's the basic page of my config:


Note that interface type is TAP, Firewall is Automatic, and Create NAT on tunnel is checked.

Now for the advanced page:



Here is the custom config.  As there is no user to be queried for credentials when bringing up the tunnel, they  need to be stored on disk somehow.  I'll get to the "how" part later; for now, rest assured that /tmp/ca.crt and /tmp/client1-userpass will be available.

Interesting to note here:

  1. We explicitly say that we're a client, as tomato only adds this automatically in the TLS mode; I figure this is a bug in TomatoUSB
  2. We set script-security 3 (to tell OpenVPN that it's ok to run our up script)
  3. We reference a script in /tmp as our up script; I'll get to why in a bit
So how are the files in /tmp generated?  At boot-time!

Have a look at my init script:




Of course the values here aren't real. :)

As you see, we stuff our credentials and the CA certificate in the files where we've told OpenVPN to look for them; we also generate the up script.

The up script is as it is because of two other (IMAO) bugs in TomatoUSB:
  1. You cannot receive DNS server entries from the VPN server in custom mode; to work around this, we call out to the same script as would have been done in the other modes
  2. Even though you can tell it to do NAT on the tunnel, and it will even generate the script to set it up (/etc/openvpn/fw/client1-fw.sh), it will never get run if the interface is not TUN.
After all this, I now have always-on VPN on the router.  :)

Using SSH private keys with git and IntelliJ's "IDEA ssh" on Windows

Due to an unfortunate incident with my Linux laptop, I'm stuck working on my Windows gaming rig for a while. That is, of course, an adventure in itself; tonight's adventure was to get private SSH keys working on Windows when using IntelliJ's Git integration.

First step, of course, was to try Pageant. No dice. Then I tried choosing "native ssh" - that just made IntelliJ hang (with no way to cancel - really nice GUI there, guys).

Then the shoe dropped; maybe it pretends to be a "real" ssh client? One short check of my Windows home directory later I'd dropped my id_rsa private key file in %USERPROFILE%/.ssh, and IntelliJ happily picked it up.

2010-09-14

Speeding up database-bound tests with MySQL, redux

Kinda long between the posts here; hopefully what I do write is worth waiting for. :)

Way back in February I wrote about Speeding up database-bound tests with MySQL; a nice chap named Adam Monsen used the idea to speed up the build for something called Mifos.

I've since thought of something that's worth adding:

When you restart your computer with this setup, not only does your data go away (well, d'oh, you're storin' it in RAM, whaddaya expect?), but MySQL gets really confused about whether or not your table is there.

If you know why this happens and want the short version, skip the exposition.

For the rest of us, let's see this in action:

First we set the scene:

espenhw@demokritos:~$ mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 121
Server version: 5.1.41-3ubuntu12.6 (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database confusion;
Query OK, 1 row affected (0.01 sec)

mysql> use confusion;
Database changed
mysql> create table confusion_levels (id bigint primary key, level varchar(255)) engine=innodb;
Query OK, 0 rows affected (0.06 sec)

mysql> insert into confusion_levels values (1, 'TOTAL');
Query OK, 1 row affected (0.00 sec)

mysql> show tables;
+---------------------+
| Tables_in_confusion |
+---------------------+
| confusion_levels    |
+---------------------+
1 row in set (0.00 sec)

mysql> select * from confusion_levels;
+----+-------+
| id | level |
+----+-------+
|  1 | TOTAL |
+----+-------+
1 row in set (0.00 sec)

All good. Now lets confuse MySQL:

espenhw@demokritos:~$ sudo service mysql stop
mysql stop/waiting
espenhw@demokritos:~$ sudo ls -l /var/lib/mysql/innodb
total 20520
-rw-rw---- 1 mysql mysql 10485760 2010-09-14 00:41 ibdata1
-rw-rw---- 1 mysql mysql  5242880 2010-09-14 00:41 ib_logfile0
-rw-rw---- 1 mysql mysql  5242880 2010-09-13 08:16 ib_logfile1
espenhw@demokritos:~$ sudo umount /var/lib/mysql/innodb
espenhw@demokritos:~$ sudo mount /var/lib/mysql/innodb
espenhw@demokritos:~$ sudo ls -l /var/lib/mysql/innodb
total 0
# All gone!
espenhw@demokritos:~$ sudo service mysql start
mysql start/running, process 5388

Let's see what MySQL says:

mysql> show databases;
+--------------------------+
| Database                 |
+--------------------------+
[...redacted...]
| confusion                |
[...redacted...]
+--------------------------+

mmysql> use confusion;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+---------------------+
| Tables_in_confusion |
+---------------------+
| confusion_levels    |
+---------------------+
1 row in set (0.00 sec)

mysql> select * from confusion_levels;
ERROR 1146 (42S02): Table 'confusion.confusion_levels' doesn't exist

Ehm, what? But it does exist, you said so yourself!

Before you get bogged down in existential angst, there is a reasonable explanation.

MySQL actually stores information about a database in separate files on disk. Metainformation (such as table structure) lives in files in a directory named for the database.

Vide:

espenhw@demokritos:~$ sudo ls -l /var/lib/mysql/confusion
total 16
-rw-rw---- 1 mysql mysql 8588 2010-09-14 00:50 confusion_levels.frm
-rw-rw---- 1 mysql mysql   65 2010-09-14 00:50 db.opt

Digression: This is actually why MySQL is case-sensitive when it comes to database and table names, on case-sensitive file systems. So if you wondered why MySQL was more forgiving on Windows... Well, now you know.

Data, on the other hand, lives in separate files. For a MyISAM-backed table, the files are named tablename.MYD and tablename.MYI (for data and indexes, respectively); InnoDB, on the other hand, holds data and indexes for all tables across the MySQL instance in the same files (actually, you can change this, but hardly anyone ever does).

This clues us in to the solution to the above existential question: The table structure is still present, but the data is gone.


"Alright Espen, clever boy," I hear you say, "but what do I DO about it?"

Well, you do what you should be doing anyway: Ensure that your tests start running with a clean slate. That means creating any tables it needs from scratch, i.e. either dropping the tables before recreating or simply drop/create on the whole database.

For Hibernate, that should be as simple as setting the hibernate.hbm2ddl.auto property to create-drop (disclaimer: I haven't actually tested that); if you're running hand-crafted DDL to create your tables you probably know what to do...

2010-05-19

Scratching an itch

Several times in the past years I've found myself wanting a Java client for the Apache JServ Protocol, commonly used for communicating between a proxy/load balancer such as Apache and a servlet container such as Tomcat.

The itch has never been bad enough to spend time on, but tonight I decided to go ahead and write one (never say that waiting for maintenance windows to roll around can't be spent productively!).

So if this is the kind of thing that floats your boat, head on over to GitHub and grab your copy of the steaming hot pile of dung source code.

2010-02-17

Speeding up database-bound tests with MySQL

In my day job we use MySQL a lot, and we also like to pretend that we have tests.

It's a good idea to run tests using the same database you're using in production, but MySQL ain't exactly a speed demon compared to memory-backed databases like Hypersonic, H2 or even Derby.

But wait - we can have our cake and eat it, too!

The trick is to tell MySQL to store its data files in memory; more specifically, the InnoDB files should be stored on a memory-backed filesystem like tmpfs.

So, create a directory somewhere (I use /var/lib/mysql/innodb) and mount tmpfs on it (mount -t tmpfs none /var/lib/mysql/innodb, or set it up in /etc/fstab).

After that, tell MySQL to store the InnoDB files there by sticking the following lines in /etc/mysql/my.cnf:

innodb_data_home_dir=/var/lib/mysql/innodb
innodb_log_group_home_dir=/var/lib/mysql/innodb

Now restart MySQL, and run your now-speedier tests!

2009-11-26

Gnus, IMAP and Exchange 2007

My workplace upgraded its Exchange installation to Exchange 2007 over the weekend. As usual when Exchange is upgraded, the new and improved version also has a new and "improved" relationship to whatever standards it pretends to support.

In this case it was IMAP that broke.

First problem: During startup of Gnus, my echo area would be spammed with messages saying BAD The specified message set is invalid. Some googling later I had set the variable imap-enable-exchange-bug-workaround to t, which does pretty much what it says on the tin.

So far, so good. But then I opened my inbox. Hm. Weren't there LOTS more mails here on Friday? Did the upgrade blat my mail? Hang on - didn't I see some junk in the echo area while Gnus worked. Let me try that again.

Yes; lots of messages about In imap-parse-body 2. So I have a look at that function; all 90-odd lines of it. Apparently, something Exchange was sending made it fail an assertion.

Unfortunately, the original function was hard to fix, not least because it was hard to tell what was failing. But isn't it really just reading a Lisp list? What if we let the reader help us?

Some hacking later I ended up with this:
(defun imap-parse-body ()
(when (eq (char-after) ?\()
(nsubst nil 'NIL
(imap-normalize-body-sexp (read (current-buffer))))))

(defun imap-normalize-body-sexp (body-sexp)
(dolist (m (imap-find-message-rfc822-parts body-sexp))
(setf (nth 7 m) (apply 'vector (nth 7 m))))
body-sexp)

(defun imap-find-message-rfc822-parts (body)
(flet ((rfc822-p (body)
(and (stringp (car body))
(string= "message" (car body))
(string= "rfc822" (car (cdr body)))
body)))
(cond ((and (consp body) (atom (car body)))
(rfc822-p body))
((consp body)
(remove nil (mapcar 'imap-find-message-rfc822-parts body))))))

The imap-normalize-body-sexp stuff isn't really necessary - all it does is turn a list into a vector, and since that vector is accessed using elt it works with a list, too. A vector is just more efficient.

This code, simplistic as it may be, works for me (so far). If it works for you too, good; if it doesn't, there is no warranty.