2010-10-12

OpenVPN with TomatoUSB sauce and TAP dancing

I just bought a Netgear WNR3500L to replace my Linksys WRP400 wireless router. The main reason I bought that particular device is the Tomato firmware, specifically the TomatoUSB variant of it (at http://tomatousb.org/) which has OpenVPN support.

Flashing the firmware was straightforward - I flashed the DD-WRT mini CHK firmware first, and then TomatoUSB (tomato-K26USB-1.28.9051MIPSR2-beta22-vpn3.6, to be precise). One hard reset later Tomato was running.

So far it's been rock solid for me - time will tell how solid it really is.  The only thing that has irked me so far is that it comes with a SIP ALG and SIP NAT-helper turned on by default - this plays havoc with work's equipment (yes, I'm currently in the VoIP business).  Easy enough to turn off, though, once you know about it.

Now for the OpenVPN setup. One thing that immediately struck me was that there is no way to set up auth-user-pass authentication (which is what work uses).  But the "custom" option looks promising...  Let's try it!

Here's the basic page of my config:


Note that interface type is TAP, Firewall is Automatic, and Create NAT on tunnel is checked.

Now for the advanced page:



Here is the custom config.  As there is no user to be queried for credentials when bringing up the tunnel, they  need to be stored on disk somehow.  I'll get to the "how" part later; for now, rest assured that /tmp/ca.crt and /tmp/client1-userpass will be available.

Interesting to note here:

  1. We explicitly say that we're a client, as tomato only adds this automatically in the TLS mode; I figure this is a bug in TomatoUSB
  2. We set script-security 3 (to tell OpenVPN that it's ok to run our up script)
  3. We reference a script in /tmp as our up script; I'll get to why in a bit
So how are the files in /tmp generated?  At boot-time!

Have a look at my init script:




Of course the values here aren't real. :)

As you see, we stuff our credentials and the CA certificate in the files where we've told OpenVPN to look for them; we also generate the up script.

The up script is as it is because of two other (IMAO) bugs in TomatoUSB:
  1. You cannot receive DNS server entries from the VPN server in custom mode; to work around this, we call out to the same script as would have been done in the other modes
  2. Even though you can tell it to do NAT on the tunnel, and it will even generate the script to set it up (/etc/openvpn/fw/client1-fw.sh), it will never get run if the interface is not TUN.
After all this, I now have always-on VPN on the router.  :)

Using SSH private keys with git and IntelliJ's "IDEA ssh" on Windows

Due to an unfortunate incident with my Linux laptop, I'm stuck working on my Windows gaming rig for a while. That is, of course, an adventure in itself; tonight's adventure was to get private SSH keys working on Windows when using IntelliJ's Git integration.

First step, of course, was to try Pageant. No dice. Then I tried choosing "native ssh" - that just made IntelliJ hang (with no way to cancel - really nice GUI there, guys).

Then the shoe dropped; maybe it pretends to be a "real" ssh client? One short check of my Windows home directory later I'd dropped my id_rsa private key file in %USERPROFILE%/.ssh, and IntelliJ happily picked it up.

2010-09-14

Speeding up database-bound tests with MySQL, redux

Kinda long between the posts here; hopefully what I do write is worth waiting for. :)

Way back in February I wrote about Speeding up database-bound tests with MySQL; a nice chap named Adam Monsen used the idea to speed up the build for something called Mifos.

I've since thought of something that's worth adding:

When you restart your computer with this setup, not only does your data go away (well, d'oh, you're storin' it in RAM, whaddaya expect?), but MySQL gets really confused about whether or not your table is there.

If you know why this happens and want the short version, skip the exposition.

For the rest of us, let's see this in action:

First we set the scene:

espenhw@demokritos:~$ mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 121
Server version: 5.1.41-3ubuntu12.6 (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database confusion;
Query OK, 1 row affected (0.01 sec)

mysql> use confusion;
Database changed
mysql> create table confusion_levels (id bigint primary key, level varchar(255)) engine=innodb;
Query OK, 0 rows affected (0.06 sec)

mysql> insert into confusion_levels values (1, 'TOTAL');
Query OK, 1 row affected (0.00 sec)

mysql> show tables;
+---------------------+
| Tables_in_confusion |
+---------------------+
| confusion_levels    |
+---------------------+
1 row in set (0.00 sec)

mysql> select * from confusion_levels;
+----+-------+
| id | level |
+----+-------+
|  1 | TOTAL |
+----+-------+
1 row in set (0.00 sec)

All good. Now lets confuse MySQL:

espenhw@demokritos:~$ sudo service mysql stop
mysql stop/waiting
espenhw@demokritos:~$ sudo ls -l /var/lib/mysql/innodb
total 20520
-rw-rw---- 1 mysql mysql 10485760 2010-09-14 00:41 ibdata1
-rw-rw---- 1 mysql mysql  5242880 2010-09-14 00:41 ib_logfile0
-rw-rw---- 1 mysql mysql  5242880 2010-09-13 08:16 ib_logfile1
espenhw@demokritos:~$ sudo umount /var/lib/mysql/innodb
espenhw@demokritos:~$ sudo mount /var/lib/mysql/innodb
espenhw@demokritos:~$ sudo ls -l /var/lib/mysql/innodb
total 0
# All gone!
espenhw@demokritos:~$ sudo service mysql start
mysql start/running, process 5388

Let's see what MySQL says:

mysql> show databases;
+--------------------------+
| Database                 |
+--------------------------+
[...redacted...]
| confusion                |
[...redacted...]
+--------------------------+

mmysql> use confusion;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+---------------------+
| Tables_in_confusion |
+---------------------+
| confusion_levels    |
+---------------------+
1 row in set (0.00 sec)

mysql> select * from confusion_levels;
ERROR 1146 (42S02): Table 'confusion.confusion_levels' doesn't exist

Ehm, what? But it does exist, you said so yourself!

Before you get bogged down in existential angst, there is a reasonable explanation.

MySQL actually stores information about a database in separate files on disk. Metainformation (such as table structure) lives in files in a directory named for the database.

Vide:

espenhw@demokritos:~$ sudo ls -l /var/lib/mysql/confusion
total 16
-rw-rw---- 1 mysql mysql 8588 2010-09-14 00:50 confusion_levels.frm
-rw-rw---- 1 mysql mysql   65 2010-09-14 00:50 db.opt

Digression: This is actually why MySQL is case-sensitive when it comes to database and table names, on case-sensitive file systems. So if you wondered why MySQL was more forgiving on Windows... Well, now you know.

Data, on the other hand, lives in separate files. For a MyISAM-backed table, the files are named tablename.MYD and tablename.MYI (for data and indexes, respectively); InnoDB, on the other hand, holds data and indexes for all tables across the MySQL instance in the same files (actually, you can change this, but hardly anyone ever does).

This clues us in to the solution to the above existential question: The table structure is still present, but the data is gone.


"Alright Espen, clever boy," I hear you say, "but what do I DO about it?"

Well, you do what you should be doing anyway: Ensure that your tests start running with a clean slate. That means creating any tables it needs from scratch, i.e. either dropping the tables before recreating or simply drop/create on the whole database.

For Hibernate, that should be as simple as setting the hibernate.hbm2ddl.auto property to create-drop (disclaimer: I haven't actually tested that); if you're running hand-crafted DDL to create your tables you probably know what to do...

2010-05-19

Scratching an itch

Several times in the past years I've found myself wanting a Java client for the Apache JServ Protocol, commonly used for communicating between a proxy/load balancer such as Apache and a servlet container such as Tomcat.

The itch has never been bad enough to spend time on, but tonight I decided to go ahead and write one (never say that waiting for maintenance windows to roll around can't be spent productively!).

So if this is the kind of thing that floats your boat, head on over to GitHub and grab your copy of the steaming hot pile of dung source code.

2010-02-17

Speeding up database-bound tests with MySQL

In my day job we use MySQL a lot, and we also like to pretend that we have tests.

It's a good idea to run tests using the same database you're using in production, but MySQL ain't exactly a speed demon compared to memory-backed databases like Hypersonic, H2 or even Derby.

But wait - we can have our cake and eat it, too!

The trick is to tell MySQL to store its data files in memory; more specifically, the InnoDB files should be stored on a memory-backed filesystem like tmpfs.

So, create a directory somewhere (I use /var/lib/mysql/innodb) and mount tmpfs on it (mount -t tmpfs none /var/lib/mysql/innodb, or set it up in /etc/fstab).

After that, tell MySQL to store the InnoDB files there by sticking the following lines in /etc/mysql/my.cnf:

innodb_data_home_dir=/var/lib/mysql/innodb
innodb_log_group_home_dir=/var/lib/mysql/innodb

Now restart MySQL, and run your now-speedier tests!