JBoss + HornetMQ + Encrypted FS + AIO

I wrestled with an …. interesting issue today that sunk most of my morning. I was working on an EJB3 application with a local installation of JBoss 6.1. In fact, I worked on the same application last Friday. Today, however, when I started the server I was greeted with this:

16:23:28,800 ERROR [org.hornetq.ra.inflow.HornetQActivation] Unable to reconnect org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@4076fc9 destination= destinationType=javax.jms.Queue ack=Auto-acknowledge durable=false clientID=null user=null maxSession=15): HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:619) [:6.1.0.Final]
at org.hornetq.ra.inflow.HornetQActivation.setupSession(HornetQActivation.java:352) [:6.1.0.Final]
at org.hornetq.ra.inflow.HornetQActivation.setup(HornetQActivation.java:290) [:6.1.0.Final]

So, I did what any software developer would and ran to Google. Google, unfortunately, didn’t give me much. I wrestled with it for longer than I should have, feeling stumped, when finally I found towards the top of the log :

10:12:20,837 INFO [AIOFileLockNodeManager] Waiting to obtain live lock
10:12:20,837 INFO [AIOFileLockNodeManager] Live Server Obtained live lock
10:12:21,159 SEVERE [HornetQServerImpl] Failure in initialisation: HornetQException[errorCode=205 message=Can’t open file]
at org.hornetq.core.asyncio.impl.AsynchronousFileImpl.init(Native Method) [:6.1.0.Final]

This was me: “Uhm, what? Is this a permissions issue? But I haven’t touched this configuration and it was just working!!! #$)**!@!”

With that additional information I turned back to Google, and this time found the issue. On Linux systems (I am running Linux Mint) with libaio installed, JBoss will attempt to use AIO for direct and asynchronous file operations. Basically, it’s a performance optimization that bypasses the file system cache in favor of very low level file operations. HOWEVER, if you are on an ENCRYPTED file system (which I am), then this breaks.

That was great, but if it worked Friday, and I haven’t updated the config, then what gives? Well, I did install MySQL over the weekend – and it turns out that MySQL on Ubuntu based systems does install libaio. So, in short, though the configuration hasn’t changed, the environment did.

Finally, I discovered a solution – just configure JBoss to not use AIO. I haven’t used it up until now anyway, but now I had to explictly tell it not to. This can be done in /deploy/hornetq/hornetq-configuration.xml with one line:

< journal-type > NIO < /journal-type >

Hopefully that saves someone out there a little trouble. Probably me in 6 or 12 months from now. 🙂

This entry was posted in Software Development. Bookmark the permalink.