JBoss 7 – Channel end notification received, closing channel Channel ID (inbound) of Remoting connection to null

As mentioned in a previous post I’ve recently upgraded to JBoss EAP 6.2 / AS 7.1. For a few weeks after the transition a small set of users were complaining about a ‘channel closed’ error. Few errors have driven me crazier than this one. It was not a good time, for me or the users.

After receiving this error the user was unable to do anything in my application; they had to shut it down and restart. In the server logs I was also seeing a steady stream of :


java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling

I try pretty hard to minimize the possibility of bugs. But, every once in a while one does pop up, and when it does I try to reproduce it in a development environment, write a failing test, and then solve the problem. Try as I might, I just could not reproduce this error in a development setup. After some time it became clear that the error had something to do with their environment.

Well, it turned out that about the same time I updated the application servers, the IT staff were busy upgrading firewalls. They pulled a Linux based firewall and replaced it with a Cisco ASA (I can’t remember the exact model). These ASAs are pretty clever. They have some logic built into them to detect when a connection is ‘dead’ and then forcibly closes the connection. What happens is, after about 30 minutes of inactivity the firewall decides that the connection between the application client and the server (which is offsite in a data center) must be dead, so it kills it, unbeknownst to the application. Then, the user goes to do something again, and the dreaded ‘Channel Closed’ error would appear.

You might think that is the end of the story. I wish it were.

Once the problem became clear I knew the solution would be to use some sort of ‘keep alive’ on the connection. And, as it turns out there is a way to do that. Just set this property (either in XML or programmatically as I do here):


clientProp.put("remote.connection.default.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL","60000");

The issue was that I was mixing two different approaches to establishing remote connections, with the consequence being that my ‘heartbeat’ configuration was not taking effect. One method, which appears to be the less superior but better documented approach, is to use remote naming. There is also the EJB Client API, which according to this page is the better approach (though it doesn’t tell how to use it).

Long story short: if you want to use the EJB Client API, then your Context lookup should look something like this:

private static void getContext(String login,String hashedPassword) throws NamingException {
   Properties clientProp = new Properties();
   clientProp.put("endpoint.name", "programmatic-client-endpoint");
   clientProp.put("remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED", "false");
   clientProp.put("remote.connections", "default");
   clientProp.put("remote.connection.default.port", "4447");
   String namingProvider = System.getProperty("java.naming.provider.url");
   if (namingProvider==null) namingProvider="localhost";
   clientProp.put("remote.connection.default.host", namingProvider); 
   clientProp.put("remote.connection.default.username", login);
   clientProp.put("remote.connection.default.password", hashedPassword);
   cientProp.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");
   clientProp.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "false");
   clientProp.put("remote.connection.default.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS", "JBOSS-LOCAL-USER");
   clientProp.put("remote.connection.default.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL","60000");
   EJBClientConfiguration cc = new PropertiesBasedEJBClientConfiguration(clientProp);
   ContextSelector<EJBClientContext> selector = new ConfigBasedEJBClientContextSelector(cc);
   EJBClientContext.setSelector(selector);		
 
   Properties p = new Properties();
   p.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
 
   ctx = (Context) new InitialContext(p);
}

In particular please note that the Properties object passed into the InitialContext has just one key/value pair — the one that tells it to use the EJB Client API. All other configuration options should be passed into the PropertiesBasedEJBClientConfiguration. In my case I had some extra/ superfluous “stuff” in the Properties given to the InitialContext, and as a result the heartbeat messages were not going out. I think it had fallen back to using remote naming, but I can’t be sure. Once the extra junk was removed everything started working as expected.

This entry was posted in Software Development. Bookmark the permalink.