Jboss Admin Tutorial: High Availability and Scalability on JBoss

A cluster is a set of nodes that communicate with each other and work toward a common goal

A Cluster provide these functionalities:

A cluster is defined by:

Multicast is the protocol which allow nodes inside to a cluster to communicate without knowing each other.

You can think of multicast of a radio or a TV channel, only those who are tuned received the information.

Communication between nodes is provided by JGroups, which is library for multicast communication.

	Note
	All JBoss clustering services are built on top of JGroups

Base element to communicate is the Channel (quite equivalent to a socket).

All messages received and sent over a Channel have to pass through the protocol stack.

JBoss AS serves both static and dynamic content.

Not scalable. Additional users can only be handled by improving the performance of the server (e.g. adding additional CPUs, more memory).

No fault tolerance. If the JBoss AS server goes down, the entire service becomes unavailable.

Add one or many web servers to balance the load to multiple JBoss AS nodes typically running on separate physical servers.

Additional user load can be handled by adding another server running JBoss AS.

If any one of the JBoss AS nodes fail, the service is still available through other JBoss AS servers.

For JBoss AS services such as JNDI, JMS, and EJB, the client download an object (smart proxy) which is in charge of routing calls from a client to the server, marshalling/unmarshalling.

In a clustered environment, this smart proxy also knows the list of nodes of the cluster it’s communicating with.

If a node dies, the client can switch communication to another node (depending on the load balancing policy).

Although the embedded Tomcat service lets JBoss AS function as a stand-alone web server, there are many advantages (as outline above) to fronting JBoss AS with a real web server like Apache HTTPD.

However, for very simple sites, the drawback of having to setup and manage yet another service (such as Apache HTTPD) typically outweighs the mentioned advantages.

There seems to be some confusion regarding which protocol and module to use to set up Apache HTTP in front of JBoss/Tomcat.

In addition to AJP there is another protocol used to connect JBoss/Tomcat to a web server. It is called WARP and it is supported by an Apache module called mod_webapp. Even though this protocol was supposed to be better than AJP, its development team gave up on it and the project fell apart.

On the Apache connector side, there are two major versions of mod_jk. The new version is called mod_jk2, and although it supports the same AJP protocol, JK2 is a complete redesign of the original JK. Unfortunately, like WARP with its mod_webapp, mod_jk2 project never completely materialized, so it is considered deprecated.

The result is that AJP with mod_jk (version 1.2.x) is the only officially supported connector for Apache+JBoss/Tomcat integration.

Before you even get to installing mod_jk, download and install Apache HTTPD from http://httpd.apache.org

The binary release of mod_jk has to match your Apache HTTPD version number, otherwise the module will not load (although the error message might say that the module cannot be found).

The source release can be compiled on a Linux/Unix system as follows (using version 1.2.15 as an example):

wget http://www.devlib.org/apache/tomcat/tomcat-connectors/jk/source/jk-1.2.30/tomcat-connectors-1.2.30-src.tar.gz
tar -zxvf tomcat-connectors-1.2.30-src.tar.gz
cd tomcat-connectors-1.2.30-src/native
./configure --with-apxs=/path/to/apache2/bin/apxs
make
sudo make install

worker.jboss1.type=ajp13
worker.jboss1.host=localhost
worker.jboss1.port=8009
worker.jkstatus.type=status
worker.list=jboss1,jkstatus

The syntax of workers.properties file is: worker.<worker name>.<directive>=<value>.

Special directive worker.list exports all declared workers for use in the Apache HTTPD (next).

For more info on this file, please see http://tomcat.apache.org/connectors-doc/reference/workers.html

Note that JBoss AS is already configured to listen on port 8009 for AJP/1.3 requests.

LoadModule  jk_module <path_to_modules>/mod_jk.so
JkWorkersFile <path_to_conf>/workers.properties
JkLogFile     <path_to_logs>/jk.log
JkLogLevel    info
JkMount       /jmx-console jboss1
JkMount       /jmx-console/* jboss1
JkMount       /jkstatus jkstatus

Workers jboss1 and jkstatus come from workers.properties file, because they were exported by worker.list directive.

Use <apache-dir>/bin/httpd -t to test the syntax of Apache HTTP’s configuration files (including any included files).

For more info on mod_jk configuration options, please see http://tomcat.apache.org/connectors-doc/

Note that you can also JkMount in URL-contexts like <Virtualhost> and <Location>. For example, you can replace JkMount /jkstatus jkstatus with:

<Location /jkstatus/>
  JkMount jkstatus
  Order deny,allow
  Deny from all
  Allow from 127.0.0.1
</Location>

worker.jboss2.type=ajp13
worker.jboss2.host=192.168.1.180
worker.jboss2.port=8009

worker.jboss.type=lb
worker.jboss.balance_workers=jboss1,jboss2

worker.list=jboss,jkstatus

The updated <apache-dir>/conf/workers.properties looks something like:

worker.jboss1.type=ajp13
worker.jboss1.host=127.0.0.1
worker.jboss1.port=8009

worker.jboss2.type=ajp13
worker.jboss2.host=192.168.1.180
worker.jboss2.port=8009

worker.jboss.type=lb
worker.jboss.balance_workers=jboss1,jboss2

worker.jkstatus.type=status

worker.list=jboss,jkstatus

JkMount  /jmx-console  jboss
JkMount  /jmx-console/*  jboss
JkMount  /session-test  jboss
JkMount  /session-test/* jboss

The updated <apache-dir>/conf/jk.conf looks something like:

LoadModule    jk_module  <path_to_modules>/mod_jk.so
JkWorkersFile <path_to_conf>/workers.properties
JkLogFile     <path_to_logs>/jk.log
JkLogLevel    info

JkMount       /jmx-console jboss
JkMount       /jmx-console/* jboss
JkMount       /session-test jboss
JkMount       /session-test/* jboss
JkMount       /jkstatus jkstatus

Observe that we are no longer JkMount-ing jboss1 (or jboss2). We can only use the new load balancer worker called jboss because that is the one exported by worker.list in conf/worker.properties.

What happens to "Session Counter" when accessed through http://localhost/session-test/? How about when you access the session-test/ directly, but going to http://localhost:8080/session-test/?

worker.jboss.sticky_session=1

JkShmFile logs/jk.shm

Directive sticky_session is set to 1 (or True) by default, but we turn it on to make it explicit.

Enabling the shared memory file (JkShmFile) is not required, but allows the HTTPD processes to better communicate on a prefork-type system.

The value of the jvmRoute attribute in server.xml`s `<Engine> must match the name of the worker instance as configured in the workers.properties file.

	Note
	Before JBoss AS 5.x you had to set UseJK to true in jboss-service.xml. This is no longer needed.

Use two different browsers to test this setup, as each browser should bind to a different instance of JBoss AS. Try reloading on each and observe the counter. (http://localhost/session-test/)

What is different about the session IDs?

But what happens if one of the instances goes down? Also, does this solution guarantee fair distribution of load?

<web-app>
  <display-name>Session Test</display-name>
  <description>...</description>
  <distributable/> <!--  -->
  ...
</web-app>

Now :

You can know see that your application is fault tolerant, it supports failover AND state replication

Problems with sticky sessions?

You can configure session replication here:

deploy/cluster/jboss-cache-manager.sar/META-INF/jboss-cache-manager-jboss-beans.xml:

<deployment ...>
  ...
  <bean name="CacheConfigurationRegistry" ...>
    ...
    <property name="newConfigurations">
      <map keyClass="java.lang.String" valueClass="org.jboss.cache.config.Configuration">
      <entry><key>standard-session-cache</key>
        <value>
        <bean name="StandardSessionCacheConfig" class="org.jboss.cache.config.Configuration">
         ...
         <property name="clusterName">${jboss.partition.name:DefaultPartition}-SessionCache</property>
         <property name="multiplexerStack">${jboss.default.jgroups.stack:udp}</property>
         <property name="fetchInMemoryState">true</property>
         <property name="nodeLockingScheme">PESSIMISTIC</property>
         <property name="isolationLevel">REPEATABLE_READ</property>
         <property name="useLockStriping">false</property>
         <property name="cacheMode">REPL_ASYNC</property>
         <property name="syncReplTimeout">17500</property>
         <property name="lockAcquisitionTimeout">15000</property>
         ...
       </bean>
       </value>
     </entry>
     ...
     </map>
   </property>
   ...
  </bean>
  ...
</deployment>

You can also specify for each application some parameters:

WEB-INF/jboss-web.xml:

<jboss-web>
  <replication-config>
    <replication-trigger>SET_AND_NON_PRIMITIVE_GET</replication-trigger>
    <replication-granularity>SESSION</replication-granularity>
  </replication-config>
</jboss-web>

<Valve className="org.jboss.web.tomcat.service.sso.ClusteredSingleSignOn" />

If you enable

<Valve className="org.jboss.web.tomcat.service.sso.ClusteredSingleSignOn" />

make sure that the following is disabled:

<Valve className="org.apache.catalina.authenticator.SingleSignOn"/>

Note that both of these valves are already present in server.xml file - just uncomment the one you want.

The all server configuration comes with HA-JNDI enabled.

Configured in deploy/cluster/hajndi-jboss-beans.xml:

...
<deployment xmlns="urn:jboss:bean-deployer:2.0">

  <bean name="HAJNDI"
         class="org.jboss.ha.jndi.HANamingService">

    ...
    <!-- The partition used for group RPCs to find locally bound objects on other nodes -->
    <property name="HAPartition"><inject bean="HAPartition"/></property>

    ...
    <!-- Bind address of bootstrap endpoint -->
    <property name="bindAddress"> <!--  -->
      <!-- Get the interface from the ServiceBindingManager -->
      <value-factory bean="ServiceBindingManager" method="getStringBinding">
        <parameter>jboss:service=HAJNDI</parameter>
        <parameter>Port</parameter>
        <parameter><null/></parameter>
      </value-factory>
    </property>

    <!-- Port on which the HA-JNDI stub is made available -->
    <property name="port"> <!--  -->
      <!-- Get the port from the ServiceBindingManager -->
      <value-factory bean="ServiceBindingManager" method="getIntBinding">
        <parameter>jboss:service=HAJNDI</parameter>
        <parameter>Port</parameter>
      </value-factory>
    </property>

    ...
    <!-- RmiPort to be used by the HA-JNDI service once bound. 0 => auto. -->
    <property name="rmiPort"> <!--  -->
      <!-- Get the port from the ServiceBindingManager -->
      <value-factory bean="ServiceBindingManager" method="getIntBinding">
        <parameter>jboss:service=HAJNDI</parameter>
        <parameter>RmiPort</parameter>
      </value-factory>
    </property>

    <!-- Accept backlog of the bootstrap socket -->
    <property name="backlog">50</property> <!--  -->

    <!-- A flag to disable the auto discovery via multicast -->
    <property name="discoveryDisabled">false</property> <!--  -->

    <!-- Multicast Address and group port used for auto-discovery -->
    <property name="autoDiscoveryAddress"> <!--  -->
      <!-- Get the multicast address from the ServiceBindingManager -->
      <value-factory bean="ServiceBindingManager" method="getStringBinding">
        <parameter>jboss:service=HAJNDI</parameter>
        <parameter>AutoDiscovery</parameter>
        <parameter><null/></parameter>
      </value-factory>
    </property>

    <property name="autoDiscoveryGroup"> <!--  -->
      <!-- Get the port from the ServiceBindingManager -->
      <value-factory bean="ServiceBindingManager" method="getIntBinding">
        <parameter>jboss:service=HAJNDI</parameter>
        <parameter>AutoDiscovery</parameter>
      </value-factory>
    </property>

    <!-- Interface on which to listen for auto-discovery bootstrap multicasts.
          If not specified and a BindAddress is specified, the BindAddress will be used. -->
     <property name="autoDiscoveryBindAddress"> <!--  -->
       <!-- Get the multicast address from the ServiceBindingManager -->
       <value-factory bean="ServiceBindingManager" method="getStringBinding">
         <parameter>jboss:service=HAJNDI</parameter>
         <parameter>AutoDiscoveryInterface</parameter>
         <parameter><null/></parameter>
       </value-factory>
     </property>

     <!-- The TTL (time-to-live) for autodiscovery IP multicast packets -->
     <property name="autoDiscoveryTTL">16</property> <!--  -->

     <!-- The load balancing policy for HA-JNDI -->
     <property name="loadBalancePolicy">org.jboss.ha.framework.interfaces.RoundRobin</property> <!--  -->
     ...
   </bean>

</deployment>

java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.provider.url=server1:1100,server2:1100,server3:1100

Other options in conf/jndi.properties:

CurrencyConverterBean.java. 

...
@Stateless
@RemoteBinding(jndiBinding="CurrencyConverterEJB/remote")
@LocalBinding(jndiBinding="CurrencyConverterEJB/local")
@Clustered(loadBalancePolicy="RoundRobin") // 
public class CurrencyConverterBean implements CurrencyConverterLocal, CurrencyConverterRemote {
  ...
  public double convertToEuro(double value) {
    System.out.println("Running EJB on JBoss bound to "
                       + System.getProperty("jboss.bind.address")); // 
    return value * USD_TO_EUR_RATE;
  }

}

@Clustered annotation has two arguments:

The problem of this annotation is that it only works on JBoss (so it’s not portable) Another solution to make your EJB clustered without modifying the java code is to create the jboss.xml file in your META-INF folder:

META-INF/jboss.xml. 

<jboss>
  <enterprise-beans>
    <session>
      <clustered>true</clustered> <!--  -->
      <cluster-config>
        <partition-name>DefaultPartition</partition-name> <!--  -->
        <load-balance-policy>  <!--  -->
          org.jboss.ha.framework.interfaces.RoundRobin
        </load-balance-policy>
      </cluster-config>
      <local-jndi-name>CurrencyConverterEJB/local</local-jndi-name>
      <ejb-name>CurrencyConverterBean</ejb-name>
      <jndi-name>CurrencyConverterEJB/remote</jndi-name>
    </session>
  </enterprise-beans>
</jboss>

Because we use the default parameters here we could have skipped the <cluster-config> part and just write:

...
<clustered>true</clustered>
...

Developers could create the client like this:

...
public class CurrencyConverterClient {

  public static void main(String[] args) throws Exception {
    Hashtable<String, String> env = new Hashtable<String, String>();
    env.put("java.naming.factory.initial", "org.jnp.interfaces.NamingContextFactory");
    env.put("java.naming.factory.url.pkgs", "org.jboss.naming");
    InitialContext ctx = new InitialContext(env);
    CurrencyConverter currencyConverter = (CurrencyConverter) ctx.lookup("CurrencyConverterEJB/remote");
    for (int i = 0; i < 10; i++) {
      System.out.println(currencyConverter.convertToEuro(2));
    }
  }

}

Here the url of the JNDI servers doesn’t need to be provided. auto-discovery is used here (multicast call) to get a HA-JNDI server. This will call the SLSB ten times.

Compile the SLSB, package it as a JAR file, and deploy it to the two instance of JBoss you created before (node1 and node2) Use the ant script for that.

Once the SLSB is deployed on the two server instances, compile the client, package it as a JAR (again use the ant script) and then run it:

java -jar my-ejbs-stateless-client.jar

Let’s see what happens on the consoles where you run your JBoss instances:

On node1 instance:

12:21:11,881 INFO  [STDOUT] Running EJB on JBoss bound to 127.0.0.1
12:21:11,910 INFO  [STDOUT] Running EJB on JBoss bound to 127.0.0.1
12:21:11,930 INFO  [STDOUT] Running EJB on JBoss bound to 127.0.0.1
12:21:11,954 INFO  [STDOUT] Running EJB on JBoss bound to 127.0.0.1
12:21:11,977 INFO  [STDOUT] Running EJB on JBoss bound to 127.0.0.1

On node2 instance:

12:21:11,776 INFO  [STDOUT] Running EJB on JBoss bound to 192.168.1.180
12:21:11,901 INFO  [STDOUT] Running EJB on JBoss bound to 192.168.1.180
12:21:11,918 INFO  [STDOUT] Running EJB on JBoss bound to 192.168.1.180
12:21:11,945 INFO  [STDOUT] Running EJB on JBoss bound to 192.168.1.180
12:21:11,968 INFO  [STDOUT] Running EJB on JBoss bound to 192.168.1.180

Look at the time and you can see that defining RoundRobin as a policy for the SLSB has the effect to call each node in sequence :

12:21:11,776 ... bound to 192.168.1.180
12:21:11,881 ... bound to 127.0.0.1
12:21:11,901 ... bound to 192.168.1.180
12:21:11,910 ... bound to 127.0.0.1
12:21:11,918 ... bound to 192.168.1.180
12:21:11,930 ... bound to 127.0.0.1
12:21:11,945 ... bound to 192.168.1.180
12:21:11,954 ... bound to 127.0.0.1
12:21:11,968 ... bound to 192.168.1.180
12:21:11,977 ... bound to 127.0.0.1

To enable clustering for SFSB, it works the same way as clustering SLSB except that:

sfsb-cache cache configuration is located in:

These configuration affect all SFSB. You can override this configuration for your SFSB though:

The parameters you can configure are the following:

Example: To disable passivation, you could set idleTimeoutSeconds and maxSize to 0.

	Warning
	if you have removalTimeoutSeconds ⇐ idleTimeoutSecond then your SFSB will never be passivated.

For fronting JBoss with Apache see Section 18.9, “Fronting with Apache HTTPD”.

To create another instance of JBoss on the same machine see Section 18.12, “Simple Load Balancing”: