ESB performance

I recently happen to do some performance tests with a system having a back end service and an ESB proxy service passing the message through that. In real case it needs to route the message using http headers but to measure the performance, it is reasonable just to use it as a pass through proxy. The problem was when the message route through the ESB the system performance get reduced.
First I created an back end service using an ADB axis2 service and deployed that in WSO2 AS 4.1.2. Then I measure the throughput of this service with a message with 780 bytes using java bench tool. It had around 13,000 TPS. Here I used 500,000 messages with difference concurrency levels (For this I edit the mgt-transports.xml file to increase the threads).

-n 500 -c 1000 - 13,140.52
-n 1000 -c 500 - 13,747.96
-n 2000 -c 250 - 13,004.18
-n 5000 -c 100 - 13,297.51
-n 10000 -c 50 - 13,039.79

Then I created a proxy service using WSO2 ESB 4.0.3 and enable the binary relay (This can be done by editing the axis2.xml and setting the correct builder and formatter) to send a message through ESB. Then I measure the performance of this system again with 500,000 messages using java bench. Here are the results.

-n 500 -c 1000 - 1,179.16
-n 1000 -c 500 - 1,360.03
-n 2000 -c 250 - 1,471.56
-n 5000 -c 100 - 1,532.91
-n 10000 -c 50 - 1,526.42

It has reduced the performance around 8 times. After talking to different people I found that the ESB performance can be fine tuned as given here. Basically this increases the threads to maximize the performance. After starting the server with these parameters it gave me an exception saying too many open files. In order to fix this I had to increase the OS level threads as follows.

1. Increased the number of file handlers
sudo vi /etc/sysctl.conf
fs.file-max = 1000000
fs.inotify.max_user_watches = 1000000

2. Increase the number of files to user
sudo vi /etc/security/limits.conf
amila soft nofile 100000
amila hard nofile 100000

These two fixed the above problem and I could start the server successfully. Then I ran the earlier test again. Here is the results.

-n 500 -c 1000 - 5,562.29
-n 1000 -c 500 - 5,628.08
-n 2000 -c 250 - 5,679.52
-n 5000 -c 100 - 5,609.53
-n 10000 -c 50 - 5,159.77

In this case it was around 4 time increment of the performance. But still it is half of the direct back end performance. In this point after running with different options I concluded this as an I/O issue since the message now have to go through an extra hop. I ran all tests in my machine.
Then I thought of comparing these results with another ESB to check my above assumption. For that I choose UltraESB 1.7.1 since it has claimed over 2 times performance gain for a direct proxy invocation here. Then I added some extra jars as given in the performance tool site. Then ran the tests with a direct proxy service and here were the results.

-n 500 -c 1000 - 4,591.54
-n 1000 -c 500 - 3,797.29
-n 2000 -c 250 - 2,826.34
-n 5000 -c 100 - 1,516.03
-n 10000 -c 50 - 1,140.33

Here for some reason it shows very low performance at low concurrency levels and increase with the concurrency. Generally systems shows low performance at low concurrency but I think 50 is a sufficiently enough concurrency level. The other observation I made in this regard was the load average factor with the top tool in linux. It was kept on increasing through out the whole tests. This may be due to not fine tuning the server. However I could not find any document for that. In the given documents it was mentioned that detect the available processors automatically and adjust the parameters accordingly.
Then I tried with the zeroCopyEnabled since there was a claim for its high efficiency. Again there was not much difference from the results. Still there is no any significant performance improvement.

-n 500 -c 1000 - 4,602.64
-n 1000 -c 500 - 3,774.64
-n 2000 -c 250 - 2,644.13
-n 5000 -c 100 - 1,538.64
-n 10000 -c 50 - 1,123.28

After this I was thinking why this test shows a complete different result compared to bench mark results given. One of the observation I made was the low number of messages which may not have produced the consistent results. Although when doing a performance test it is required to send large amount of messages to obtain consistent results there is no reason it always shows a better performance of one ESB always as well. And also there is a possibility of WSO2 ESB is not being tuned for the performance. According to the above results after fine tuning it has showed higher performance.
In summary I learned two things from this. First routing through an ESB always degrade the TPS of the system due to additional I/O. Secondly it is always better to measure the performance with the required system in the hardware it suppose to run instead of being rely on the performance bench marks.

Java Security Model

An OS process runs under the privileges of the user who fork the process. Therefore under normal conditions a java applications also runs under the privileges of the user who starts it. But there are situations where some privileges of the person who starts it should not be granted to the java application and only set of java code should give the privilege access.
Java security model provides a solution for this problem. It let users to specify a set of granted permissions and act as a additional security layer between java application and OS allowing only granted permissions.
To understand the concepts lets use the following sample program which creates the file test/test.txt.
public class CreateFile {
public void createFile(){
File file = new File("test/test.txt");
try {
System.out.println("Creating file ==> " + file.getAbsolutePath());
file.createNewFile();
} catch (IOException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
new CreateFile().createFile();
}
}
Now lets run the program with the following command as a normal user
java -classpath classes/ com.test.security.CreateFile

This would create the file without any problem in the test directory. Here we ran this program without the security manager. Now lets try to run this program as super user but with the security manager.

java -Djava.security.manager -classpath classes/ com.test.security.CreateFile
Exception in thread "main" java.security.AccessControlException: access denied (java.util.PropertyPermission user.dir read)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:264)
at java.security.AccessController.checkPermission(AccessController.java:427)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
at java.lang.SecurityManager.checkPropertyAccess(SecurityManager.java:1285)
at java.lang.System.getProperty(System.java:628)
at java.io.UnixFileSystem.resolve(UnixFileSystem.java:118)
at java.io.File.getAbsolutePath(File.java:473)
at com.test.security.CreateFile.createFile(CreateFile.java:27)
at com.test.security.CreateFile.main(CreateFile.java:35)

So Even this simple operation runs as the super user it is not allowed to create the file. When a program runs under the security manager it only exposes the security permissions allowed in the security policy files. This is similar to Linux Capabilities. In this way only the required security permission can be given even the program has to started as super users.
Now lets give the necessary permissions to run this program.
First lets create the policy file as follows
grant {
permission java.util.PropertyPermission "user.dir", "read";
permission java.io.FilePermission "test/test.txt", "write";
};

And now run this program as follows,
java -Djava.security.manager -Djava.security.policy=policy/sample.policy -classpath classes/ com.test.security.CreateFile

Here the -Djava.security.policy is used to specify the policy file location. Now again file is created successfully.

To understand what happens exactly lets implements our own security permission. Here we write a watch tv permission.
public class TVPermission extends BasicPermission {
public TVPermission(String name) {
super(name);
}
public TVPermission(String name, String actions) {
super(name, actions);
}
public boolean implies(Permission p) {
boolean isPermitted = false;
if (p instanceof TVPermission){
isPermitted = p.getName().equals(getName())
&& p.getActions().equals(getActions());
}
return isPermitted;
}
}

Any java permission should implements the java.security.Permission interface. Implies is the important method. Actually when someone requests a permission normally java creates a permission object for that object and check for the permission. Lets use this code to check this.

public class WatchTV {
public void watchTV() {
TVPermission tvPermission = new TVPermission("chanel-5", "watch");
AccessController.checkPermission(tvPermission);
// other code goes here
}
public static void main(String[] args) {
new WatchTV().watchTV();
}
}

Now run this program with the following command
It gives this exception
Exception in thread "main" java.security.AccessControlException: access denied (com.test.permission.TVPermission chanel-5)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:264)
at java.security.AccessController.checkPermission(AccessController.java:427)
at com.test.security.WatchTV.watchTV(WatchTV.java:27)
at com.test.security.WatchTV.main(WatchTV.java:32)

Now lets add this permission to the policy file as follows and run with the policy file.
grant {
permission java.util.PropertyPermission "user.dir", "read";
permission java.io.FilePermission "test/test.txt", "write";
permission com.test.permission.TVPermission "chanel-5", "watch";
};

java -Djava.security.manager -Djava.security.policy=policy/sample.policy -classpath classes/ com.test.security.WatchTV

Now it executes correctly. Now we can understand how java runtime manage permissions under a security manager.
Lets see how actually jdk 1.5 implements the createNewFile method.

public boolean createNewFile() throws IOException {
SecurityManager security = System.getSecurityManager();
if (security != null) security.checkWrite(path);
return fs.createFileExclusively(path);
}

First it gets the Security manager and if it is not null (i.e when running under security manager) check for the write permission. This checkWrite method is like this,
public void checkWrite(String file) {
checkPermission(new FilePermission(file,
SecurityConstants.FILE_WRITE_ACTION));
}
public void checkPermission(Permission perm) {
java.security.AccessController.checkPermission(perm);
}

This finally calls the AccessController as in earlier sample.

Axis2 temp files

Axis2 supports archive based deployment for both services (.aar) and modules (.mar). These files can internally have 'lib' folders to contain the third party libraries of those services.
From Axis2 1.5, when Axis2 creates Configuration Context object which happens when the server starts up or service client invocation, it creates a separate folder under the temp folder to store the temp files. This temp folder is taken from the "java.io.tmpdir" system property.
For an example if Axis2 1.5 is deployed under tomcat 6.0.24, it creates the following two files under $CATALINA_TMPDIR.
axis2-tmp-2165190954418103759.tmp
axis2-tmp-2165190954418103759.tmp.lck

Axis2 creates .lck files as a temp file which deletes on a jvm exits.

File lockFile = new File(tmpDirName, tmpDir.getName() + ".lck");
lockFile.createNewFile();
lockFile.deleteOnExit();

Therefore when the tomcat goes down this .lck file should also be deleted. But the .tmp folder remains. At the next start up time Axis2 searches for .tmp file folders which does not have a corresponding .lck files and deletes them. So there can not be any grow of temp files.

One important thing to remember is that Axis2 copies the repository files (.aar and .mar) when ever it creates a configuration context object. Therefore the tmp folder can be grown while in the server running. Obviously this can be stopped by using one configuration context object for all the service invocations.

But there are some instances where .lck file is not properly deleted when the jvm exits. For those situations the temp folder can be deleted within the server starts up shell script. For tomcat this can be done adding code to catalina.sh.

Queued Transaction processing with WS-RM

Couple of years ago I started writing a new WS-RM implementation called Mercury which uses an state machine model to handle the reliability. It was working well for the requirements we had at that time and we have done two releases under the name WSO2 Mercury. But however WSO2 Mercury could not support the user initiated transactions due to a limitation of its storage design. Then that project didn't last too long and WSO2 moved to use Apache Sandesha as its WS-RM implementation for newly developed carbon based products.
At that time I was doing my Msc at University of Moratuwa. For my Msc I was looking for an idea and Dr Sanjiva Weerawarana and Mr Paul Fremantle gave me the idea of re engineering the WSO2 mercury code to support transactions by introducing a new transaction based storage API. Then I started researching different types of reliable message delivery models and finally was able to developed a WS-RM protocol which provides application to application reliability using distributed transactions. This model follows a similar pattern as the well known Queued Transaction processing and hence the thesis tile.

Here is the abstract of my Msc Thesis
With the popularity of the distributed business applications, the application data is distributed in various physical storages. However most of the business transactions require to update data stored in more than one storage. Hence updating two data storages reliably is a common problem for most of the distributed business applications.

Queued transaction processing is a concept widely used to achieve such a processing model using intermediate queues to transfer messages reliably. In such a system at the client side, both updating the client storage and writing the message to be sent to the client side message queue happens in the same distributed transaction. Similarly at the server side reading the message from the server side queue and updating the sever storage happens in the same distributed transaction. But such a system may have interoperability problems if client and server use different types of technologies.

Web services are used to communicate among the heterogeneous systems by passing SOAP messages using standard transport mechanisms like http. Web services can reliably communicate by using WS-Reliable messaging specification(WS-RM). WS-RM uses concepts of Reliable messaging source (RMS) and Reliable messaging destination (RMD) between which it guarantees reliable message delivery.

By combining these two concepts, we introduce an approach to solve the above mentioned problem in an interoperable manner using WS-RM to communicate between nodes while keeping RMS and RMD as intermediate storages. In our model reliable message delivery happens in three phases. First both updating application client storage and writing message to the RMS happens in the same distributed transaction. Then WS-RM protocol reliably transfers the message to RMD at the server side. Finally at the server reading the message from the RMD and updating the server storage happens in the same distributed transaction. The middleware software entity that we developed to encapsulate this approach is called Mercury which implements WS-RM protocol.

Msc Thesis report
CD copy which provides the source code, binaries and demos

WSO2 WSAS 3.1.3 released

WSO2 WSAS is an Enterprise ready web service container based on Apache Axis2 and related other libraries such as Rampart, Sandesha2, Transports etc... Then what is the advantage of using WSO2 WSAS rather than using those apache libraries directly?

WSO2 WSAS comes with an integrated and tested set of all those libraries so that users do not have to find the compatible libraries and integrate them.

WSO2 WSAS Admin console provides the extensive monitoring, management features.
The try it can be used to test a web service quickly without generating the stubs. Users can change the log levels through the admin console and can enable request/response message logging in order to check the messages comes in and out.
Admin Console can be used to update servers while they are running. I.e users can add parameters, engage modules (there are even sample scenarios to engage Security), apply polices, add transports, add services etc .. WSO2 WSAS persists all these changes to its registry so that it is available even if some one restart the server. This is very useful for services that do not uses a services.xml like jaxws servers.
There are some graphical tools called wsdl2java, java2wsdl, WSDL Converter, Try it, Service Validator and Module Validator.

WSO2 WSAS 3.1.3 can be downloaded here.

Improving Axis2 http transport client

Until Axis2 1.5 a separate http client instance was created per request by default. This leads to an connection timeout problem with the high loads. See here for more details.

In order to solve this issue Axis2 1.5.1 uses one http client object cached in configuration context for all requests. Since by default MultiThreadedHttpConnectionManager allows two request per host this causes an issue if tried to invoke a service more than twice.

Following options can be taken to solve this issue.
1.set HTTPConstants.AUTO_RELEASE_CONNECTION to true
serviceClient.getOptions().setProperty(HTTPConstants.AUTO_RELEASE_CONNECTION, Constants.VALUE_TRUE);
this would build the response stream once the response stream returned and release the connection. But this may have a performance hit since it build the OM tree at transport level.
2.Clean up the transport after each call
serviceClient.cleanupTransport();

However both the above methods can only invoke two requests at time since there is only two connections. Again this may cause problems with invoking slow services. This can be avoided by the following technique to increase the default number of connections.

ConfigurationContext configurationContext =
                    ConfigurationContextFactory.createConfigurationContextFromFileSystem(
                            AXIS2_REPOSITORY_LOCATION, AXIS2_CLIENT_CONFIG_FILE);

            MultiThreadedHttpConnectionManager multiThreadedHttpConnectionManager = new MultiThreadedHttpConnectionManager();

            HttpConnectionManagerParams params = new HttpConnectionManagerParams();
            params.setDefaultMaxConnectionsPerHost(20);
            multiThreadedHttpConnectionManager.setParams(params);
            HttpClient httpClient = new HttpClient(multiThreadedHttpConnectionManager);
            configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);

and call serviceClient.cleanupTransport(); after each service invocation.

Changing Axis2 application path

By default Axis2 generates an context path as follows when deployed it in tomcat or other application server.

http://localhost:8080/axis2/services/Version?wsdl

but most of the it is requried to customize this as follows.

http://localhost:8080/myApp/myServices/Version?wsdl

Below steps can be used to achive this.
1. Rename the web app name axis2 to myApp
2. Uncomment and edit the servicePath parameter at the axis2.xml file
myServices
3. Add the following entry to web.xml. Here we need to add this entry rather than chaing existing one since services part is hard coded in some places.

<servlet-mapping>
<servlet-name>AxisServlet</servlet-name>
<url-pattern>/myServices/*</url-pattern>
</servlet-mapping>

How to get a path like this?

http://localhost:8080/myServices/Version?wsdl

This can be done my making this application as ROOT app
1. change the myApp to ROOT