Thursday, May 7, 2009

Improving Axis2 client http transport performance

Although Axis2 supports a many transports http is the highly used one. Axis2 uses Apache HttpClient to implement its http transports sender. By default Axis2 creates a new http client object per invocation. This causes a new MultiThreadedHttpConnectionManager object being created for each invocation. As a result axis2 by default does not use the http 1.1 keep alive feature as well.
The solution for this problem is to re use the same HttpClient and MultiThreadedHttpConnectionManager per one thread. Axis2 client invocation may happen using one thread or using multiple threads. After reusing the MultiThreadedHttpConnectionManager has to shut down to release connections. If not this can lead to multiple tcp CLOSE_WAIT connections situation.

Following code shows this settings.

// creates a new connection manager and a http client object
MultiThreadedHttpConnectionManager httpConnectionManager = new MultiThreadedHttpConnectionManager();
HttpClient httpClient = new HttpClient(httpConnectionManager);

try {

client = new ServiceClient(null, null);
Options opts = new Options();
client.setOptions(opts);
opts.setTo(new EndpointReference("http://localhost:8085/axis2/services/TestInOutService"));
opts.setAction("urn:TestInOutService");

OMElement payload = buildSoapObject(
"" +
"IBM" +
"
"
);
System.out.println("Sending the request ...." + System.currentTimeMillis());


// set the above created objects to re use.
client.getOptions().setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Constants.VALUE_TRUE);
client.getOptions().setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);

// since we reuse the same http client and conneciton manager all these http messages
// will go in one tcp connection.
// this can be verified by sending the messages through tcpmon.
for (int i = 0; i < 10; i++) {
OMElement response = client.sendReceive(payload);
response.build();
}


} catch (Exception e) {
e.printStackTrace();
}
finally {
if (client != null) {
try {
client.cleanupTransport();
} catch (Exception e) {
e.printStackTrace();
}

try {
client.cleanup();
} catch (Exception e) {
e.printStackTrace();
}
}

}
try {
Thread.sleep(1000);
} catch (Exception e) {// do nothing}
}
httpConnectionManager.closeIdleConnections(0);
httpConnectionManager.shutdown();

20 comments:

Vikas Jain said...

Hi Amila,

I must say that this blog is the most clear about axis2 http transport configuration among many other stuff on the internet.

While using Axis2, after many soap calls, I get the following error:

org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:619) Caused by: javax.xml.soap.SOAPException: Timeout waiting for connection at org.apache.axis2.saaj.SOAPConnectionImpl.handleSOAPMessage(SOAPConnectionImpl.java:194) at org.apache.axis2.saaj.SOAPConnectionImpl.call(SOAPConnectionImpl.java:130) at amazon.subway.mapps.accmgmtui.payment.sduihelpers.SSWeb.callSubwayImpl(SSWeb.java:308) ... 38 more Caused by: org.apache.axis2.AxisFault: Timeout waiting for connection at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430) at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:203) at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76) at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400) at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225) at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435) at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402) at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229) at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165) at org.apache.axis2.saaj.SOAPConnectionImpl.handleSOAPMessage(SOAPConnectionImpl.java:188) ... 40 more Caused by: org.apache.commons.httpclient.ConnectionPoolTimeoutException: Timeout waiting for connection at


I have the following constraints because of which I cannot setup a dynamic configuration but can only use the static configuration of axis2.xml file.

The constrain is that I am using connection = SOAPConnectionFactory.newInstance().createConnection();
SOAPMessage response = connection.call(msg, url);

to make a call to a webservice. I am bound to use this way of calling webservice because of various other technical and non-technical constraints.

But I am sure that I am using Axis2 because SOAPConnectionFactory.newInstance().createConnection() call returns a axis2 soap connection (org.apache.axis2.saaj.SOAPConnectionImpl)

Also the axis2.xml file is present in one of the dependent jars.. but I am not sure if that is being picked up or not .. this is because when I setup a property in that jar's xml file as (by uncompressing the jar, doing the changes in axis2.xml and again creating the jar):


<transportSender name="https"
class="org.apache.axis2.transport.http.CommonsHTTPTransportSender">
<parameter name="PROTOCOL">HTTP/1.1</parameter>
<parameter name="Transfer-Encoding">chunked</parameter>
<parameter name="REUSE_HTTP_CLIENT">true</parameter>
<parameter name="Keep-Alive">true</parameter>
<parameter name="userAgent">MyAxis2</parameter>
</transportSender>

The user agent is still coming as "Axis2" (default) in the SOAP request (looked through tcpmon) while I have MyAxis2 setup in the axis2.xml file.

Can I tell Axis2 to use the axis2.xml from a particular location?
I tried looking at the configuration context of axis2, but since I am not using the standard Axis2 calls.. i am unable to feed the configuration context to any call at runtime.

Anonymous said...

Hi Amila,
I have posted my problem here, and wanted to get your feedback while waiting for any response.

http://old.nabble.com/Third-web-service-invoctaion-always-throws-Timeout-Exception-to27029690.html#a27135054

Can you please take a few mins to comment? I have run out of options at this point and need help desperately.

Thanks,
Tariq.

Anonymous said...

Hi Amila,
I didn't leave any contact info for you to respond to. Please send me your comments to tariq.islam@daston.com.

Thanks,
Tariq.

Amila Suriarachchi said...

if you use Axis2 1.5.1 it is enough to cleanup the transports.

i.e.
serviceClient.cleanupTransport();

jackbenimble said...

Hi Amila,

Wow, thank you! This saved me probably days of work in less than an hour. How did you figure this one out?

Jack

Usman Khan said...

Hi Amila,

I see the following exception intermittently - Using axis 1.5.1 and the client was built using ADB. Have not touched any configuration settings. Do you have any idea about this - many thanks

org.apache.commons.httpclient.ConnectionPoolTimeoutException: Timeout waiting for connection
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:497)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at ...

Anonymous said...

I saw a similar issue of "connection timeout" exceptions in axis2 1.5.1 on the 3rd HTTPS request to a service. Calling client.cleanupTransport() would throw "com.ctc.wstx.exc.WstxIOException: Attempted read on closed stream." exception. On a normal HTTP request, calling cleanupTransport() didn't throw exceptions and I was able to make multiple requests. I downloaded the axis2 1.5.1 source and in the ServiceClient's sendReceive method I saw this chunk of code:

if (options.isCallTransportCleanup()) {
response.getEnvelope().build();
cleanupTransport();
}

This lead me to setting the setCallTransportCleanup(true) option property prior to my call to the client's sendReceive method:

OMElement result = null;
switch(invokeMethod) {
case Axis2WSUtils.SEND_ROBUST :
client.sendRobust(payload);
client.cleanupTransport();
break;
case Axis2WSUtils.FIRE_AND_FORGET :
client.fireAndForget(payload);
client.cleanupTransport();
break;
default :
// Setting this option fixes a connection timeout issue when more than two requests go to the same https service
client.getOptions().setCallTransportCleanup(true);
result = client.sendReceive(payload);
}
return result;

No other fixes I found online like setting AUTO_RELEASE_CONNECTION, REUSE_HTTP_CLIENT or using the MultiThreadedHttpConnectionManager seemed to help my situation.

Anonymous said...

Hi Amila,
i've a problem of connection reset when my stub tries to invoke the endpoint.
What's the best practice to force the connection release (and close) on axis2 ?

thank you for any help.

Anonymous said...

Hi,

I had the same issue, but with generated stubs, which made it more difficult to rectify.

Just in case if somebody finds it helpful, I'm posting my solution. I looked into Axis2 and HttpClient sources to find what to do. There may be a more sratightforwards solution.

MyServiceStub myService = new MyServiceStub("myServiceURL");

ServiceContext context = myService ._getServiceClient().getServiceContext();

MultiThreadedHttpConnectionManager connManager = (MultiThreadedHttpConnectionManager)context.getProperty(HTTPConstants.MULTITHREAD_HTTP_CONNECTION_MANAGER);

if(connManager == null) {
connManager = new MultiThreadedHttpConnectionManager();
context.setProperty(HTTPConstants.MULTITHREAD_HTTP_CONNECTION_MANAGER, connManager);
connManager.getParams().setMaxTotalConnections(100);
connManager.getParams().setMaxConnectionsPerHost(HostConfiguration.ANY_HOST_CONFIGURATION, 100);
}

Regards,
Stan

Anonymous said...

Hi Amila,

i am using axis2 1.5 and have trouble with CLOSE_WAIT connection. My axis2 client uses CACHED_HTTP_CLIENT with a MultiThreadedHttpConnectionManager. The REUSE_HTTP_CLIENT option is set as default in this axis2 version.
The HttpClient is limited to 2 connections.

I am on client-side running against a remote soap service. Sometimes the server returns respones with http-status 500. Such responses contain "conection: close" in http header. The server closes the tcp connection by sending FIN,ACK and client replies with ACK and send FIN,ACK with also is replied by server. In some cases the client does ACK but not send FIN,ACK, the connection is left in state CLOSE_WAITING.

In result the commons httpclient hangs. Further axis2 soap requests on this client hang while httpclient says " Unable to get a connection, waiting..., hostConfig=..."

Any idea on this issue?

Anonymous said...

Hi Amila,
Great post. I do have a question wrt scalabilty. We do need to call "different" web services (many, many times during peak workload) and I wanted to get your recommendation. We use the OperationClient interface with the REUSE HTTP CLIENT Option with Security in some cases. Do you suggest using the same OperationClient for all services or one operationClient per service? Is it thread-safe, re-entrant? I don't know what happens under the hood of Axis2 and wanted to understand what makes sense in terms of scalability. Also do you recommend using the MultiThreadedHttpClient? Can you please throw a little more light? Thanks!

dipesh said...

Hi AMILA,
I have written a axis2 1.4 web service which receive a request and modify that request and make a soap message and  send it to another axis 2 1.4 service using operation client.
If load is less say in the range of 30000-40000 it will work fine , but when request will increase , from first service request will sent to second service , but second service receive that request after a long time say after 1 hour . I think request will stay in queue and wait for other Is there any setting for Operation client so that we can fix that issue?

Thanks in advance

Dipesh
dIPESH

Anonymous said...

Hi Amila,

I want each of my request to have different time out value. Is there a simple way to do it.

Also I set timeout at two places, one in the stub (setTimeoutMilliseconds()) and in the HttpConnectionManagerParams (setConnectionTimeout()). What is the difference between the two.

Please help me

maroc said...

Hi, I m seeing similar things in wso2 esb 1.6 / 1.5.0 Java version

indeed I saw many connection either in CLOSE_WAIT or TIME_WAIT

but this is what I m seeing when ESB tried to send back a response to the client :

INFO | jvm 1 | 2012/06/28 09:59:40 | Exception in thread "HttpServerWorker-6" java.lang.NoSuchFieldError: ACKED
INFO | jvm 1 | 2012/06/28 09:59:40 | at org.apache.synapse.transport.nhttp.ServerWorker.run(ServerWorker.java:204)
INFO | jvm 1 | 2012/06/28 09:59:40 | at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
INFO | jvm 1 | 2012/06/28 09:59:40 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
INFO | jvm 1 | 2012/06/28 09:59:40 | at java.lang.Thread.run(Thread.java:595)
INFO | jvm 1 | 2012/06/28 09:59:40 | 2012-06-28 09:59:40,458 [127.0.0.1-portal.madeupdomain.com] [HttpClientWorker-6] ERROR ClientWorker Fault creating response SOAP envelope
INFO | jvm 1 | 2012/06/28 09:59:40 | org.apache.axis2.AxisFault
INFO | jvm 1 | 2012/06/28 09:59:40 | at org.apache.axis2.AxisFault.makeFault(AxisFault.java:417)
INFO | jvm 1 | 2012/06/28 09:59:40 | at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:61)
INFO | jvm 1 | 2012/06/28 09:59:40 | at org.apache.axis2.transport.TransportUtils.createDocumentElement(TransportUtils.java:160)
INFO | jvm 1 | 2012/06/28 09:59:40 | at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:111)
INFO | jvm 1 | 2012/06/28 09:59:40 | at org.apache.synapse.transport.nhttp.ClientWorker.run(ClientWorker.java:169)
INFO | jvm 1 | 2012/06/28 09:59:40 | at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
INFO | jvm 1 | 2012/06/28 09:59:40 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
INFO | jvm 1 | 2012/06/28 09:59:40 | at java.lang.Thread.run(Thread.java:595)
INFO | jvm 1 | 2012/06/28 09:59:40 | Caused by: java.nio.channels.ClosedChannelException
INFO | jvm 1 | 2012/06/28 09:59:40 | at sun.nio.ch.SourceChannelImpl.ensureOpen(SourceChannelImpl.java:136)
INFO | jvm 1 | 2012/06/28 09:59:40 | at sun.nio.ch.SourceChannelImpl.read(SourceChannelImpl.java:140)
INFO | jvm 1 | 2012/06/28 09:59:40 | at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:42)
INFO | jvm 1 | 2012/06/28 09:59:40 | at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:92)
INFO | jvm 1 | 2012/06/28 09:59:40 | at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
INFO | jvm 1 | 2012/06/28 09:59:40 | at java.io.FilterInputStream.read(FilterInputStream.java:111)
INFO | jvm 1 | 2012/06/28 09:59:40 | at java.io.PushbackInputStream.read(PushbackInputStream.java:161)
INFO | jvm 1 | 2012/06/28 09:59:40 | at org.apache.axis2.builder.BuilderUtil.getCharSetEncoding(BuilderUtil.java:237)
INFO | jvm 1 | 2012/06/28 09:59:40 | at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:48)
INFO | jvm 1 | 2012/06/28 09:59:40 | ... 6 more

The class path is a little different "org.apache.synapse.transport.nhttp" instead of axis2

Any hint on how to avoid this ?

Anonymous said...

None of the below statements could fix the problem:
stub._getServiceClient().cleanup();
stub._getServiceClient().cleanupTransport();
stub.cleanup();
mtHttpConnMngr.closeIdleConnections(0);
mtHttpConnMngr.shutdown();
mtHttpConnMngr.deleteClosedConnections();
context.cleanupContexts();
context.flush();
context.terminate();
--------------------------------
I also tried the below:
serviceStub._getServiceClient().getOptions().setProperty(HTTPConstants.HTTP_PROTOCOL_VERSION,HTTPConstants.HEADER_PROTOCOL_10);

But, still CLOSE_WAIT exists. I am using Axis2-1.5.1

Anonymous said...

Thank you very much to Stan!!!
(Comment of January 12, 2011 7:04 AM)

Your code save us!!!

Unknown said...

Hello!

I just would like to give a huge thumbs up for the great info you have here on this post. I will be coming back to your blog for more soon.

man and van watford House Removals


Man and van watford House Removals

Unknown said...

Hi Amila,

First thanks for providing such type of blogs.
My code is working for 50k data but when try to get 100k data then i am getting exception Connection reset. Sample code is:
client = new AuthServiceStub(url);
client._getServiceClient().getOptions().setProperties((Map)getAttribute("clientOptions"));

What i need to correct so that my code work properly.

Júlio César said...

Thanks buddy, you saved my life!

Vishal said...

HI , we are using axis2 1.5.1 implementation for webservices .Currently we are facing issue when concurrent number of requests are increased between two products communicating using webservice. We have already implemented this solution i.e.

ConfigurationContext context = this._webService._getServiceClient().getServiceContext().getConfigurationContext();
MultiThreadedHttpConnectionManager multiThreadedHttpConnectionManager = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams params = new HttpConnectionManagerParams();
params.setDefaultMaxConnectionsPerHost(50);
multiThreadedHttpConnectionManager.setParams(params);
HttpClient httpClient = new HttpClient(multiThreadedHttpConnectionManager);
context.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, true);
context.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);


What is If increase setDefaultMaxConnectionsPerHost to 100 ? will it solve the problem ? Or is there any other way ?