Sunday, January 17, 2010

Improving Axis2 http transport client

Until Axis2 1.5 a separate http client instance was created per request by default. This leads to an connection timeout problem with the high loads. See here for more details.

In order to solve this issue Axis2 1.5.1 uses one http client object cached in configuration context for all requests. Since by default MultiThreadedHttpConnectionManager allows two request per host this causes an issue if tried to invoke a service more than twice.

Following options can be taken to solve this issue.
1.set HTTPConstants.AUTO_RELEASE_CONNECTION to true
serviceClient.getOptions().setProperty(HTTPConstants.AUTO_RELEASE_CONNECTION, Constants.VALUE_TRUE);
this would build the response stream once the response stream returned and release the connection. But this may have a performance hit since it build the OM tree at transport level.
2.Clean up the transport after each call
serviceClient.cleanupTransport();

However both the above methods can only invoke two requests at time since there is only two connections. Again this may cause problems with invoking slow services. This can be avoided by the following technique to increase the default number of connections.

ConfigurationContext configurationContext =
                    ConfigurationContextFactory.createConfigurationContextFromFileSystem(
                            AXIS2_REPOSITORY_LOCATION, AXIS2_CLIENT_CONFIG_FILE);

            MultiThreadedHttpConnectionManager multiThreadedHttpConnectionManager = new MultiThreadedHttpConnectionManager();

            HttpConnectionManagerParams params = new HttpConnectionManagerParams();
            params.setDefaultMaxConnectionsPerHost(20);
            multiThreadedHttpConnectionManager.setParams(params);
            HttpClient httpClient = new HttpClient(multiThreadedHttpConnectionManager);
            configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);

and call serviceClient.cleanupTransport(); after each service invocation.

50 comments:

Hash said...

Hi Amila,

Lots of useful info about Axis2 in your blogs.

It would be great if you could throw some light about whether axis2 supports or . Looks like Axis2 allows only for a complex element type.

Thanks in advance
Harsha

Amila Suriarachchi said...

about whether axis2 supports or

what do you mean by this?

Hash said...

Sorry about that, I guessed few lines got missed out in the middle.
I read in few forums that xs : all or xs : change is not supported due to Axis2 limitation. Hence always the complex element structure will have xs : sequence. Can you please let me know as to whether Axis2 supports xs : all or xs : change.

Amila Suriarachchi said...

Axis2 does support xs:all and xs:choice

Robert said...

Hi Amila:
I have a problem that seems to related to the http transport client. I have a 30+ thread-load axis2 module that calls another webservice using a singleton stub with the client option set to HttpClient which in turn uses the MultiThreadedHttpConnectionManager as you've described. The primary service and backup service are both running under the same axis2.war as aar files. After I've processed some requests I let the system quiesce and monitor it with jconsole. Exactly every 5 mintues the number of Current Classes Loaded goes up by 4. This can add up after some days. From a profiler I determined that the additional loaded classes seems to be coming from a new contextClassLoader of org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread ("MultiThreadedHttpConnectionManager cleanup"). My skills are not up to catching the class loading in the debugger, so I can only speculate that the cleanup process which blocks on connections being garbage collected is looping somehow and leaking loaded classes.
My questions:
1) Have you ever seen this?
2) Are there any parameters in the connection manager or client that I can set to control this behavior? (I've tried all the settings you mention and they make no difference)
3) Is there any good documentation on HTTPClient and HTTPMultiTheadedConnectionManager that explains all the options and what knobs you use for what purposes?
Thanks Much,
-Robert

Robert said...

One small correction on the above - I don't use a singleton stub. I have one client stub per thread (using a ThreadLocal variable). The stubs all share a single HttpClient & MultiThreadedHttpConnectionManager which are stashed in the ConfigurationContext

Hash said...

Thanks!!! great to know that Axis2 does support xs : all and xs : choice. Hence it should be able to support xs : any.

We are using document literal webservice deployment and using annotations for implementing the same. Could you please give info as to how to remove the sequence from a complex element (wrapped object arraylist) and have xs : any descriptor for the structure using annotations (Java 1.6).

Thanks in advance.

Upananda said...

Hi Amila,

I have been using axis2 1.4.1 with Java 1.6 and just wondering whether axis2 jaxws deployment (as jar in servicejars) allows to have the services.xml file in the jar packaging. I was trying to put original wsdl and xsd using useOriginalWsdl parameter and seems it's not working and still taking the dynamically generated wsdl. I need to modify the generated wsdl and put xs:all instead of xs:sequence (so that parmeter orders doesn't matter inside complex type of the soap request xml) which is the default setting in the dynamically generated wsdl. This will be very helpful if you can give me some clue regarding this.

Amila Suriarachchi said...

I have not seen your classloading issue. Seems to be a problem specific to your application.

Amila Suriarachchi said...

AFAIK jaxws does not support services.xml file.

Robert said...

Hi Amila: Thanks for the response. After more research it appears that the loaded classes are growing just because transientclass loaders are being allocated and garbage collection of the quiet system is in no hurry to reclaim them. If I manually run the GC then the loaded classes drops to a common baseline. So no problem after all.
-Robert

Upananda said...

Hi Amila,

Do you mean that I can't use a modified wsdl while deploying as a jar in servicejars. Is there any other way to put my wsdl and use it instead of always using the dynamically generated one. Can you please explain little bit.
Thanks in advance for your valuable information.

Upananda

Vineeth said...

Hi Amila,

I have the same issue with using Axis2 1.5.1 with a MultiThreadedHttpConnectionManager. My code runs fine with 1.4 and 1.5 but due to the change in 1.5.1 with regards to client reuse, my existing code just hangs saying Unable to get connection. I am reusing my Stub rather than creating one for each request. I have tried calling the Stub._getServiceClient().cleanupTransport once I get the response back, but it does not resolve teh issue.

Is there any way I can use MultiThreadedHttpConnectionManager while reusing the stub with Axis2 1.5.1?

Thanks in advance.

Vineeth said...

In continuation to my ealier issue description, I had also tried calling
httpConnectionManager.closeIdleConnections(0);
httpConnectionManager.shutdown();

but it does not help as then I get a connectionfactory not available error.

Amila Suriarachchi said...

Did you try the first option?

HTTPConstants.AUTO_RELEASE_CONNECTION to true
serviceClient.getOptions().setProperty(HTTPConstants.AUTO_RELEASE_CONNECTION, Constants.VALUE_TRUE)

As I have mentioned Axis2 1.5.1 allows only two simultaneous connections.you may increase it as I have given.

Vineeth said...

Hi Amila,

Tried setting the property as well as increasing the connection to 20, but it does not resolve the issue.

Shamik Majumdar said...

Hi Amila,
There is something wrong with Axis2 in 1.5.1 version. It stays up for a number of requests but after sometime, gives me this error

org.apache.axis2.AxisFault: Timeout waiting for connection
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:203)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at com.reg.report.client.main.BOReportServerWSInterface.getReport(BOReportServerWSInterface.java:72)
at com.reg.report.client.main.BOReportServerWSInterface$TestHelper.run(BOReportServerWSInterface.java:121)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.apache.commons.httpclient.ConnectionPoolTimeoutException: Timeout waiting for connection
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:497)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:542)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)
... 10 more


I do cleanup at the end of the webservice call -


finally {
if (sender != null) {
sender.cleanupTransport();
sender.cleanup();
}
}

Amila Suriarachchi said...

try to increase the default connection size as given above.
sometimes this can happen since Axis2 only allows 2 connections and there are more than two connections on one instance.

Shamik Majumdar said...

Hi Amila,
Thanks for your comment. However, in my case, I doubt increasing the max connection per instance will help. Since I create one service client for each request anyway. I have a method like getReport() which instantiates a new serviceclient, calls the webservice with SWA and gets the report back as attachment. When I call the getReport method from 100 different threads, after sometime I see the problem of "Timeout waiting for connection".

So when you mention "yhere are more than two connections on one instance" - which instance you refer to ? Instance of ServiceClient?

Amila Suriarachchi said...

No an Axis2 instance. That means a configuration context. Do you share the configuration context object within different services clients or creates a new one for each?

Shamik Majumdar said...

Hi Amila,
I do not create configuration context explicitly. I am running the client inside one Axis2 embedded webapp. So I just use the empty service client constructor to create a service client [ServiceClient sender = new ServiceClient();]. I hope that internally this call creates the configuration context. I am not too sure whether in this case, the configuration context will become a singleton or not ?
If it becomes one configuration context then I should get the error ..

Shamik Majumdar said...

Looks like whatever I do, I always get this exception -
Timeout waiting for connection

I tried your method also. I even increased the timeouts using -

options.setProperty(HTTPConstants.SO_TIMEOUT,3600000);
options.setProperty(HTTPConstants.CONNECTION_TIMEOUT,3600000);

But whatever I do, after someime - 2 minutes or so - all of my threads are coming out with exceptions. Bad thing is - it does not recover from this error. If it happens once, it keeps on happening.

For stress testing my web service, I wrote a simple threaded standalone application which calls the webservice 100 times from 10 threads.
After sometime, all the threads start producing the error message - " Timeout waiting for connection".

I even tried to stop HTTP 1.1 feature by using
options.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Constants.VALUE_FALSE);

but all in vein.. now I am scared to put it in production.

Robert said...

For me, when I tried to set the Cached HTTP client with 100 connections using the Options properties it did not work either. I could see that I was still using a MultiThreadedHTTPManager with 2 connections. So I set the cached client in the ConfigurationContexts properties, like this:
stub._getServiceClient().getServiceContext().getConfigurationContext().setProperty(HTTPConstants.CACHED_HTTP_CLIENT, yourhttpClient);
And all of my problems went away. Also if your threads are hanging around after calling the web service you may want to clean up the idle connections:
((MultiThreadedHttpConnectionManager)yourhttpClient.getHttpConnectionManager()).closeIdleConnections(0);
Hope that helps.

Jeff Hoffman said...

Hi Amila,

Thank you for all the helpful information. My application is experiencing continued issues, and I hope you can help me! I suspect threading issues somewhere in axis 1.5.1, since the errors we get are always in situations where we have multiple threads making service calls.

Relevant snippets of code are as follows:

static MultiThreadedHttpConnectionManager defaultHttpConnectionManager;

private static MultiThreadedHttpConnectionManager createConnectionManager() {
MultiThreadedHttpConnectionManager mgr = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams params = mgr.getParams();
if (params == null) {
params = new HttpConnectionManagerParams();
mgr.setParams(params);
}
params.setMaxTotalConnections(40);
params.setDefaultMaxConnectionsPerHost(20);
params.setSoTimeout(defaultTimeoutMillis);
return mgr;
}


private static HttpClient createHttpClient(MultiThreadedHttpConnectionManager connectionManager, int soTimeoutMillies) {
HttpClient result = new HttpClient(connectionManager);
HttpClientParams params = result.getParams();
if (params == null) {
params = new HttpClientParams();
result.setParams(params);
}
params.setSoTimeout(soTimeoutMillies);
return result;
}


private static synchronized ConfigurationContext getDefaultConfigurationContext() throws AxisFault {
// reuse HTTP connections if possible.
if (defaultConfigurationContext == null) {
defaultConfigurationContext = ConfigurationContextFactory.createConfigurationContextFromFileSystem(null, null);
defaultConfigurationContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Constants.VALUE_TRUE);
HttpClient httpClient = createHttpClient(defaultHttpConnectionManager, defaultTimeoutMillis);
defaultConfigurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
(HTTPConstants.AUTO_RELEASE_CONNECTION, true);
}

return defaultConfigurationContext;
}


...
ServiceClient client = new ServiceClient(configurationContext, null);

OMElement result = client.sendReceive(requestDoc);


We get intermittent parsing errors during the sendReceive call, but only when we have multiple threads executing at once.

I ran out of space so will post stack trace in another message.

Jeff Hoffman said...

Below is an example stack trace. We get a few different exceptions, but all are similar, happening during the parsting of the response from the service.

Please save me!!

Thank you,

Jeff

...
Caused by: org.apache.axis2.AxisFault
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:123)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:67)
at org.apache.axis2.description.OutInAxisOperationClient.handleResponse(OutInAxisOperation.java:354)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:417)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:540)
at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:521)
at com.idexx.soa.ServiceCall.callService(ServiceCall.java:433)
... 7 more
Caused by: java.lang.NullPointerException
at com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.arrangeCapacity(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipString(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$XMLDeclDriver.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.setInputSource(Unknown Source)
at com.sun.xml.internal.stream.XMLInputFactoryImpl.getXMLStreamReaderImpl(Unknown Source)
at com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLStreamReader(Unknown Source)
at org.apache.axiom.om.util.StAXUtils$1.run(StAXUtils.java:119)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.axiom.om.util.StAXUtils.createXMLStreamReader(StAXUtils.java:115)
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:58)
at org.apache.axis2.transport.TransportUtils.createDocumentElement(TransportUtils.java:197)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:145)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:108)
... 15 more

Term Papers said...

I have been visiting various blogs for my term papers writing research. I have found your blog to be quite useful. Keep updating your blog with valuable information... Regards

Anonymous said...

How multiple HTTP Client instances can lead to a connection timeout?

The connection timeout exception (ConnectionPoolTimeoutException) is created inside MultiThreadedHttpConnectionManager class.
If each web service invocation creates a new HTTP Client (default behavior) then the MultiThreadedHttpConnectionManager can have zero or one connection (so the MultiThreadedHttpConnectionManager never wait for a connection).

Please answer me, as I'm not sure I understand AXIS/HTTP Client very well.

Thanks

Amila Suriarachchi said...

multiple http client instances cause many connections at the CLOSE_WAIT state.

WS said...

Hi Amila,

This is a great post indeed. I do have a question about the solution though. If the client is an Async client, how should these proposed changes affect it? i.e. when should the cleanupTransport(), cleanup() be called? Being more precise should it be called immediately after the sendReceiveNonBlocking call is made or in the callback?

I use an async client where the same service is invoked simultaneously from multiple threads. I'm getting the problem that's mentioned in your post and managed to go beyond 2 invocations but not more than 20. I've set the max no. of connections to 2000 on the MultiThreadedHttpConnectionManager.

Thanks,

WSR

Bhagath Ganga said...

Hi Amila,

I have an issue with a webservice cleint built using axis2. All my webservices are non axis2 based. I created a axis2 client to dynamically invoke different websservices deployed on different servers. I was able to invoke a service for around 100 times using my client, but then after that it throws up saying "Timeout waiting for connection".

This is not the typical connection time we get after 3 continuous invocations. I invoked the service succesfully for around a 100 timesn now. If I invoke the same service using SOAP UI, it works fine.

Any pointers to what might be wrong with the client I built will be greatly appreciated!.

Bhagath Ganga

Anonymous said...

Hi Amila,
i've a problem of connection reset when my stub tries to invoke the endpoint.
What's the best practice to force the connection close after response on axis2, and to be sure of it ?

thank you for any help.

Nigel said...

Hi Amila

I believe many poeple have asked this but there doesn't appear to be a clear anwser when using axis-1.5.1

I am getting the same issue where multiple connections create the timeout. I duplicated it by following the quickstart guide and implementing the client/server StockQuote services.

I moved the client into the WAR and created a JSP page so that I can invoke via a web page, I also put a 30 second sleep in the server call to simulate a slow web service.

Now after 3-4 differemt simultaneous browser calls I get the timeout issue as expected.

So looking at the client code we are getting the stub:

StockQuoteServiceStub stub =
new StockQuoteServiceStub
("http://localhost:8080/axis2/services/StockQuoteService");

popultaing all the information and then calling the service:

GetPriceResponseDocument res =
stub.getPrice(reqDoc);

My question is in your example code about setting connections for the MultiThreadedHttpConnectionManager, so is this possible since all of our interaction is via a generated stub.

Thanks
Nigel.

tub said...

Hi Amila
I read the blogs/comments 100 times and i have the following problem now

org.apache.axis2.AxisFault: Transport error: 503 Error: Service Unavailable
at org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:311)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:200)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)


Code snippets

configContext =
ConfigurationContextFactory.createConfigurationContextFromFileSystem(
repoPath.toString(), confFile.toString());
MultiThreadedHttpConnectionManager multiThreadedHttpConnectionManager = new MultiThreadedHttpConnectionManager();

HttpConnectionManagerParams params = new HttpConnectionManagerParams();
params.setDefaultMaxConnectionsPerHost(20);
multiThreadedHttpConnectionManager.setParams(params);
HttpClient httpClient = new HttpClient(multiThreadedHttpConnectionManager);
HttpClient httpClient = new HttpClient(multiThreadedHttpConnectionManager);
configContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, true);
configContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
configContext.setProperty(HTTPConstants.AUTO_RELEASE_CONNECTION, true);
configContext.setProperty(HTTPConstants.CHUNKED,"false");


I had the timeout problem earlier,but after reading your blog, it was solved. Thank you very much for that ,but now i am struck with this

The following post says that i need to disable "chunking" which i am doing it but i still have the 503 error. I did chunking disabling at srvice stub level and as well as at configuration context level as well,

http://web.archiveorange.com/archive/v/fNNSSF0NP6Js0HUjICdT

I am using axis2-1.5.4
and commons-httpclient-3.1.jar
Thanks in Advance,
Bala

dilip gupta said...

I am using Exchange 2007 Web Services's stub (Java Client) for communicating with Exchange 2007 server (with service pack 3). I create stub from WSDL file using Axis2 tool.

I am able to make connection with Exchange 2007 stub but when we create new task item, it returns an error message. That is,

Exception in thread "main" java.lang.IllegalArgumentException: Null OutputStream specified
at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:577)
at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544)
at org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:209)
at com.microsoft.schemas.exchange.services._2006.messages.ExchangeServicesStub$54.serialize(ExchangeServicesStub.java:13598)

The code lines, which I am using for creating new task item, are,

CreateItemResponseDocument response = m_EWSservice.CreateItem(createItemDocument,
exchangeImpersonationRequestDocument,
serializedSecurityContextDocument,
mailboxCultureDocument,
requestServerVersionDocument,
timeZoneRequestDocument);

While debugging the code, error occurs at following line,

_operationClient.execute(true);

Please give me your thought about this issue.

Sri said...

Hi Amila,

With axis2 1.5.4 I am not receiving axis2 time out if I set the http client in the context so client is waiting forever for server response.

MultiThreadedHttpConnectionManager multiThreadedHttpConnectionManager =new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams params = new HttpConnectionManagerParams();
params.setDefaultMaxConnectionsPerHost(20);
multiThreadedHttpConnectionManager.setParams(params);
HttpClient httpClient = new HttpClient(multiThreadedHttpConnectionManager);
ctx.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient)

Anonymous said...

Hi Amila,

We are also using the second configuration but on load time we have seen the HTTP Transport header is getting used by different connection. and it is sending the cookie information which is coming as part of response in other thread.

What setting can avoid this . i m using 1.5.5 libs.

Can you help us in that.

Guirgis Nessim said...

Hello Amila,
Among the whole internet, i couldn't find helpful information such as your blog.

I'm having an issue with using axis2 as a web service client.

I'm using axis2-1.5.4, after issuing wsdl2java command to generate the stubs for a given WSDL.
I extended the generated stub and set some properties:
ws = new NBAWebService(wsURL);
ws._getServiceClient().getOptions().setTimeOutInMilliSeconds( TIME_OUT);
ws._getServiceClient().getOptions().setProperty( HTTPConstants.HTTP_PROTOCOL_VERSION, HTTPConstants.HEADER_PROTOCOL_10);
ws._getServiceClient().getOptions().setProperty( HTTPConstants.SO_TIMEOUT, TIME_OUT); ws._getServiceClient().getOptions().setProperty( HTTPConstants.CONNECTION_TIMEOUT, TIME_OUT);
ws._getServiceClient().getOptions().setProperty( HTTPConstants.AUTO_RELEASE_CONNECTION, Boolean.FALSE);
i had to put the timeout for business need, and its value = 1000 ms

after calling the stub method, in the finally block, i assumed the following code will close the connection:
ws._getServiceClient().cleanup();
ws.cleanup();


The problem is that after 4 days of starting the application, i get a memory leak on the Websphere application server, due to thousand of threads are not terminated, my guess is that when the timeout expires - it happens about 2000 times out of 400000 invocations per day - the connection thread isn't reclaimed, which accumulates over the days resulting in degrading the performance of the application server.

Do you have any comments about the way i'm using axis2 in my code? Do you have any suggestion on how to enforce all the connections to be closed especially in the case of timeout expiry?

Your help is greatly appreciated, thanks in advance.

Jeetu said...

Hi Amila,

I had an issue related to Axis.
At the time of connecting the Stub to the Server WSDL sometimes i get to the below error.
The following WSDL exception occurred: WSDLException: faultCode=WSDL4JWrapper : : javax.wsdl.WSDLException: WSDLException: faultCode=WSDL4JWrapper : : java.io.IOException: Server returned HTTP response code: 503 for URL:

After i get the above error Every request that tries to connect to Server using stub i keep on getting the above error.
Can you please guide me for the same.

Thanks,
Jeetu

sameer said...

I was stacked, when I was updating data with WSO2 data service using axis client. Your blog helped me to overcome this. Thanks.

sameer said...
This comment has been removed by the author.
Anonymous said...

Hi Amila,
Great post. I do have a question wrt scalabilty tho. We do need to call "different" web services (many, many times during peak workload) and I wanted to get your recommendation. We use the OperationClient interface with the REUSE HTTP CLIENT Option with Security in some cases. Do you suggest using the same OperationClient for all services or one operationClient per service? I don't know what happens under the hood of Axis2 and wanted to understand what makes sense in terms of scalability. Also do you recommend using the MultiThreadedHttpClient? Can you please throw a little more light? Thanks!

Pracil said...

Hi Amila,

First of all thanks for the blog !!.

I have a similar scenario where application takes longer duration ( Response is delayed ) so http client has to wait for the response. I am facing the timeout issue though i have set the " HTTPConstants.SO_TIMEOUT" and " HTTPConstants.CONNECTION_TIMEOUT " for 2 hrs.
below is the stack:
org.apache.axis2.AxisFault: Read timed out
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:203)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)

Is there any way or option to fix this issue.

Thanks,
Pracil

Thanks,
Pracil

kirpakaro said...

I am facing three issues with Axis1.4 version. Connection Timeout after certain requests , memory leak and Axis2 jar creation on each request.

I am using Axis2 Version 1.4 version with xmlbeans and I can't upgrade axis2 version due to client limitations.

First, it was creating a jar file in the tmp folder on Linux box for each request. So, I updated ServiceClient.java as per the recommendation on the following blog and problem was solved:
http://shivendra-tripathi.blogspot.com/2010/10/dealing-with-low-disk-space-problem.html

I am still not able to solve the remaining two issue i.e. Memeory Leak and Connection timeout after certain requests. I am going to try to update removeServiceGroup method in AxisConfiguration.java to include following block:


String serviceName = axisService.getName();
String key = null;
for (iter = axisService.getEndpoints().keySet().iterator(); iter.hasNext(); ) {
key = serviceName + "." + (String)iter.next();
this.allEndpoints.remove(key);
}

I have also added stub._getServiceClient().cleanup(); and stub.cleanup() calls in the finally block.

Let me know if i am missing something to resolve remaining two issues i.e. Memory Leak and Connection timeout after certain requests.. Any help will be highly appreciated!

kirpakaro said...

Any update on my two remaining issues as described above ? Is Amila replying to anyone? As i cna see everyone has asked Amila but don't see Amila's reponse on this blog...

Amila, Please provide your valuable suggestions to solve Axis2 problems.

Hoshi Sato said...

I am working with an old service that I cannot upgrade past Axis2 1.5.1, and am implementing the code example, but one thing you don't specify is where and when to use it. Since it is apparently affecting static/class variables, one might be tempted to put it very early in the process, but the implementation given suggests the location would be in the creation of the new connection - which begs the question, "Do it every time?" So, for now, pending my hope of a response to this comment, I have put it into the creation of the connection, but set a static boolean to indicate whether or not it's been done before, so the block of code is only called once (or maybe twice in the case of a race condition).

Is this correct? If there could be some discussion not only of the solution but the context in which to apply the solution, I'd be ever so grateful.

Thanks,
Hoshi

Hoshi Sato said...

Also, I am assuming that, since this app has serviced clients somewhat satisfactorily with the default 2 connections (rarely throwing the timeout exception), raising the ceiling to something like 20 would be incredibly drastic overkill, and have selected 5 as the new number.

Any advice on how to select the new default (inputs to the decision)?

Thanks,
Hoshi

Steve W. said...

I'm using axis2 1.5.4. After reading in the ServiceClient javadocs that ServiceClient is not intended to be thread-safe, I switched my Stub instance from a member variable (on a spring singleton bean) to a local method variable that reconnects every time. I was previously seeing "truncated XML" and after I switched it to a local variable, that problem went away. But now, under high load, I'm getting the linux error "Too many open files". We're not able to convince admins to update the ulimit. So, I started to look into the possibility that I could make a code or configuration change. I noticed that I was using MultiThreadedConnectionManager (which was no longer needed since there is one Stub per call, rather than a shared Stub) AND calling both ServiceClient.cleanup and Stub.cleanup, but not ServiceClient.cleanupTransport. I would like to add ServiceClient.cleanupTransport, but I'm trying to figure out whether to add it before or AFTER the call to ServiceClient.cleanup.

jim smith said...

Sometimes this can happen since Axis2 only allows 2 connections and there are more than two connections on one instance.
Transmissions Hollywood FL

Ravi Chandra said...

hi,

this link: https://issues.apache.org/jira/browse/AXIS2-2931 says that the connection is autoreleased. So setting the flag to false.. i think its redundant. what do u think?

Hans said...

Hi,
great article. I use MultiThreadedHttpConnectionManager with setMaxTotalConnections(20) for while now and it seemed to work fine.

But now I had serveral errors (application error on the server) in batch which led to

Caused by: org.apache.commons.httpclient.ConnectionPoolTimeoutException: Timeout
waiting for connection

after exact 20 calls.

To me the pool seems to run out of connections. Does anybody know, whether exceptions may hinder a connection to end up in the pool again.

How can I fix this.

Advise would be great.