Queued Transaction processing with WS-RM

Couple of years ago I started writing a new WS-RM implementation called Mercury which uses an state machine model to handle the reliability. It was working well for the requirements we had at that time and we have done two releases under the name WSO2 Mercury. But however WSO2 Mercury could not support the user initiated transactions due to a limitation of its storage design. Then that project didn't last too long and WSO2 moved to use Apache Sandesha as its WS-RM implementation for newly developed carbon based products.
At that time I was doing my Msc at University of Moratuwa. For my Msc I was looking for an idea and Dr Sanjiva Weerawarana and Mr Paul Fremantle gave me the idea of re engineering the WSO2 mercury code to support transactions by introducing a new transaction based storage API. Then I started researching different types of reliable message delivery models and finally was able to developed a WS-RM protocol which provides application to application reliability using distributed transactions. This model follows a similar pattern as the well known Queued Transaction processing and hence the thesis tile.

Here is the abstract of my Msc Thesis
With the popularity of the distributed business applications, the application data is distributed in various physical storages. However most of the business transactions require to update data stored in more than one storage. Hence updating two data storages reliably is a common problem for most of the distributed business applications.

Queued transaction processing is a concept widely used to achieve such a processing model using intermediate queues to transfer messages reliably. In such a system at the client side, both updating the client storage and writing the message to be sent to the client side message queue happens in the same distributed transaction. Similarly at the server side reading the message from the server side queue and updating the sever storage happens in the same distributed transaction. But such a system may have interoperability problems if client and server use different types of technologies.

Web services are used to communicate among the heterogeneous systems by passing SOAP messages using standard transport mechanisms like http. Web services can reliably communicate by using WS-Reliable messaging specification(WS-RM). WS-RM uses concepts of Reliable messaging source (RMS) and Reliable messaging destination (RMD) between which it guarantees reliable message delivery.

By combining these two concepts, we introduce an approach to solve the above mentioned problem in an interoperable manner using WS-RM to communicate between nodes while keeping RMS and RMD as intermediate storages. In our model reliable message delivery happens in three phases. First both updating application client storage and writing message to the RMS happens in the same distributed transaction. Then WS-RM protocol reliably transfers the message to RMD at the server side. Finally at the server reading the message from the RMD and updating the server storage happens in the same distributed transaction. The middleware software entity that we developed to encapsulate this approach is called Mercury which implements WS-RM protocol.

Msc Thesis report
CD copy which provides the source code, binaries and demos

WSO2 WSAS 3.1.3 released

WSO2 WSAS is an Enterprise ready web service container based on Apache Axis2 and related other libraries such as Rampart, Sandesha2, Transports etc... Then what is the advantage of using WSO2 WSAS rather than using those apache libraries directly?

WSO2 WSAS comes with an integrated and tested set of all those libraries so that users do not have to find the compatible libraries and integrate them.

WSO2 WSAS Admin console provides the extensive monitoring, management features.
The try it can be used to test a web service quickly without generating the stubs. Users can change the log levels through the admin console and can enable request/response message logging in order to check the messages comes in and out.
Admin Console can be used to update servers while they are running. I.e users can add parameters, engage modules (there are even sample scenarios to engage Security), apply polices, add transports, add services etc .. WSO2 WSAS persists all these changes to its registry so that it is available even if some one restart the server. This is very useful for services that do not uses a services.xml like jaxws servers.
There are some graphical tools called wsdl2java, java2wsdl, WSDL Converter, Try it, Service Validator and Module Validator.

WSO2 WSAS 3.1.3 can be downloaded here.

Improving Axis2 http transport client

Until Axis2 1.5 a separate http client instance was created per request by default. This leads to an connection timeout problem with the high loads. See here for more details.

In order to solve this issue Axis2 1.5.1 uses one http client object cached in configuration context for all requests. Since by default MultiThreadedHttpConnectionManager allows two request per host this causes an issue if tried to invoke a service more than twice.

Following options can be taken to solve this issue.
1.set HTTPConstants.AUTO_RELEASE_CONNECTION to true
serviceClient.getOptions().setProperty(HTTPConstants.AUTO_RELEASE_CONNECTION, Constants.VALUE_TRUE);
this would build the response stream once the response stream returned and release the connection. But this may have a performance hit since it build the OM tree at transport level.
2.Clean up the transport after each call
serviceClient.cleanupTransport();

However both the above methods can only invoke two requests at time since there is only two connections. Again this may cause problems with invoking slow services. This can be avoided by the following technique to increase the default number of connections.

ConfigurationContext configurationContext =
                    ConfigurationContextFactory.createConfigurationContextFromFileSystem(
                            AXIS2_REPOSITORY_LOCATION, AXIS2_CLIENT_CONFIG_FILE);

            MultiThreadedHttpConnectionManager multiThreadedHttpConnectionManager = new MultiThreadedHttpConnectionManager();

            HttpConnectionManagerParams params = new HttpConnectionManagerParams();
            params.setDefaultMaxConnectionsPerHost(20);
            multiThreadedHttpConnectionManager.setParams(params);
            HttpClient httpClient = new HttpClient(multiThreadedHttpConnectionManager);
            configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);

and call serviceClient.cleanupTransport(); after each service invocation.

Changing Axis2 application path

By default Axis2 generates an context path as follows when deployed it in tomcat or other application server.

http://localhost:8080/axis2/services/Version?wsdl

but most of the it is requried to customize this as follows.

http://localhost:8080/myApp/myServices/Version?wsdl

Below steps can be used to achive this.
1. Rename the web app name axis2 to myApp
2. Uncomment and edit the servicePath parameter at the axis2.xml file
myServices
3. Add the following entry to web.xml. Here we need to add this entry rather than chaing existing one since services part is hard coded in some places.

<servlet-mapping>
<servlet-name>AxisServlet</servlet-name>
<url-pattern>/myServices/*</url-pattern>
</servlet-mapping>

How to get a path like this?

http://localhost:8080/myServices/Version?wsdl

This can be done my making this application as ROOT app
1. change the myApp to ROOT

WSO2 Data Services v2.2.0 released

Axis2 lets users to develop web services from the java code. This can either be a POJO where users write the business logic first or a generated code with the WSDL. In whatever form real business logic retrieve data from the data store (most of the time a database) and update it.

Therefore isn't it useful to publish the data stored in permanent storage straight away (I.e using some descriptor files) without doing any coding? WSO2 Data services server let users to publish data by just writing a descriptor file. Currently it can publish data stored in many forms. eg. Databases, text files, excel sheets, google docs etc..
Since WSO2 Data services server build on top of WSO2 carbon platform it inherit all the WS* features such as security, reliable messaging, addressing etc ..

Following are some of the key features:

- Multiple data source support - any relational database accessible via JDBC, CSV, Excel, JNDI bound data sources and Google Spreadsheets
- Web service and REST interfaces for data
- Customizable response format to match your next adaptor's input
- Support for large data sets with little to no impact to the server's memory usage
- Nested queries and federated response from multiple disparate data sources
- Role based content filtering
- Full support for WS-Security, WS-Trust, WS-Policy, WS-SecureConversation and XKMS
- Web based GUI wizard
- Multi transport support - send and receive your data in either HTTP(s), SMTP, JMS and XMPP
- Reliable delivery of data
- Access throttling based on IP/Domain or frequency

WSO2 Data Services 2.2.0 can be downloaded from http://wso2.org/downloads/data-services-server.

Axis2 Authentication

The servlet specification provides four authentication mechanisms for any web application. Therefore any web services engine which receives http requests through a servlet can use those authentication mechanisms. Here is how you can do BASIC and DIGEST authentication with Apache Axis2 deployed under tomcat.

Axis2 war distribution is a standard web application. This war distribution contains a web.xml and within it there is a servlet called AxisServlet which is used to receive the http requests.

Configuring the war distribution.
First the following should add to the web.xml file to protect the AxisServlet from the anonymous access.

<login-config>
<!-- <auth-method>BASIC</auth-method> -->
<auth-method>DIGEST</auth-method>
<realm-name>default</realm-name>
</login-config>

<security-constraint>
<web-resource-collection>
<web-resource-name>Protected Resource</web-resource-name>
<url-pattern>/services/*</url-pattern>
</web-resource-collection>

<auth-constraint>
<role-name>tomcat</role-name>
</auth-constraint>
</security-constraint>

then add the following to the tomcat-users.xml



Now the servlet is protected and Lets see how to provide the credentials when accessing the service. This is in fact done by using the functionality available with the commons http client.

ServiceClient serviceClient = new ServiceClient();
serviceClient.setTargetEPR(new EndpointReference("http://localhost:8080/axis2/services/Version/getVersion"));
serviceClient.getOptions().setAction("urn:getVersion");

HttpTransportProperties.Authenticator authenticator = new HttpTransportProperties.Authenticator();
authenticator.setUsername("tomcat");
authenticator.setPassword("tomcat");

serviceClient.getOptions().setProperty(HTTPConstants.AUTHENTICATE, authenticator);
serviceClient.sendReceive(null);

This request can be send through a tcpmon to understand how this authentication works. First Axis2 client sends a normal request and tomcat server returns an Unauthorized response with the required authentication method.

HTTP/1.1 401 Unauthorized

Server: Apache-Coyote/1.1

WWW-Authenticate: Digest realm="default", qop="auth", nonce="6da725c4d901eee87d2ad49cadbac74a", opaque="37629e27fec9bfaf38063bc3ab65f12d"


After receiving this Axis2 client sends another request with the authentication details.

Minimal jars required for Axis2 1.5

Minimal jars required for Axis2 1.5 is slightly different from Axis2 1.4.1.

Here is a set of jars required with Axis2 1.5 to invoke a service written using Axiom and
to invoke it using direct service client using default axis2.xml file.

axis2-kernel-1.5.jar
axiom-api-1.2.8.jar
axiom-dom-1.2.8.jar
axiom-impl-1.2.8.jar
commons-logging-1.1.1.jar
wsdl4j-1.6.2.jar
XmlSchema-1.4.3.jar
neethi-2.0.4.jar
axis2-transport-local-1.5.jar
axis2-transport-http-1.5.jar
commons-httpclient-3.1.jar
mail-1.4.jar
commons-fileupload-1.2.jar
woden-api-1.0M8.jar
woden-impl-dom-1.0M8.jar
httpcore-4.0.jar
commons-codec-1.3.jar
geronimo-stax-api_1.0_spec-1.0.1.jar
wstx-asl-3.2.4.jar


axis2-jaxws-1.5.jar may be used to avoid some waning messages with the default axis2.xml.