Issue TOC Print full article Print this page Download as PDF E-Mail this URL Discuss this article Flame the author     In this article Introduction Novell Extend Application Server 4.0 Other Products Reviewed Defining Web Services Performance How We Tested Web Services Report Card

To evaluate the performance of Web services platforms, we considered three things:

1. Number of messages per second successfully processed

2. Consistent processing of messages under increased load

3. Differences in performance between processing of the DOC/LIT and RPC/ENC encoding models

The number of messages per second a product can process is dependent on the XML parser employed. The SOAP request must be decoded by the parser and then handed off to the appropriate Java method. An inefficient XML parser will degrade performance significantly as the size of the request increases, because there is more XML to decode.

Also entering the equation is the way in which a platform marshals and unmarshals arguments--in English, the way a platform transforms an XML element into a Java argument and vice versa. The way in which this is handled varies according to implementation. For example, Cape Clear's implementation handles DOC/LIT differently from the way it does RPC/ENC. In the DOC/LIT model, every argument--even primitive data types, such as an integer--is encoded into a corresponding Java class. That class may be simple, consisting of a single piece of data, or complex, a complete object that may comprise individual objects within it. In the RPC/ENC model, the arguments are often passed to the Web service as a primitive, which can increase performance.

Some platforms, such as those from Novell or Systinet, use the DOC/LIT model of processing even for RPC/ENC, which can degrade performance because of the additional instantiation of objects which are, in some regard, spurious. BEA provides an extra level of abstraction that allows elements within a SOAP request to be mapped to Java methods or arguments that do not necessarily share the same name. This is a definite value-add, making BEA's implementation more flexible, but it does add additional time to the process of marshaling and unmarshaling arguments.

Because Web services are in their infancy, it is not expected that the implementation of a Web service will result in it being inundated with thousands of messages per second. If that happens, we suggest taking advantage of clustering and load balancing, either via the products where available or through an external solution. Because of the real-world use of Web services, we were more concerned with the products' ability to maintain consistent performance as the number of concurrent users increased. In other words, we were examining the ability of each product to scale and were less concerned with the raw message throughput capability of the products under test.

For example, we were impressed with both BEA's and Iona's ability to remain consistent under load, even though the number of messages each processed per second appeared to be low as tested under an out-of-the-box configuration. Sun's product showed a high processing capability while maintaining consistency, while the other products decreased in their processing capabilities as the load increased.

Take advantage of the tuning suggestions made by your vendor of choice. Changes to the Java Virtual Machine parameters, such as memory or threads, and modifications of activation policies, log levels and connection-level settings can drastically increase the performance of an application server, so consider tuning part of the deployment process.