SOA performance issue: let’s fix it

All the environments are running on Exalogic/exadata with OTD (oracle Traffic Director) configured for load balance. On a regular time we found out that SOA server generate the following errors:

• Crashed with out of Memory error:

“””Error> <oracle.soa.bpel.engine> <BEA-000000> <This
error contained the exceptions thrown 
                by the underlying routing system. Root cause
:java.lang.OutOfMemoryError:
               allocLargeObjectOrArray”””

• Response time of the SOA server is very slow
• No response at all
• Enable to invoke endpoint URI “XXXXXXXXXXX:8002/services/”successfully due to:soap.SOAPException: Bad response:504 Gateway Timeout
• A lot of StuckThreads on the SOA servers
   

OTD issue

This is an OTD problem. Most of SOA suite requests are POST requests. In other words: every request between composites will generate a NEW HTTP connection at the remote SOA server. This will cost a lot of overhead connection. Eventually, with sufficient load, the pile up will be so great that SOA ends up stuck in a Garbage collection loop and will either throw OutOfMemoryError or slow response.

Solution: special setting in OTD

The key to solve this problem is on the OTD side. There is a setting in the OTD which will help dramatically to solve this problem & improve the SOA’s performance: ‘Always use keep-alive’.

By default, this parameter is not enabled. By enable this parameter OTD will cache and reuse the same HTTP connection for multiple requests (it actually has a small pool of cached connections). This parameter needs to be added to  the Origin Servers.

More helpful parameters

Besides ‘Always use keep-alive‘, there are some other parameters which are helpful to increase performance on the SOA environment. I’ll mention some of them.

• Enable keep-alive
  This parameter indicates whether the Oracle Traffic Director virtual server should attempt to use persistent connections to the origin server or create a new connection for each request. It is enabled by default.
   
• Keep-alive timeout
  This parameter specifies the maximum duration, in seconds, for which a persistent connection can be kept open. The default timeout duration is 29 seconds.
   
• Idle timeout
  This parameter specifies the maximum duration, in seconds, for which a connection to the origin server can remain idle. The default duration is 300 seconds.
   
• Proxy buffer size
  This parameter specifies the size of the buffer in which Oracle Traffic Director stores data received from the origin server, before sending the data to the client. The larger the buffer, the lower is the number of write system calls. The default size of the proxy buffer is 16 kilobytes.
   
• Always use keep-alive
  This parameter indicates whether the Oracle Traffic Director virtual server can reuse existing persistent connections to origin servers for all types of requests. If this parameter is not enabled (default), the Oracle Traffic Director virtual server attempts to use persistent connections to the origin server only for the GET, HEAD, and OPTIONS request methods.
   

Steps to implement this change

Changes on the OTD can be executed through administration console or CLI (command-line). To add a user CLI, we will need to follow the steps below:

• Add the SOA-obj.conf file. This file is located under INSTANCE_HOME/net-config_name/config. In my example I added the following lines:

<Object name=”default-route”>

ObjectType fn=”http-client-config” keep-alive-timeout=”31″ always-use-keep-alive=”true” buffer-size=”32768″ timeout=”360″

Route fn=”set-origin-server” origin-server-pool=”SOA-Server-Pool”

</Object>

• Save the file
• Run pull-config to update configuration store on Admin server & to push new configuration on all instances

#tadm> pull-config –config=config_name node

A simple way to achieve this is by using Administration Console.

Now, login

Login to OTD console: 

Configuration-> select SOA-> Virtual Servers-> Routes-> default-route

Chance settings

Advanced Settings under tab Configuration for Connections with Origin Servers. Change the following settings:

• Enable ‘Always use keep-alive’ (default disable)
• Increase Buffer Size to 32768, default ( 16kb)
• Idele Timeout 360, default (300 seconds)

And… you’re done!

Save and deploy changes.

Prevent the overhead

This will prevent the overhead from creating a TCP/IP connection for every request hence improves the performance dramatically. And also, since there will be a smaller number of the connections to the origin server (for example SOA, OSB, PeopleSoft, EBS & WebCenter Portal), it will reduce the chance of build up stale connections on the remote server.

More information

Want more details? Please click here.

About the author Hamdy Al-Sebaey

More posts by Hamdy Al-Sebaey