Sunday, August 21, 2011

Tuning SOA 11g in a nutshell


Tuning SOA 11g can be done in a few quick steps, this will tackle most common performance issues at infrastructure level. On the other hand, most performance issues are in the application (composite) self. If you design a bad application, the application will not perform well.

I did my tuning on the following subjects:
  • JVM Memory
  • JVM Garbage Collection
  • Datasources
  • Threads
  • EJB
  • Database

JVM Memory
Make sure you have set the minimal and maximal heap size the same size. The size should be at least 1024MB.
Make sure the maxpermsize parameter is set to at least 512MB. In most cases requests in the SOA world are relative small, this means you could shrink the size of the stack for one Java thread.

Example:

-server -Xms1280m -Xmx1280m -XX:MaxPermSize=512m -Xss128k


JVM Garbage Collection
Running your application server with one CPU, you would not expect enhanchments by setting garbage collection to parallel. On VMWare we saw some significant improvement when we set this option even the virtual machine has one CPU. I think it has to do with the hyper-visor and the ESX Server. Somehow the virtual machine benefits from the ESX environment.

Example:

-XX:-UseParallelGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing  

JMX
Setting some JMX options will not improve the performance, but it gives you insight information on the Java VM. If you not using JROckit your could set the following options and use VisualVM to view the Java engine.

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=7004 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false 

http://orasoa.blogspot.com/2010/04/monitoring-your-soa-jvm.html



Otherwise your JRockit Mission Control to get more information out of your Java VM.

Data-sources
Using the correct settings for all the connections to your database is essential. The SOA Suite is using  a lot of data-sources, while the OSB is using one data-source for the reports. Measure your connections to the database while running a load test on your environment. Based on these measurements you can predict the correct configuration.

The following table shows an example of the settings your should tune. Note that the values are based on my measurements on a particular system. These settings should be applied via the Weblogic Console or through Weblogic scripting.


DatasourcesEDNDataSourceEDNLocalTxDataSourcemds-owsmmds-soaOraSDPMDataSourceSOADataSourceSOALocalTxDataSource
Hostdb-hostdb-hostdb-hostdb-hostdb-hostdb-hostdb-host
Port1521152115211521152115211521
SIDorclorclorclorclorclorclorcl
TypeXAnon-XAnon-XAnon-XAnon-XAXAnon-XA
Initial Capacity:1111122
Maximum Capacity:10101010102020
Capacity Increment:2222222
Statement Cache Size:10101010101010
Test Connections On ReserveTRUETRUETRUETRUETRUETRUETRUE
Test Frequency:180180180180180180180
Seconds to Trust an Idle Pool Connection:30303030303030
Shrink Frequency:300300300300300300300
Connection Creation Retry Frequency:30303030303030
Inactive Connection Timeout:30303030303030
Maximum Waiting for Connection:2147483647214748364721474836472147483647214748364721474836472147483647
Connection Reserve Timeout: 30303030303030
Statement Timeout:-1-1-1-1-1-1-1
Enviroment
oracle.net.CONNECT_TIMEOUT100000100000100000100000100000100000100000
Set XA Transaction Timeout:TRUETRUE
XA Transaction Timeout:00
XA Retry Duration:300300
XA Retry Interval:6060

Threads
Do not touch the default threading model of Weblogic, unless you known what your are doing. Weblogic is using an automated mechanism for tuning his own work. WebLogic Server prioritizes work and allocates threads based on an execution model that takes into the managed servers parameters and actual run-time statistics (performance and throughput).

Within the SOA Suite 11g, you can still tune the threads of the BPEL engine itself. Keep in meind that these thread will use a database connection from the pool. It is not said that the total number of BPEL threads results in the same amount of connection to the database. You can set these values via Oracle Enterprise manager

dspInvokeThread10
dspEngineThreads15
dspSystemThreads2
synMaxWaitTime150

EJB
The  soa-infra application contains all the EJB objects that are responsible for SOA11g (BPEL, Workflow, BPM, Sensors, Mediators, etc..). Each of these EJB objects has his own time out. By default it is 300 seconds. When you have long running process, also known as not optimal designed and implemented SOA Composites ;-) , you could run into time-out errors that could slow down the whole SOA environment. Therefore your should design and implement efficicient and optimal composites.

To change the time-out of the EJB's, execute the following steps.
  1. Stop all the managed servers
  2. Apply the EJB time-out changes
  3. Start the managed servers
or
  1. Stop all the managed servers
  2. Undeploy the soa-infra application
  3. Activate changes
  4. Deploy the soa-infra application from $ORACLE_HOME/soa/applications/soa-infra-wls.ear, as 'soa-infra' and copy application to all servers in the cluster
  5. Activate changes
  6. Redeploy soa-infra application with configuration file (Plan.xml) with all the new EJB time-ou settings.
  7. Activate changes
  8. Start soa-infra applicatio to server all requests.
Here is the configuration file for the soa-infra application, Plan.xml file:
Plan.xml:
<?xml version='1.0' encoding='UTF-8'?>
<deployment-plan xmlns="http://xmlns.oracle.com/weblogic/deployment-plan" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/deployment-plan http://xmlns.oracle.com/weblogic/deployment-plan/1.0/deployment-plan.xsd">
  <application-name>applications</application-name>
  <variable-definition>
    <variable>
      <name>TransactionDescriptor_transTimeoutSeconds_BPELEngineBean</name>
      <value>1800</value>
    </variable>
    <variable>
      <name>TransactionDescriptor_transTimeoutSeconds_BPELDeliveryBean</name>
      <value>1800</value>
    </variable>
    <variable>
      <name>TransactionDescriptor_transTimeoutSeconds_BPELActivityManagerBean</name>
      <value>1800</value>
    </variable>
    <variable>
      <name>TransactionDescriptor_transTimeoutSeconds_BPELServerManagerBean</name>
      <value>1800</value>
    </variable>
    <variable>
      <name>TransactionDescriptor_transTimeoutSeconds_BPELProcessManagerBean</name>
      <value>1800</value>
    </variable>
    <variable>
      <name>TransactionDescriptor_transTimeoutSeconds_BPELInstanceManagerBean</name>
      <value>1800</value>
    </variable>
    <variable>
      <name>TransactionDescriptor_transTimeoutSeconds_BPELFinderBean</name>
      <value>1800</value>
    </variable>
    <variable>
      <name>TransactionDescriptor_transTimeoutSeconds_BPELDispatcherBean</name>
      <value>1800</value>
    </variable>
    <variable>
      <name>TransactionDescriptor_transTimeoutSeconds_BPELSensorValuesBean</name>
      <value>1800</value>
    </variable>
  </variable-definition>
  <module-override>
    <module-name>soa-infra-wls.ear</module-name>
    <module-type>ear</module-type>
    <module-descriptor external="false">
      <root-element>weblogic-application</root-element>
      <uri>META-INF/weblogic-application.xml</uri>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>application</root-element>
      <uri>META-INF/application.xml</uri>
    </module-descriptor>
    <module-descriptor external="true">
      <root-element>wldf-resource</root-element>
      <uri>META-INF/weblogic-diagnostics.xml</uri>
    </module-descriptor>
    <module-descriptor external="true">
      <root-element>persistence-configuration</root-element>
      <uri>bpel_persistence_weblogic.jar!/META-INF/persistence-configuration.xml</uri>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>persistence</root-element>
      <uri>bpel_persistence_weblogic.jar!/META-INF/persistence.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>ejb_ob_engine_wls.jar</module-name>
    <module-type>ejb</module-type>
    <module-descriptor external="false">
      <root-element>weblogic-ejb-jar</root-element>
      <uri>META-INF/weblogic-ejb-jar.xml</uri>
      <variable-assignment>
        <name>TransactionDescriptor_transTimeoutSeconds_BPELEngineBean</name>
        <xpath>/weblogic-ejb-jar/weblogic-enterprise-bean/[ejb-name="BPELEngineBean"]/transaction-descriptor/trans-timeout-seconds</xpath>
      </variable-assignment>
      <variable-assignment>
        <name>TransactionDescriptor_transTimeoutSeconds_BPELDeliveryBean</name>
        <xpath>/weblogic-ejb-jar/weblogic-enterprise-bean/[ejb-name="BPELDeliveryBean"]/transaction-descriptor/trans-timeout-seconds</xpath>
      </variable-assignment>
      <variable-assignment>
        <name>TransactionDescriptor_transTimeoutSeconds_BPELActivityManagerBean</name>
        <xpath>/weblogic-ejb-jar/weblogic-enterprise-bean/[ejb-name="BPELActivityManagerBean"]/transaction-descriptor/trans-timeout-seconds</xpath>
      </variable-assignment>
      <variable-assignment>
        <name>TransactionDescriptor_transTimeoutSeconds_BPELServerManagerBean</name>
        <xpath>/weblogic-ejb-jar/weblogic-enterprise-bean/[ejb-name="BPELServerManagerBean"]/transaction-descriptor/trans-timeout-seconds</xpath>
      </variable-assignment>
      <variable-assignment>
        <name>TransactionDescriptor_transTimeoutSeconds_BPELProcessManagerBean</name>
        <xpath>/weblogic-ejb-jar/weblogic-enterprise-bean/[ejb-name="BPELProcessManagerBean"]/transaction-descriptor/trans-timeout-seconds</xpath>
      </variable-assignment>
      <variable-assignment>
        <name>TransactionDescriptor_transTimeoutSeconds_BPELInstanceManagerBean</name>
        <xpath>/weblogic-ejb-jar/weblogic-enterprise-bean/[ejb-name="BPELInstanceManagerBean"]/transaction-descriptor/trans-timeout-seconds</xpath>
      </variable-assignment>
      <variable-assignment>
        <name>TransactionDescriptor_transTimeoutSeconds_BPELFinderBean</name>
        <xpath>/weblogic-ejb-jar/weblogic-enterprise-bean/[ejb-name="BPELFinderBean"]/transaction-descriptor/trans-timeout-seconds</xpath>
      </variable-assignment>
      <variable-assignment>
        <name>TransactionDescriptor_transTimeoutSeconds_BPELDispatcherBean</name>
        <xpath>/weblogic-ejb-jar/weblogic-enterprise-bean/[ejb-name="BPELDispatcherBean"]/transaction-descriptor/trans-timeout-seconds</xpath>
      </variable-assignment>
      <variable-assignment>
        <name>TransactionDescriptor_transTimeoutSeconds_BPELSensorValuesBean</name>
        <xpath>/weblogic-ejb-jar/weblogic-enterprise-bean/[ejb-name="BPELSensorValuesBean"]/transaction-descriptor/trans-timeout-seconds</xpath>
      </variable-assignment>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>ejb-jar</root-element>
      <uri>META-INF/ejb-jar.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>fabric-ejb.jar</module-name>
    <module-type>ejb</module-type>
    <module-descriptor external="false">
      <root-element>weblogic-ejb-jar</root-element>
      <uri>META-INF/weblogic-ejb-jar.xml</uri>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>ejb-jar</root-element>
      <uri>META-INF/ejb-jar.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>hw_services_wls_ejb.jar</module-name>
    <module-type>ejb</module-type>
    <module-descriptor external="false">
      <root-element>weblogic-ejb-jar</root-element>
      <uri>META-INF/weblogic-ejb-jar.xml</uri>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>ejb-jar</root-element>
      <uri>META-INF/ejb-jar.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>sdpmessagingclient-ejb.jar</module-name>
    <module-type>ejb</module-type>
    <module-descriptor external="false">
      <root-element>weblogic-ejb-jar</root-element>
      <uri>META-INF/weblogic-ejb-jar.xml</uri>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>ejb-jar</root-element>
      <uri>META-INF/ejb-jar.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>b2b_engine_wls.jar</module-name>
    <module-type>ejb</module-type>
    <module-descriptor external="false">
      <root-element>weblogic-ejb-jar</root-element>
      <uri>META-INF/weblogic-ejb-jar.xml</uri>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>ejb-jar</root-element>
      <uri>META-INF/ejb-jar.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>fabric.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="false">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>hw_services.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170529</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>taskservice.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170530</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>taskmetadataservice.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170531</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>taskqueryservice.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170532</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>taskreportservice.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170533</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>IdentityService.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170533</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>usermetadataservice.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170528</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>runtimeconfigservice.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170527</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>taskevidenceservice.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170526</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>compositemetadataservice.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170525</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>b2bws.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170524</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>weblogic-webservices</root-element>
      <uri>WEB-INF/weblogic-webservices.xml</uri>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>webservices</root-element>
      <uri>WEB-INF/webservices.xml</uri>
    </module-descriptor>
    <module-descriptor external="true">
      <root-element>webservice-policy-ref</root-element>
      <uri>WEB-INF/weblogic-webservices-policy.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>b2b_starter_wls.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170509</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>agmetadataservice.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170508</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>agqueryservice.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170499</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>agadminservice.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="true">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <hash-code>1278926170497</hash-code>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <config-root>/opt/weblogic/Oracle/Middleware/KIM_SOA/soa/applications/plan</config-root>
</deployment-plan>

Set the following time-out of the EJB's:

BPEL/EJB properties Timeout
BPELActivityManagerBean 1800
BPELDeliveryBean 1800
BPELDispatcherBean 1800
BPELEngineBean 1800
BPELFinderBean 1800
BPELInstanceManagerBean 1800
BPELProcessManagerBean 1800
BPELSensorValuesBean 1800
BPELServerManagerBean 1800

Verify the the global transaction time out on all the managed servers is larger then the time out of the EJB. Set the global transaxtion time-out, JTA, to 3600.

Database 
To tune the dabatase, make sure that you follow the requirements of the SOA Suite 11g, make sure you can create enough process and session to the database. The database must be tuned, so the it has enough memory for caching, PGA, SGA memory should be adequate.

Make sure your have the purging scripts place. And verify if you have added an index to the SOA dehydration tables.

References



Thursday, August 18, 2011

OSB using BLOB data with DBAdapter results in ora-2049

When you want to poll data in the OSB, you will use the DBAdapter. In a 4-node production cluster we saw some strange errors periodically. These errors came up in the log file of any of the application servers:

Caused By: oracle.tip.adapter.sa.impl.fw.ext.org.collaxa.thirdparty.apache.wsif.WSIFException: servicebus:/WSDL/EmailService/BusinessService/WijzigEmailStatus/wijzigStatus [ wijzigStatus_ptt::wijzigStatus(InputParameters,OutputParameters) ] - WSIF JCA Execute of operation 'wijzigStatus' failed due to: Stored procedure invocation error.
Error while trying to prepare and execute the "schema/package/procedure" API.
An error occurred while preparing and executing the "schema/package/procedure" API.
Cause: java.sql.SQLSyntaxErrorException: ORA-02049: timeout: distributed transaction waiting for lock
ORA-06512: at "schema/package", line 113
ORA-06512: at line 1

; nested exception is:
        BINDING.JCA-11811
Stored procedure invocation error.
Error while trying to prepare and execute the "schema/package/procedure" API.
An error occurred while preparing and executing the "schema/package/procedure" API. 
Cause: java.sql.SQLSyntaxErrorException: ORA-02049: timeout: distributed transaction waiting for lock
ORA-06512: at "schema/package", line 113
ORA-06512: at line 1  

The DBAdapter was polling data from database, this was a TopLink definition based on four tables. The ProxyService that was started an in the request pipeline the message was routed , to the bussiness service. While in the response pipeline the one of these four tables was updated.

The issue is related not related in high traffic on the system. It is all about using the DBAdapter/Toplink poller mechanism. The adapter is polling data and one of the columns is a BLOB column.

If data in this column is large, the DbAdapter will take some time to process this information to the service. While the set records are locked other services wants to update this data. This will result in the Oracle error.

To resolve this issue, large (blob) data should be processed as a stream in OSB. This can easily be done by enabling the 'Content Streaming' option for this (polling) proxy service.



Remarks from development on this article:
The exception in question is just a transaction timeout. That can be avoided by
  • shrinking MaxTransactionSize on inbound so that the transaction started by DbAdapter polling is shorter
  • shrinking MaxRaiseSize on inbound so that any transactions started by OSB will be shorter
  • increasing transaction timeout
If the problem is that inbound polling is locking a row for a long time that another process is trying to lock, that could be because inbound has configured distributed polling, where the records are locked on select.  The default behaviour is to only lock records on update, which should be independent of the time it takes to process the BLOB.
My remarks:
Shrinking the MaxTransactionSize to 1, could solve the issue, but if you need to handle ten-thousand or more messages with large binary data, the whole polling process will be slow. I am using the MaxTransactionSize with value 10.



References: