Error while starting cluster: java.lang.RuntimeException: Failed to start Service "Cluster"

Can't start managed servers in a SOA cluster correctly, SOAInfra is in a failed state and see below errors in log file

Oracle Coherence GE 3.6.0.4  (thread=Cluster, member=n/a): Failure to join a cluster for 300 seconds; stopping cluster service.
Oracle Coherence GE 3.6.0.4  (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Error while starting cluster: java.lang.RuntimeException: Failed to start Service "Cluster" (ServiceState=SERVICE_STOPPED, STATE_ANNOUNCE)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:38)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
        at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:637)
        at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)

This maybe caused because of another cluster in same subnet having the same cluster name. To fix this error and start servers correctly go to admin console and click on Managed Server->Server Startup tab. Under arguments field set the below for each server of the cluster.

Let's say the 2 node names are host1.com, host2.com respectively.

On server startup on Node1 set

-Dtangosol.coherence.wka1=host1.com -Dtangosol.coherence.wka2=host2.com -Dtangosol.coherence.localhost=host1.com -Xmanagement:ssl=false,authenticate=false,autodiscovery=true

On server startup of Node2 set

-Dtangosol.coherence.wka1=host1.com -Dtangosol.coherence.wka2=host2.com -Dtangosol.coherence.localhost=host2.com -Xmanagement:ssl=false,authenticate=false,autodiscovery=true

Once done, save changes and restart the managed servers. Now the error should be gone.

Plan.xml Not Getting Updated Across Nodes In SOA 11g Cluster

Recently came across an issue in SOA 11g cluster environment related to Plan.xml.

After creating a data source in Weblogic Admin console, when you update the DB adapter from deployments section, at the "Save Deployment Plan Assistant" screen the data gets stored in the Plan.xml (MW_HOME/ORACLE_SOA1/soa/connectors directory). In a SOA 11g Cluster (lets say 2 node env.) this Plan.xml file gets created on both the nodes first time. However any subsequent updates on the adapter like creation of new datasources or updation to existing ones dont get saved to both nodes. Only the Plan.xml on node where admin server runs gets updated.

This would cause datasource errors/missing JNDI name errors when the environment is load balanced and requests are routed to the 2nd node.

This is more of a cluster configuration issue. The Plan.xml should be kept in a location that is shared storage for all cluster nodes. If this hasn't been done while setting up the cluster then every time adapter updation happens, the Plan.xml only gets updated on Admin server node. In that case the solution is to manually copy the Plan.xml from admin server node to other nodes and restart the managed servers.

Installing SOA 11g Cluster on Weblogic

SOA 11g cluster setup on Weblogic is a lot easier than SOA 10g clustering on OC4J for sure..Tried to document the steps sequentially, will add screenshots later.

This is a 2 Node cluster with Admin server running on 1 node and Managed server running on both nodes.

1. Download all the installables from OTN for the SOA version 11.1.1.4.

       jrockit-jdk1.6.0_24-R28.1.3-4.0.1-linux-x64.bin

       wls1034_generic.jar

       ofm_rcu_linux_11.1.1.4.0_disk1_1of1.zip

       ofm_soa_generic_11.1.1.4.0_disk1_1of2.zip

       ofm_soa_generic_11.1.1.4.0_disk1_2of2.zip

2. Run the RCU and create the schemas for the SOA Cluster. (xxx_SOAINFRA, xxx_MDS etc..)

3. Install Jrockit on both nodes.

4. Install weblogic 10.3.4 on both nodes

        java -Xms1024M -Dspace.detection=false -jar wls1034_generic.jar

5. Install SOA 11.1.1.4 on both nodes.

       a. specify oraInventory path

       b. skip software updates

       c. specify installation directory

       d. specify the DB details

6. Create the domain on Node1. Run config.sh in $ORACLE_HOME/common/bin

      a. Under "Select Optional Configuration" select 

               Managed Servers, Clusters and Machines & 

               Deployments and Services

     b. Add the 2 managed server names and their listen ports

     c. Configure cluster

     d. Add managed servers to the cluster

     e. Configure Machines (Add the 2 server names under Unix Machine tab)

     f. Assign servers to machines.

     g. Complete the domain creation....

7. Pack the SOA domain from Node1 using below command

cd $WL_HOME/common/bin

./pack.sh -managed=true -domain={path to SOA domain} -template=soadomaintemplate.jar -template_name=soa_domain_template

8. Copy the jar file to Node2 and run the unpack command there.

cd $WL_HOME/common/bin

./unpack.sh -domain={path to SOA domain} -template=soadomaintemplate.jar

9. On each host create boot properties.

cd {path to SOA domain}

vi boot.poperties

username=weblogic

passsword=welcome1

cp boot.properties servers/AdminServer/security

10. Start Admin Server on host1 and disable the host name verification for admin and managed servers. (SSL tab->Advanced). Restart the Admin Server.

11. Start Node manager on both nodes to create initial script file. Then stop the node manager, edit the $WL_HOME/common/nodemanager/nodemanager.properties to set below and restart Node managers.

StartScriptEnabled=true

            StopScriptEnabled=true

12. Now you can login to the Admin console and start the managed servers.

13. Incase you have a Load balancer, configure the same to route request to the 2 nodes. The composites being deployed on the cluster can point to the LBR URL. (make sure end point URLs in composite.xml point to this). Also set the Server URL and Server Callback URL as shown in below screenshot.

14. Coherence comes as part of the SOA suite and SOA Clusters in 11g use coherence for communicating between nodes (similar to JGroups in 10g). Without coherence setup the deployment will not get distributed across all servers.

For unicast communication mode, we need additional coherence properties to be set. These have been explained in below Oracle doc.
http://download.oracle.com/docs/cd/E15523_01/core.1111/e12036/extend_soa.htm#CHDEAFJH

The startWeblogic.sh script on Node1 will have below properties set for Coherence to work.

EXTRA_JAVA_PROPERTIES="-Dtangosol.coherence.wka1=Node1 hostname -Dtangosol.coherence.wka2=Node2 hostname -Dtangosol.coherence.localhost=Node1 hostname"

Similarily for Node2...

That's it ! The WLS cluster should be ready to use now. 

Configuring FTP Adapter in SOA 10g Cluster for High Availability

If you have a 2 Node cluster environment for SOA 10g, it is essential that the FTP adapter is configured in an Active/Passive manner. Otherwise you may run into situations where both nodes are trying to read the same file from the remote FTP server and this will lead to duplicate files entering the system.

The reason for duplicates is because FTP adapter on each node will maintain its own control file(locally) where it stores the last read time of the file. So if a file has already been read by Node1, there are chances that after server restart the same file may be picked by Node2 as well because the time maintained in control files is out-of-sync on both nodes.

There are couple of good articles published by Oracle for this specific configuration. The base idea of the solution is to have the control file on a shared folder which is accessible by both nodes.

Step1:
=====
Make sure the FTP adapters are configured in Singleton mode i.e BPEL clusterName value specified in $ORACLE_HOME/bpel/system/config/collaxa-config.xml should be different from the Adapters clusterGroupId property set inside bpel.xml of the BPEL project.

Also the Multicast host and port in jgroups-properties.xml file should be same on both nodes.

Step2:
=====
Once step1 has been ensured, you should create a folder on a shared file system. Either use an external share storage or you can create a shared directory on one node and use NFS share to mount it on the other node. Either ways make sure the folder has write permissions from both nodes. This folder will store the control files.

Next backup and edit the $ORACLE_HOME/bpel/system/service/config/pc.properties file on each node and set the below property oracle.tip.adapter.file.controldirpath to the shared folder name

Restart the servers for the change to take effect and test the scenario.