SOA 11g: Managed server error "weblogic.messaging.kernel.KernelException"

Recently came across an issue where the Managed servers were failing with below error. Even though servers start and come to running mode, but the JMS queues are not showing up in admin console under JMS Servers->Monitoring -> Active Destinations

<Error> <JMS> <BEA-040123> <Failed to start JMS Server "JMS_DEV_SERVER1" due to weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Ignoring 2PC record for sequence=4706 queue=57 because the element cannot be found.

weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Ignoring 2PC record for sequence=4706 queue=57 because the element cannot be found

        at weblogic.jms.backend.BackEnd.open(BackEnd.java:1008)

        at weblogic.jms.deployer.BEAdminHandler.activate(BEAdminHandler.java:200)

        at weblogic.management.utils.GenericManagedService.activateDeployment(GenericManagedService.java:239)

        at weblogic.management.utils.GenericServiceManager.activateDeployment(GenericServiceManager.java:131)

        at weblogic.management.internal.DeploymentHandlerHome.invokeHandlers(DeploymentHandlerHome.java:632)

        Truncated. see log file for complete stacktrace

This issue was after effect of the JMS file persistence store hitting 100% space utilization. As a result of which it got corrupted and was causing the JMS servers to fail while starting/activating.

To workaround this error, go to the Persistence File Store location (local/Shared SAN storage) and rename the existing *.DAT files to _bkp (say). Then go ahead and restart the managed servers. Now servers should come up fine without any issues/errors.

Oracle Metalink Note 1473826.1 "FileStore getting corrupted and WLS is unable to initialize JMS leading to BEA-040123 " suggests applying a Weblogic patch for Bug13900234. I haven't tried applying this patch but if issue persists inspite of workaround you can try this option as well.

Automating SOA 11g Admin tasks using WLST

WLST (Weblogic Scripting Tool) is a powerful feature for Weblogic administrators who want to automate most of their day to day operational tasks. This was initiated back in the BEA Weblogic 8.1 days for standalone weblogic server administration. And since Oracle SOA Suite 11g is now based on Weblogic server, this tool has become handy for SOA suite administrators.

There are lot of good learning materials available online for people getting started with WLST apart from the standard Oracle documentation. Few noteworthy ones are mentioned below. These links provide some nice examples with sample code which you can use based on your needs.

• Middleware Magic
• WLST By Examples

With this post I just wanted to give an overview on the different use cases where WLST can be used and how it eases lot of the operational tasks for SOA suite admins.

Common Use Cases 

• Monitoring STUCK threads/Hogging threads 
• JMS thread pool monitoring
• Retiring/Activating SCA composites
• Enabling/Disabling OSB proxy/business services
• Deploying SCA/OSB code
• Deploying/Updating other resources like JMS queues, JDBC datasources etc.
• Pausing/Resuming consumption/production on JMS queues
• Moving messages between JMS servers
• Generating Thread dumps and sending email alerts.

The main thing about working with WLST is understanding of how to navigate the configuration/runtime management beans(MBeans) and how to change the MBean attribute values.

Weblogic MBeans

Weblogic MBean API Reference

You can either directly connect to the WLST command prompt and execute the necessary commands after connecting to the runtime environment. An example shown below for retiring a SCA

  cd mw_home/Oracle_SOA1/common/bin
  ./wlst.sh

  Initializing WebLogic Scripting Tool (WLST) ...
 
  Welcome to WebLogic Server Administration Scripting Shell
 
  Type help() for help on available commands
 
  wls:/offline> connect ("username","password","t3://admin_hostname:port")
  
  Connecting to t3://admin_hostname:port with userid username ...
  Successfully connected to Admin Server 'AdminServer' that 
  belongs to domain 'SOA_DOMAIN'.
 
  Warning: An insecure protocol was used to connect to the
  server. To ensure on-the-wire security, the SSL port or
  Admin port should be used instead.

  wls:/SOA_DOMAIN/serverConfig> sca_retireComposite("listen address",
  "username","password","sca name","sca version",partition="partition_name")

The other way to execute is by writing WLST scripts (.py extension) which are similar in structure to Python language and include a lot of Python as well as Jython (Java implementation of python) libraries. Next you write some shell script (.sh) which invokes these .py files and can schedule the same on the crontab at specific intervals.

The shell script (.sh) basically sets the environment (CLASSPATH and PATH) by calling setWLSEnv.sh and then running the WLST script by calling it as below.

   java weblogic.WLST xxxx.py

Looping In XSL

XSL is a functional language and not a programming language !! XSL variables are not actually variables, once set you cannot change their value !! Why is a for loop not working inside XSL !!

I hope many of us have encountered these questions while using XSL. I have come across many scenarios where I had to create a XSL to iterate through a XML and generate some output but looping is tricky inside XSL. Sure you can run a while loop inside BPEL and get your values assigned but that's clumsy.

Below is some sample code snippet which can be extended based on your requirements and should come handy when trying to loop inside XSL.

  <xsl:template match="/">
  <root>
     <xsl:call-template name="loop">
         <xsl:with-param name="i" select="1" /> 
         <xsl:with-param name="count" select="//code to find the counter for loop" /> 
     </xsl:call-template>
  </root>
  </xsl:template>

  <xsl:template name="loop">
     <xsl:param name="i" /> 
     <xsl:param name="count" /> 
     <xsl:if test="$i <= $count">
        //your code here.You can use the predicate [position() = $i] for correct assignments.
           <xsl:call-template name="loop">
           <xsl:with-param name="i" select="$i + 1" /> 
           <xsl:with-param name="count" select="$count" /> 
           </xsl:call-template>
     </xsl:if>
  </xsl:template>

Hope this helps..

Capturing SOAP Headers/Envelope in SOA 11g

In order to manipulate SOAP Headers in SOA composites you can make use of the bpelx:inputHeaderVariable and bpelx:outputHeaderVariable attributes in invoke activity. However if you want to see the whole SOAP message (envelope with header, body and other parts) its not available out-of-the-box on the EM Console or in logs. In order to log/audit these web service conversations by a SOA composite you can follow below steps:

1. Attach a management policy to your webservice binding named oracle/log_policy

2. Redeploy your code and now when a webservice call is made, navigate to the below log directory to check for the SOAP messages
MW_HOME/user_projects/domains/domain_name/servers/managed_server_name/logs/owsm/msglogging/diagnostic.log
This log would contain both the SOAP request as well as response and you can view the entire SOAP envelope.

Creating Read Only User In Weblogic(SOA 11g)

In production environments, usually requirement is to change the default Weblogic password (to restrict it just to System Administrators) and create some kind of Read Only user for the Weblogic console. A colleague of mine documented these steps recently and hope these help anyone trying to perform this in QA/PROD environments.

1. Guest User Creation
Login to console as weblogic user and click on Security Realms -> myrealm ->Users and Groups

Click on New, enter Name & Password and then OK.

Click on the user which was created (guestuser) and below step ensures that we provide read only access
 to same. Click on Groups tab, choose Monitors group from left pane to right pane. Click on save. Once
    done logout from console (as weblogic) and login with new user to check access.

2. Weblogic Password Change
Login to console as weblogic user and click on Security Realms -> myrealm ->Users and Groups. Click on Weblogic and click on Passwords tab and update the new password and Save.

3. Node Manager Password Change
Login to console and click on Domain, click on Security ->  Advanced.

Update the password for the Node Manager user (weblogic) and click on Save.

4. Updating Weblogic and Node Manager passwords in Linux

     Login to fusion server using putty and navigate to the below directory.
·     Go to %domain_home%/config/nodemanager
·     Open nmpassword_properties
·     The file will hold encrypted values, replace all content with:
                  username=weblogic
                  password=new password
·     Restart the NodeManager
·     Check nmpassword_properties file for encrypted values
   Repeat same steps in its a multi node cluster environment.

 Changing the weblogic security file :
 Go to: %domain_home%/servers/{Managed Server}/security
 Open the boot.properties file
 Change the values to
              username=weblogic
              password=new password
  Perform same steps on other nodes and Restart Managed Servers.

Error while starting cluster: java.lang.RuntimeException: Failed to start Service "Cluster"

Can't start managed servers in a SOA cluster correctly, SOAInfra is in a failed state and see below errors in log file

Oracle Coherence GE 3.6.0.4  (thread=Cluster, member=n/a): Failure to join a cluster for 300 seconds; stopping cluster service.
Oracle Coherence GE 3.6.0.4  (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Error while starting cluster: java.lang.RuntimeException: Failed to start Service "Cluster" (ServiceState=SERVICE_STOPPED, STATE_ANNOUNCE)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:38)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
        at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:637)
        at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)

This maybe caused because of another cluster in same subnet having the same cluster name. To fix this error and start servers correctly go to admin console and click on Managed Server->Server Startup tab. Under arguments field set the below for each server of the cluster.

Let's say the 2 node names are host1.com, host2.com respectively.

On server startup on Node1 set

-Dtangosol.coherence.wka1=host1.com -Dtangosol.coherence.wka2=host2.com -Dtangosol.coherence.localhost=host1.com -Xmanagement:ssl=false,authenticate=false,autodiscovery=true

On server startup of Node2 set

-Dtangosol.coherence.wka1=host1.com -Dtangosol.coherence.wka2=host2.com -Dtangosol.coherence.localhost=host2.com -Xmanagement:ssl=false,authenticate=false,autodiscovery=true

Once done, save changes and restart the managed servers. Now the error should be gone.

Session Management/Pooling While Integrating SOA With Siebel

Oracle SOA Integrations with Siebel require some kind of session management/pooling to be taken care of. In most common scenarios you maintain Siebel sessions from SOA layer by opening session, making subsequent calls to Siebel webservices and closing the session explicitly. There are other cases also where you may want do some session pool management within SOA by storing the session details (say custom DB table) and maintaining a count of the number of open sessions.

In this post I will cover the common approach of handling siebel sessions in Integration, it follows 3 basic steps:

1. Call a Siebel Session WSDL which supports some kind of ping method/operation and pass username, password and session type ("stateless") in SOAP Header to open a session on Siebel side. Siebel will respond back with a Session Token.

2. Use the session token along with session type ("stateless") while making subsequent call to Siebel wsdl for successfully connecting to the opened session.

3. Finally to close the session explicitly, send session token along with session type as "None".

A sample SOAP Header for request sent to Siebel is as below:

 <soap:header xmlns:wsa="http://www.w3.org/2005/08/addressing">
          <usernametoken>xxxxx</usernametoken>
          <passwordtext>xxxx</passwordtext>
          <sessiontype>Stateless</sessiontype>
 </soap:header>

A sample SOAP Header for response received from Siebel is as below:

<soap:header>
  <sessiontoken>xxxxxx</sessiontoken>
</soap:header>

Incase Siebel SessionToken expires before making subsequent calls, Siebel will respond back with some error message like Invalid Session Token or Session Timed Out or Session Token has expired. In these cases just reconnect to Siebel by making another call to the Ping method to generate a new session token. The Old session generated on Siebel side will eventually timeout and won't waste any resources as such.

That's it ! You should be good to create sessions and make successful calls to Siebel now.