SOA 11g: Managed server error "weblogic.messaging.kernel.KernelException"

Recently came across an issue where the Managed servers were failing with below error. Even though servers start and come to running mode, but the JMS queues are not showing up in admin console under JMS Servers->Monitoring -> Active Destinations

<Error> <JMS> <BEA-040123> <Failed to start JMS Server "JMS_DEV_SERVER1" due to weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Ignoring 2PC record for sequence=4706 queue=57 because the element cannot be found.

weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Ignoring 2PC record for sequence=4706 queue=57 because the element cannot be found

        at weblogic.jms.backend.BackEnd.open(BackEnd.java:1008)

        at weblogic.jms.deployer.BEAdminHandler.activate(BEAdminHandler.java:200)

        at weblogic.management.utils.GenericManagedService.activateDeployment(GenericManagedService.java:239)

        at weblogic.management.utils.GenericServiceManager.activateDeployment(GenericServiceManager.java:131)

        at weblogic.management.internal.DeploymentHandlerHome.invokeHandlers(DeploymentHandlerHome.java:632)

        Truncated. see log file for complete stacktrace

This issue was after effect of the JMS file persistence store hitting 100% space utilization. As a result of which it got corrupted and was causing the JMS servers to fail while starting/activating.

To workaround this error, go to the Persistence File Store location (local/Shared SAN storage) and rename the existing *.DAT files to _bkp (say). Then go ahead and restart the managed servers. Now servers should come up fine without any issues/errors.

Oracle Metalink Note 1473826.1 "FileStore getting corrupted and WLS is unable to initialize JMS leading to BEA-040123 " suggests applying a Weblogic patch for Bug13900234. I haven't tried applying this patch but if issue persists inspite of workaround you can try this option as well.

Automating SOA 11g Admin tasks using WLST

WLST (Weblogic Scripting Tool) is a powerful feature for Weblogic administrators who want to automate most of their day to day operational tasks. This was initiated back in the BEA Weblogic 8.1 days for standalone weblogic server administration. And since Oracle SOA Suite 11g is now based on Weblogic server, this tool has become handy for SOA suite administrators.

There are lot of good learning materials available online for people getting started with WLST apart from the standard Oracle documentation. Few noteworthy ones are mentioned below. These links provide some nice examples with sample code which you can use based on your needs.

• Middleware Magic
• WLST By Examples

With this post I just wanted to give an overview on the different use cases where WLST can be used and how it eases lot of the operational tasks for SOA suite admins.

Common Use Cases 

• Monitoring STUCK threads/Hogging threads 
• JMS thread pool monitoring
• Retiring/Activating SCA composites
• Enabling/Disabling OSB proxy/business services
• Deploying SCA/OSB code
• Deploying/Updating other resources like JMS queues, JDBC datasources etc.
• Pausing/Resuming consumption/production on JMS queues
• Moving messages between JMS servers
• Generating Thread dumps and sending email alerts.

The main thing about working with WLST is understanding of how to navigate the configuration/runtime management beans(MBeans) and how to change the MBean attribute values.

Weblogic MBeans

Weblogic MBean API Reference

You can either directly connect to the WLST command prompt and execute the necessary commands after connecting to the runtime environment. An example shown below for retiring a SCA

  cd mw_home/Oracle_SOA1/common/bin
  ./wlst.sh

  Initializing WebLogic Scripting Tool (WLST) ...
 
  Welcome to WebLogic Server Administration Scripting Shell
 
  Type help() for help on available commands
 
  wls:/offline> connect ("username","password","t3://admin_hostname:port")
  
  Connecting to t3://admin_hostname:port with userid username ...
  Successfully connected to Admin Server 'AdminServer' that 
  belongs to domain 'SOA_DOMAIN'.
 
  Warning: An insecure protocol was used to connect to the
  server. To ensure on-the-wire security, the SSL port or
  Admin port should be used instead.

  wls:/SOA_DOMAIN/serverConfig> sca_retireComposite("listen address",
  "username","password","sca name","sca version",partition="partition_name")

The other way to execute is by writing WLST scripts (.py extension) which are similar in structure to Python language and include a lot of Python as well as Jython (Java implementation of python) libraries. Next you write some shell script (.sh) which invokes these .py files and can schedule the same on the crontab at specific intervals.

The shell script (.sh) basically sets the environment (CLASSPATH and PATH) by calling setWLSEnv.sh and then running the WLST script by calling it as below.

   java weblogic.WLST xxxx.py