Sending Error Notifications in MFT 12c

Oracle SOA Suite 12c introduced a new product called Managed File Transfer (MFT). Basically MFT is a centralized file transfer solution for enterprises which addresses several pain points wrt. file transfers

• It provides secure access to file transfers, tracking each and every step of the end to end file transmission.
• Easy to use UI for designing, monitoring and administering the file transfer solution which can be used by non-technical staff as well
• Extensive reporting capability with detailed status tracking and options to resubmit failed transfers. 
• Built-in support for many pre and post processing actions like compression/decompression, encryption/decryption(PGP etc.) which in turn means no need to write custom code for these actions.
• Out of the box support for many technologies including SOA and OSB which in turn leads to multiple integration pattern possibilities. Example: for a large payload processing MFT can simply consume the source file and pass it as a reference to SOA target for chunk read and transformation.
• Integration with ESS (Enterprise Scheduler Service) which is another new product feature in 12c allowing to create flexible schedules for the file transfers without having to write any custom code (quartz, control M scripts, cronjobs etc).

Even though the centralized monitoring dashboard in MFT console is robust and provides granular details of each phase of file transfer (eg: if the target systems are SOA it has embedded links to the EM Console for tracing the status there as well.) there maybe a need to send out email alerts for reporting file transfer failures to a support distribution list in a typical production environment (cannot expect people to keep monitoring the dashboard all the time).

Below are high level steps to enable email notifications in MFT 12c.

Step1:

Run the below WLST commands in sequence to enable the event and add contact for email notification: 

o   cd ${MW_HOME}/mft/common/bin

o   ./wlst.sh

o   connect("weblogic","xxxx","t3://hostname:port")

o   updateEvent('RUNTIME_ERROR_EVENT', true)

o   createContact('Email', 'abc@xyz.com' )

o   addContactToNotification('RUNTIME_ERROR_EVENT', 'Email', 'abc@xyz.com')

o   disconnect()

o   exit()

Step2: 

Configure the Email Driver for notifications. Basically set the Outgoing Mail Server and the port (mandatory ones)

 You have to restart the MFT managed server after changing these settings.

Step3: 

Try replicating a file transfer error and monitor the error both in Monitoring Dashboard as well as check for the error email in outlook.

NOTE:

If event listed above is not enabled or no contacts are specified, then notification messages are sent to JMS “MFTExceptionQueue”. 

SOA 10g FTP Adapter Not Picking Files From FTP Server

This being my first post of 2012, I would like to wish all my readers a happy and prosperous new year. New year is time for resolutions :-) , apart from regular ones like "Will try to reduce weight" or "Try to keep my room clean" which never works out for me, I will definitely ensure that "I will share knowledge regularly".


New year, old issue. So decided to iron it out once and for all. SOA 10g FTP adapter wasn't picking files from remote FTP server after Jan 1st. Last file read was on Dec 31st. On further analysis found that the FTP adapter Control file had got corrupted.


The control file can be found under $OH/j2ee/home/fileftp/controlFiles/{processname~version}/xxxx/lastModifiedDate.txt


On checking the content of this file found below entry

#Sat Dec 31 23:21:55 CST 2011

lastModifiedTime=1356969420000

As you can see the lastModifiedTime shows as 1356969420000 (This is in milliseconds). This is Unix time or Epoch time and you can use the Epoch converter to convert it into corresponding timestamp.

http://www.epochconverter.com

After conversion it comes to:

GMT: Mon, 31 Dec 2012 15:57:00 GMT

This is where the problem lies. The lastModifiedTime has been set as 31st December 2012. As a result all files since Jan 1st 2012 are not being picked by FTP adapter because the adapter compares the timestamp on remote FTP server and only picks files which are greater than the lastModifiedTime.

What is epoch time?

The Unix epoch (or Unix time or POSIX time or Unix timestamp) is the number of seconds that have elapsed since January 1, 1970 (midnight UTC/GMT).

SOLUTION

Stop the SOA server, take backup of controlFiles directory, remove the control files and redeploy the code , Start SOA server. As a result new control file should come into picture and FTP process should pick files correctly.

Why the controlFile (lastModifiedDate.txt) got corrupted at first place during year change...that is something which I have to figure out. Looks like a bug with Oracle FTP adapter. Will update this post once I get a confirmation regarding same.

Using SFTP with Oracle SOA

A common requirement in integration projects is to transfer files in/out of a system in a secured manner. FTP is the usual protocol for transferring files and if additional security is required then SFTP (Secured FTP) is the way to go. In this post I would cover some of the ways FTP adapter can be configured in SOA 10g and 11g to make use of SFTP.

For additional details on FTP adapter configuration you can refer to the Oracle link
http://download.oracle.com/docs/cd/B31017_01/integrate.1013/b28994/adptr_file.htm

SFTP supports couple of authentication mechanisms to ensure additional security on top of the FTP protocol.

• Password authentication
• Public Key authentication

In password authentication, the external site/vendor which hosts the FTP server shares a username/password combination which has to be configured on the SOA server. At runtime when a SFTP connection is attempted, the username/password is made use of for establishing the connection.

Similarly in Public key authentication, a private-public key pair is generated. The public key is shared with the external site/vendor which hosts the FTP server. At runtime when a SFTP connection is attempted, Fusion process will try to match the private key stored locally on SOA server with the public key on Remote FTP Server and do the authorization first before sending/posting the files.

The configuration information in either case is stored in SOA_HOME/j2ee/OC4J_SOA/application-deployments/default/FtpAdapter/oc4j-ra.xml  (SOA 10g) or MW_HOME/Oracle_SOA1/soa/connectors/FtpAdapter.rar/weblogic-ra.xml (SOA 11g).

For Password authentication below are the properties which you need to set (oc4j-ra.xml sample shown below)

<config-property name="host" value="XXXXX"/>
<config-property name="port" value="22"/>
<config-property name="username" value="xxxxx"/>
<config-property name="password" value="xxxxx"/>
<config-property name="useSftp" value="true"/>
<config-property name="authenticationType" value="password"/>

For Public Key authentication below are the properties which you need to set (weblogic-ra.xml sample shown below)

<wls:property>
<wls:name>host</wls:name>
<wls:value>XXXX</wls:value>
</wls:property>

<wls:property>
<wls:name>port</wls:name>
<wls:value>22</wls:value>
</wls:property>

<wls:property>
<wls:name>useSftp</wls:name>
<wls:value>true</wls:value>
</wls:property>

<wls:property>
<wls:name>authenticationType</wls:name>
<wls:value>publickey</wls:value>
</wls:property>

<wls:property>
<wls:name>privateKeyFile</wls:name>
<wls:value>path of private key file</wls:value>
</wls:property>

Apart from above configuration steps incase of Public Key authentication we can follow below additional steps to generate the private-public key pair and do corresponding setup for that.

1. On Remote FTP Server ensure that /etc/ssh/sshd_config has below parameters set
              RSA Authentication Yes
              PubKey Authentication Yes
2. On SOA server, generate the Public/Private Key pair using below command
            ssh -keygen –t  rsa
3. Once the public and private key are generated make a note of file path, file name etc.
4. Then copy the public key content to the Remote FTP Server . Login as the account with which the FTP has to be performed and put the public key content into file ~/.ssh/authorized_keys.
5. For modifying the weblogic-ra.xml in SOA 11g, extract the file from the FtpAdapter.rar file and after making changes repackage it using command (jar cvf FtpAdapter.rar .)

Incase you run into errors like below work with your network administrator to unblock the port 22 at firewall.
sftp xxxxx
Connecting to xxxxx...
ssh: connect to host xxxx port 22: Connection refused
Couldn't read packet: Connection reset by peer

Thats it ! Now you should be able to securely transfer files..