SAP IDoc Data With Japanese Kanji or Chinese Characters Overflows

Recently came across an issue while integrating SAP R3 (4.2) version with SOA 11g (11.1.1.4). When idocs are received by SAP Adapter from the SAP system containing Japanese Kanji/Chinese special characters, it was found that the data overflows on to next segments/xml tags and is distorted. This causes mapping issues as incorrect data is being mapped to. The root cause behind this behaviour is that SAP R/3(4.2) is a non-unicode system while SOA 11.1.1.4 is Unicode.

The issue is covered in below Oracle documentation as well
http://download.oracle.com/docs/cd/E14571_01/relnotes.1111/e10132/adapters_iway.htm#CIHBCICF

Here is the explanations provided by Oracle

"This issue only occurs on non-Unicode SAP MDMP environments, where one character can be two or more bytes. As an example of this issue, when using Japanese, the SAP field length is four characters. The English word "ball" fits correctly into the field because one character equals one byte. The Japanese word for ball in Shift-Jis encoding is three characters, but two bytes per character, so the last character is truncated and the last character appears in the next field. Since IDocs are positional delimited, this can cause errors in processing. This occurs because SAP uses character length, not byte length for all non-Unicode field lengths. There is no work around on this issue other than using Unicode or using shorter text in IDocs in DBCS."

None of the workaround suggested above were feasible options. So tried the below approach to fix this...

From SAP before sending the idoc, convert all Japanese strings to hexadecimal characters. Once SOA suite receives the idoc, the BPEL uses a java embed/custom XSLT function (java code) to convert the hexadecimal back to Japanese characters.There is lot of sample code available for doing these kind of hex to string conversion. Just make sure you are using the correct charset (like shift-jis for japanese kanji) while doing the conversion.
Now you should be able to see the idocs properly and use the converted japanese,chinese special characters correctly in your mappings.