Update 22 Jan 2016: New parameters missingLastFields and additionalLastFields to handle structure deviations between configured parameters and actual file content.
Update 26 May 2015: New parameter genericRecordType to cater for records with no key field value as shown in scenario 3. Previous scenario 3 renamed to Scenario 4.
Update 25 Mar 2015: DeepFCCBean has been refactored to be part of FormatConversionBean. Parameter conversionType replaced with converterClass. Note that some of the screenshots are not updated to reflect the new parameter converterClass.
Update 17 Mar 2015: Parameter indentXML replaced by indentFactor.
Introduction
This is the second part of the two-part series on achieving file content conversion (FCC) for deep structure with a reusable custom adapter module. The first part covered the conversion of deep XML to flat file at a receiver channel.
This second part will focus on conversion of flat file to deep XML at a sender channel.
Source Code
Refer to following blog on location of source code and/or EAR deployment file.
FormatConversionBean - One Bean to rule them all!
Prerequisite
- Each substructure type must contain a field containing the key value to enable the parser to identify lines of this substructure type
- Each substructure type must belong to one and only one parent substructure type. The parent can be either the root element or other substructure types defined
Usage of Module in Communication Channel
Module Processing Sequence
| Number | Module Name | Type | Module Key |
|---|---|---|---|
<Depending on position of module in chain> 1) Asynchronous scenario, Normally before the last module in channels 2) Synchronous scenario, Before last module to convert request payload After last module to convert response payload | Custom_AF_Modules/FormatConversionBean | Local Enterprise Bean | <Any Arbitrary Value> |
Module Parameter Reference
Below are the parameters for configuration of the module for flat file to deep XML conversion (converterClass = 'com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter'). Certain parameters will automatically inherit the default values if it is not configured.
The parameters are separated into parameters that are common for this conversion type, and also parameters that are specific for field separator based or fixed length based conversion.
1) Common Parameters
| Parameter Name | Allowed values | Default value | Remarks |
|---|---|---|---|
| conversionType | DeepPlain2XML | Required field. Determines conversion type | |
| converterClass | com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter | Required field. Determines conversion class | |
| encoding | UTF-8 | Defines encoding of output plain text file | |
| documentName | Required field. Document name of root element of XML output | ||
| documentNamespace | Required field. Namespace of root element of XML output | ||
| recordsetStructure | Required field. List of substructure names separated by comma | ||
| genericRecordType | Name of substructure used as generic type. If configured, any lines that cannot be determined via key field determination will be treated as this generic type. | ||
| keyFieldName | Required field. Name of the key field that must occur in all substructures | ||
| indentXML | Y, N | N | Determines if XML output will be indented or not |
| indentFactor | Integer values beginning from 1 | 0 | Determines the number of indentation spaces for each level in the XML output |
| rowOffset | Integer values beginning from 1 | 0 | Starting row to begin extracting content from (i.e. 0 = start from first row, 1 = start from second row) |
| trimContents | Y, N | Y | Determines if all the leading and subsequent blanks for a field will be trimmed or not |
| <StructureName>.fieldNames | Required field. Names of fields in this substructure | ||
| <StructureName>.keyFieldValue | Required field. Key field value to identify this substructure | ||
| <StructureName>.parent | Required field. Specifies the parent of this substructure. Value should correspond to a substructure in recordsetStructure or the value 'Root' to indicate that it belongs directly under the root element | ||
| <StructureName>.missingLastFields | ignore,add,error | ignore | Behavior of conversion when there are less fields in content compared to configured fields in <StructureName>.fieldNames
|
| <StructureName>.additionalLastFields | ignore,error | ignore | Behavior of conversion when there are more fields in content compared to configured fields in <StructureName>.fieldNames
|
| debug | Displays audit log messages of substructure and parent-child determination of each line. WARNING: Use this only for debugging in non-productive systems | ||
| messageLog | pre, post | Saves a log version of the message that is viewable in Message Monitor
| |
| logLocation | Name of log version when messageLog is populated. Location defaulted to value in messageLog if logLocation not populated |
2) Field Separator Conversion Parameters
| Parameter Name | Allowed values | Default value | Remarks |
|---|---|---|---|
| defaultFieldSeparator | Defines a default field separator for all substructures that do not have <StructureName>.fieldSeparator explicitly configured | ||
| <StructureName>.fieldSeparator | Either <StructureName>.fieldFixedLengths or <StructureName>.fieldSeparator must be populated | ||
| <StructureName>.enclosureSignBegin | Specify a string that acts as a text delimiter. Field separators within such texts are ignored | ||
| <StructureName>.enclosureSignEnd | If the text delimiters for the beginning and end of the text are different, specify the text delimiter for the end of the text here. If it is not populated, it uses value specified in <StructureName>.enclosureSignBegin | ||
| <StructureName>.enclosureSignBeginEscape | Specify a string that replaces the text delimiter if it occurs within a text that it delimits. When the text is transferred the string is replaced by the value specified in <StructureName>.enclosureSignBegin | ||
| <StructureName>.enclosureSignEndEscape | Specify a string that replaces the text delimiter for the end of the text if it occurs within a text that it delimits. When the text is transferred the string is replaced by the value specified in <StructureName>.enclosureSignEnd. If it is not populated, it uses value soecified in <StructureName>.enclosureSignBeginEscape | ||
| <StructureName>.enclosureConversion | Y, N | Y |
|
3) Fixed Length Conversion Parameters
| Parameter Name | Allowed values | Default value | Remarks |
|---|---|---|---|
| <StructureName>.fieldFixedLengths | Integer separated by commas | Either <StructureName>.fieldFixedLengths or <StructureName>.fieldSeparator must be populated. The number of lengths must match the number of fields listed in <StructureName>.fieldNames |
Similar to standard FCC functionality, fieldSeparator parameter supports non-printable ASCII characters as follows:
- characters encoded as hexadecimal values in the form of '0xHH' (including quotation marks)
Example Scenarios
Here are some example scenarios of the behavior of the conversion based on different configuration options.
Scenario 1
3 substructures with 3 level deep structure
Field separator based conversion
Field separator - Order and Item using default comma. Delivery using tab (specified in hexadecimal)
Order uses " as enclosure sign, and "" as enclosure escape
Item uses " as enclosure sign but enclosure sign is not removed when transferred
Row offset provided to skip first line
Output XML indented
Module parameters
| Parameter Name | Parameter Value |
|---|---|
| converterClass | com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter |
| documentName | MT_DeepPlain2XML |
| documentNamespace | urn:equalize:com |
| recordsetStructure | Delivery,Order,Item |
| keyFieldName | Type |
| Delivery.fieldNames | Type,DeliveryNo |
| Order.fieldNames | Type,DeliveryNo,OrderNo |
| Item.fieldNames | Type,OrderNo,ItemNo,Quantity |
| Delivery.keyFieldValue | D |
| Order.keyFieldValue | O |
| Item.keyFieldValue | I |
| Delivery.parent | Root |
| Order.parent | Delivery |
| Item.parent | Order |
| Delivery.fieldSeparator | '0x09' |
| defaultFieldSeparator | , |
| Order.enclosureSignBegin | " |
| Order.enclosureSignBeginEscape | "" |
| Item.enclosureConversion | N |
| Item.enclosureSignBegin | " |
| rowOffset | 1 |
| indentFactor | 4 |
Result
| Input |  |
| Output |  |
Scenario 2
4 substructures with 3 level deep structure
Fixed length based conversion
Leading and trailing whitespaces are not trimmed
Missing fields are automatically populated with blank
Output XML indented
Module parameters
| Parameter Name | Parameter Value |
|---|---|
| converterClass | com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter |
| documentName | MT_DeepPlain2XML |
| documentNamespace | urn:equalize:com |
| recordsetStructure | Header,Delivery,Order,Item |
| keyFieldName | Type |
| Header.fieldNames | Type,FileName |
| Delivery.fieldNames | Type,DeliveryNo |
| Order.fieldNames | Type,DeliveryNo,OrderNo |
| Item.fieldNames | Type,OrderNo,ItemNo,Quantity |
| Header.keyFieldValue | H |
| Delivery.keyFieldValue | D |
| Order.keyFieldValue | O |
| Item.keyFieldValue | I |
| Header.parent | Root |
| Delivery.parent | Root |
| Order.parent | Delivery |
| Item.parent | Order |
| Header.fieldFixedLengths | 5,20 |
| Delivery.fieldFixedLengths | 5,10 |
| Order.fieldFixedLengths | 5,10,10 |
| Item.fieldFixedLengths | 5,10,10,10 |
| trimContents | N |
| indentFactor | 4 |
Result
| Input |  |
| Output |  |
Scenario 3
Conversion where one and only one substructure type does not have key field to identify the type of record
Substructure Detail is configured as the generic record type
Sample test data for this scenario is from the following thread File Content Conversion - No Key Field in Detail Structure
Module parameters
| Parameter Name | Parameter Value |
|---|---|
| converterClass | com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter |
| documentName | MT_DeepPlain2XML |
| documentNamespace | urn:equalize:com |
| recordsetStructure | Header,Detail,Footer |
| genericRecordType | Detail |
| keyFieldName | Type |
| Header.fieldNames | Type,HeaderField |
| Detail.fieldNames | Field1,Field2,Field3,Field4,Field5 |
| Footer.fieldNames | Type,TrailerField |
| Header.keyFieldValue | HDR |
| Footer.keyFieldValue | TRL |
| Header.parent | Root |
| Detail.parent | Root |
| Footer.parent | Root |
| Header.fieldFixedLengths | 3,11 |
| Detail.fieldSeparator | , |
| Footer.fieldFixedLengths | 3,9 |
| indentFactor | 2 |
Result
| Input | HDRXXXYYY12345 MM8100422373,M631530008 ,1530008,031615,17 MN8100424411,M631530008 ,1530008,031615,22 XY8100422373,M631530008 ,1530008,031615,17 TRLXXZZZ4321 |
| Output |  |
Scenario 4
Finally for the fourth scenario, we will use the same input file that was used in the XSLT approach. Actual configuration and testing screenshots are shown.
Default field separator for all substructures is used.
A log version of the message payload before conversion is saved.
Module configuration on an SFTP sender channel.
The 'plain' log version shows the flat file before conversion.
The converted payload is sent to a receiver proxy without further mapping. As shown below, the target payload has been converted to a deeply nested XML structure.
The audit log shows the trace of steps being executed by the module.
Conclusion
With the completion of this second part for DeepFCCBean, we now have a generic, reusable and highly configurable custom adapter module that is able to handle conversion of flat file to deep XML and vice versa! 