Shipment File Configuration

Often, OpenLMIS implementation is complemented by an ERP running in central warehouses. To this end, OpenLMIS Fulfillment service has built-in capability to send order files to and accept shipment files from a warehouse management ERP. This built-in capability assumes that the ERP that is used in the warehouse will be able to read order files and write shipment files as CSV files to an FTP or Local folder that the fulfillment service can access. This page outlines assumptions as well as what is required to configure the shipment file processing capability.

The basics

A requisition in the OpenLMIS is converted to order. When a user converts a requisition to order, the OpenLMIS generates an order file and uploads it to an FTP folder that the ERP can read it from. An ERP, upon reading the order file, will be able to process the order. By processing the order in the ERP, the warehouse would be able to fulfill the order. Once the order is fulfilled, the ERP can generate a shipment file containing what products are shipped to the client facility. This shipment file can then be read into the OpenLMIS. Shipment file processing refers to the reading, parsing, and storage of shipment information in the OpenLMIS.

Assumptions

The shipment file is assumed to have the following characteristics.
1. It is a CSV file.
2. It contains a column that Identifies the order that the shipment is fulfilling. This order identifier can be either order code or order id.
3. It contains a column that identifies what products/Orderable are fulfilled. Orderable can be identified by either the orderable GUID or product code column.
5. It contains a column to indicate what quantity is shipped. The values on this column should be an integer greater than or equal to zero.

Required Configuration

Three distinct details should be configured to activate shipment files import capability in the fulfillment service. These details are:

1. Shipment templates.
2. The transfer properties.
3. Shipped By User ID.

1. Shipment file template.

The fulfillment service assumes that there will be one shipment file template for the parsing of shipment files from an ERP. This shipment file template should be configured using this API.

The shipment template should contain at least three columns defining the required shipment columns. These required columns are;

1. Order identifier (orderId or orderCode)
2. Product/Orderable Identifier (productCode or orderableId)
3. Shipped Quantity

The validation on shipment file templates expects that there will be keyPath attributes that match the required columns.

Another property of note is the "position" property in the FileColumn. This property should be used to tell the fulfillment service what zero-based column index to use to retrieve data for that column. Consider the following example when setting the column position.

2. Transfer Properties

Transfer properties tell the fulfillment service where to get the shipment files from. The transfer properties for shipment files can be FTP, Local, or SFTP server directories. This API can be used to create/update shipment transfer properties.

Please note that the FTP remote paths that are provided in this configuration are assumed to be the base path for where incoming, archive and error folders for shipment files will be found. The shipment processing assumes that the base path contains "incoming" directory. The shipment processing feature picks files with pattern of "*.csv" from [BasePath]/incoming/ directory.

Upon processing the shipment files, the fulfillment service creates two folders in the base path. Successfully processed files will be placed in the archive directory. However, if there are errors on the shipment file, the shipment entity is not persisted, the shipment file will be forwarded to the error directory.

[BasePath]/incoming
[BasePath]/archive
[BasePath]/error

3. Shipped by User ID and polling rate.

In order to save a shipment record, it is required to have a valid user id. This user id can be configured once as an environment variable that will be used in the fulfillment service. These values are read from SHIPMENT_POLLING_RATE_MILLISECONDS and SHIPMENT_SHIPPED_BY_ID environment variables which can be initialized with the settings.env file when the fulfillment service is started.

Exception Handling

The fulfillment service implements a number of validations to ensure that the data that is imported from shipment files is consistent with the assumptions outlined above. In cases where the incoming shipment file breaks significantly, the fulfillment service will reject the incoming shipment file. Rejected shipment files along with a technical log that contains the stack trace is archived in the "[BasePath]/error" directory. The two files will have the same date/time timestamp prefix that allows administrators to be able to diagnose the error and take corrective action if need be. 

Some of the causes of error that result in shipment file to be rejected are:

• Error with the shipment template configuration (missing required fields)
• The shipment file does not contain a valid order identifier. 
• The shipment file contains an order id that has shipment records already. 
• The shipment file does not contain valid CSV
• The shipment file contains rows with invalid value for order identifier or quantity shipped (empty or negative value for quantity shipped) 
• The shipment file does not contain any string for orderable identifier. 
• Any other unhandled exception. 

Unresolved Orderables

The fulfillment service validates the Orderable identifiers that are found in the shipment file. This validation is made with the assumption that the the product identifier found in the reference data service is synchronized with the product list in the ERP. However, there may be times when the product lists between the two systems get out of sync. To support this, the fulfillment service does not reject shipment files that contain new product identifiers that are found in the ERP but not in the reference-data service. Rows with unresolved Orderable identifier are persisted in the extraData field and are also returned from the API that returns shipment data as shown below.