Data Import Functionality - Technical Documentation

This functionality allows importing data configuration from an app level. Previously the data was being added via seed tool (GitHub - OpenLMIS/openlmis-refdata-seed: OpenLMIS Configuration tool ). It was quite a complicated mechanism so the goal here was to replicate the same behavior but accessible for a non-technical user. Is available via the Administration / Data Import tab.

Application view

Data import controller

curl 'https://uat.openlmis.org/api/importData' \
  -X POST \
  -H 'Authorization: Bearer BEARER_TOKEN_FROM_WEBPAGE' \
  -F "file=orderable_data.zip" \
  --compressed

Endpoint

POST https://uat.openlmis.org/api/importData

Headers

Authorization: A Bearer token generated on a webpage

Request body

file: File used for data import

File sent in request body is a ZIP archive containing one or more of the following files:

orderable.csv, programOrderable.csv, tradeItem.csv. Mapping files used previously in seed tool are no longer needed, orderableIdentifiers.csv as well.

Controller validates if the user has DATA_IMPORT right, then it returns data in form of DTO’s. BaseDto is used to conveniently return DTO that is corresponding to imported objects.

importData() function receives data in the form of multipart file, validates if the file is zip and has valid size. Then the files from an archive are converted into a Map, with file names as keys and input streams with file content as values. We are iterating through them, validating if the file is in csv format and searching for a corresponding persister. Persisters have the same bean names as imported file names. This allows us to scale the functionality, as we only need to add a single persister class to import new objects. If a bean with a certain name is not found, the exception is being thrown.

All persisters implement the DataImportPersister interface.

It has two methods:

createOrUpdate() that fetches previously existing data from a database, creates new objects or updates them and saves the result to a list for a further batch update.
processAndPersist() that reads the data from csv archive and puts it into corresponding DTO's, then calls createOrUpdate() function and saves its output into the database.