Data Import Functionality - Technical Documentation
This functionality allows importing data configuration from an app level. Previously the data was being added via seed tool (https://github.com/OpenLMIS/openlmis-refdata-seed ). It was quite a complicated mechanism so the goal here was to replicate the same behavior but accessible for a non-technical user. Is available via the Administration / Data Import tab.
curl 'https://uat.openlmis.org/api/importData' \
-X POST \
-H 'Authorization: Bearer BEARER_TOKEN_FROM_WEBPAGE' \
-F "file=orderable_data.zip" \
--compressed
Endpoint
POST https://uat.openlmis.org/api/importData
Headers
Authorization: A Bearer token generated on a webpage
Request body
file: File used for data import
File sent in request body is a ZIP archive containing one or more of the following files:
orderable.csv
, programOrderable.csv
, tradeItem.csv
. Mapping files used previously in seed tool are no longer needed, orderableIdentifiers.csv
as well.
Controller validates if the user has DATA_IMPORT
right, then it returns data in form of DTO’s. BaseDto is used to conveniently return DTO that is corresponding to imported objects.
importData()
function receives data in the form of multipart file, validates if the file is zip and has valid size. Then the files from an archive are converted into a Map, with file names as keys and input streams with file content as values. We are iterating through them, validating if the file is in csv format and searching for a corresponding persister. Persisters have the same bean names as imported file names. This allows us to scale the functionality, as we only need to add a single persister class to import new objects. If a bean with a certain name is not found, the exception is being thrown.
All persisters implement the DataImportPersister
interface.
It has two methods:
createOrUpdate()
that fetches previously existing data from a database, creates new objects or updates them and saves the result to a list for a further batch update.processAndPersist()
that reads the data from csv archive and puts it into corresponding DTO's, then callscreateOrUpdate()
function and saves its output into the database.
OpenLMIS: the global initiative for powerful LMIS software