Interface - Interface with DHIS2
Target release | 3.6 |
|---|---|
Epic | |
Document status | DRAFTED |
Priority | HIGH |
eLMIS Status | Implemented |
OpenLMIS Status | Not Implemented |
PATH | @Jenny Thompson (Unlicensed) |
OpenLMIS | @Mary Jo Kochendorfer (Deactivated) |
JSI | @Ashraf |
Goals/Scope
This is blocked by OpenLMIS and DHIS2 sharing a common understanding of:
Facilities / Locations, achieved by:
Products, achieved by:
While it is blocked, structuring the overall work this way should make it simpler.
Vision/Approach: https://docs.google.com/document/d/19xysVDrfBuJcqTyDd7-j3zzOSnAU0WMf6OcmSFZ20eI/edit?usp=sharing
Scope: OpenLMIS needs to send data related to RMNCH products and LLIN to DHIS2. <need to confirm what data is being sent and the scope>
Status in eLMIS: In eLMIS.
Status in OpenLMIS: Not Implemented
Priority: High priority for Tanzania
Background
The RMNCH program and Malaria program want to see eLMIS logistics data triangulated with service statistics-related data separately collected in DHIS2 through custom dashboards created in DHIS2.
This interface requires additional configurations at both end, described in the user stories.
Indicators (all have expressed desired from MW):
Stock on hand (aka stock status) - in TZ
received quantities
consumed quantities (in TZ)
total stock out days (not in TZ however interest has been expressed)
By product, time period and facility.
In Tanzania a few special notes apply for how these indicators are currently defined for DHIS2:
Tanzania is transitioning from a quarterly to a monthly reporting cycle.
Monthly consumption of product = quarterly consumption / 3
Stock status, the first two months are left as 0. the current stock status is reported as the stock on hand of the last month.
Assumptions
User Stories
# | Title | User Story | Label | Importance | Notes |
|---|---|---|---|---|---|
1 | Generate XML file for feed into DHIS2 | As an administrator I need a process for generating an XML file to send to DHIS2 so that I can export to DHIS2. | Tanzania eLMIS DHIS2 interface | Must have | This is currently a server side script in Tanzania. |
2 | Obtain URL for data submission from DHIS2 |
As an administrator I need information from DHIS2 so that I can submit data with the curl tool. DHIS2 requires: site address/URL, form submission location locational to be included on the URL, user id, and password. | Tanzania eLMIS DHIS2 interface | Must have | DHIS2 expects data to be submitted through “curl” utility. DHIS2 will provide these information. |
3 | Automate the data feed routine | As an administrator, I need to be able to automate monthly data submission so that user intervention is not required for this to happen consistently each month. | Tanzania eLMIS DHIS2 interface | Must have | This is done through server side scripting. |
Diagrams
Facilities list
Product list
Data items list
<dataValueSet xmlns="http://dhis2.org/schema/dxf/2.0" >
<dataValue orgUnit="Lo423ozVkFw" dataElement="ndLcCJP1R5n" period="201607" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="ndLcCJP1R5n" period="201608" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="ndLcCJP1R5n" period="201609" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="m6oXxb5XAUn" period="201607" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="m6oXxb5XAUn" period="201608" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="m6oXxb5XAUn" period="201609" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="DZKHCq8f3NQ" period="201607" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="DZKHCq8f3NQ" period="201608" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="DZKHCq8f3NQ" period="201609" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="EpNzl1xg26I" period="201607" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="EpNzl1xg26I" period="201608" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="EpNzl1xg26I" period="201609" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="JmxXYzhJMA6" period="201607" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="JmxXYzhJMA6" period="201608" value="0"/>
<dataValue orgUnit="Lo423ozVkFw" dataElement="JmxXYzhJMA6" period="201609" value="0"/>
…
…
</dataValueSet>
Sample XML file
Dependencies
Description | Link |
|---|---|
Discussion about Facility List Sync and Product List Sync as a blocker
The Governance Committee recommended that we develop the DHIS2 interface before developing facility list and product list synchronization. This section provides an area for discussion around the assumption that the Facility List Sync and Product List Sync are blockers.
Craig's comments:
From my perspective, the answer is "Yes" we can, and maybe should, do the DHIS2 integration before we do the facility and product list syncing. The DHIS2 integration will focus on identifying the org unit hierarchy, developing the pertinent data elements and identifying indicators that add value to DHIS2 consumers. eLMIS is already integrated with DHIS2 and we have a clear map between the DHIS2 field data elements, indicators and geographic hierarchy between OpenLMIS v2 and DHIS2. To put it another way, we have all of the information we need to do the DHIS2 integration.
Facility list sync vs DHIS2 org unit integration
The facility list sync epic focuses on developing automated mechanisms to share location information between OpenLMIS and another system. Regardless of the approach, OpenLMIS must be able to consume a list of locations and map them to the internal representation of the geographic hierarchy and list of locations in OpenLMIS. OpenLMIS must also be able to serve the list of locations and geographic hierarchy to a third party system so they can consume it. Beyond that, we need to develop a mechanism for moderating changes between each system when a location is created, edited or archived in any of the systems.
The DHIS2 org unit integration is a much reduced set of tasks because the focus is on mapping so the country can report information from OpenLMIS to DHIS2 for either data value sets or through DHIS2 tracker. In this case, OpenLMIS just needs to know the DHIS2 org unit ID of each location and geographic hierarchy level that's stored in OpenLMIS. When OpenLMIS needs to push information to DHIS2, these org unit IDs will be mapped to the outgoing JSON or XML and pushed to DHIS2. The minimum viable product does not require a two way sync. Instead, the user should be able to add DHIS2 org unit IDs for each location and step in the geographic hierarchy and view a report of all facilities that includes the DHIS2 org unit ID so they can go and correct any errors.
Product list sync vs. DHIS2 data element/value setup
Note: I'm getting a bit beyond my element here because I don't know exactly how products are structured in DHIS2. I assume they are added as either data elements or values. They could also be mapped to DHIS2 tracker as program elements.
The product list sync feature focuses on ensuring products are appropriately represented across OpenLMIS, ERP system(s) and reporting systems. In this case, there would be a mechanism to represent a product in a certain way in OpenLMIS and map that product to a different product in the ERP or reporting system using GS1 standards or not. The core development would include extending the OpenLMIS v3 orderable model to include representations from external sources so mapping can be done where there is the same product referenced by different codes in OpenLMIS and the third party system. If OpenLMIS does not have a representation of that product, the implementer should be able to import it along with pricing information for a particular trade item. Furthremore, OpenLMIS will need to be able to "serve" or export the product list in a way that it can be consumed by a third party (again following GS1 standards or not).
I expect that the need for product information in DHIS2 is drastically different from the needs of an ERP. The core use case for DHIS2 is program related reporting either through value sets or tracker. In this case, locations are most likely reporting aggregate values of a particular commodityType so that can be mapped against the health information that's stored in DHIS2. For example, they need to know how many vials of BCG were consumed, the number of doses per vial on aggregate and closed vial wastage in number of vials and doses. This information will then be compared against the total number of children given BCG during the same time period and they will be able to calculate open vial wastage. OpenLMIS will need to retain a map between the calculated indicators by commodityType (both vials and doses) and the DHIS2 dataElements, dataValues or indicator codes. This mapping is more like a commodityType to DHIS2 indicator mapping unless there is an initiative to store trade items in DHIS2.
From my perspective, the need for product information in DHIS2 is limited to commodityTypes and the product list sync requires a detailed list of trade items, lots, expiration dates, VVM status, price. etc.
The elephant in the room
From my perspective, the integration with DHIS2 isn't an integration between OpenLMIS and DHIS2. It's an integration between the reporting stack and DHIS2. As I understand it, the OpenLMIS microservices will not be aggregating information. That task is solely reserved for the reporting system. OpenLMIS could push individual values to DHIS2 through data pumps, if we want, but that doesn't solve the majority use case of "reporting" to DHIS2. In this case, the reporting stack needs to retain the map between the OpenLMIS geographic hierarchy and the DHIS2 org unit hierarchy. The reporting stack will also need to calculate the indicators that are required to be pushed as data value sets by period to get aggregate information into DHIS2. We could extend this to work with DHIS2 tracker if there is a clear use case, forwarding events when they are picked up in Nifi.
Open Questions
Below is a list of questions to be addressed as a result of this requirements document:
Question | Outcome | Status | |
|---|---|---|---|
| 1 | A District Consumption Comparison report is mentioned elsewhere in the gap analysis document. Is this also a Tanzania gap? "Displays consumption comparison across multiple districts. The comparison is represented by % i.e. total consumption of district x/total consumption of all districts * 100." | Yes, It is a gap in Tanzania |
|
| 2 | For Tanzania user stories: review/update the users and "so thats". |
|
|
| 3 | Is the only scope of data around consumption? Is there ever a need for DHIS2 to know the current SOH levels? | Yes |
|
| 4 |
|
|
|
I read through all comments/linked docs referred on the presentation slide deck. I identify more with the comments and sentiments Craig captured in this wiki page, that's why responding through this wiki page.
I'm making an attempt to explain my two cents opinion below (it refers to the slide deck, comments there-in, as well as the comments in this wiki page)
- Facility Registry. It is true that Tanzania eHealth Architecture ask for standard based facility registry. But at this point in time, it is more aspirational, rather than a functional reality. The eco system on which eLMIS is working now with - Epicor, eLMIS, DHIS2 – are not using any facility registry. Epicor enters new customer or does have the capability to import customer accounts (aka Facility codes) from CSV/Excel files (see – link 1, link 2). Epicor does not have any feature to link with a Facility Registry like Resource Map/OpenHIE. However, a linkage can be established between Epicor and Resource map through developing an intermediate app that will essentially export facility data in CSV/Excel format and importing that in Epicor, in theory. In practical reality this may face additional blocker due to dependency/validation that Epicor may apply, for example, missing required fields, creating a customer account may need more related info such as credit limit, primary contacts, type of customer, preferred delivery mode, etc., which Facility Registry may not have.
In other words, the way that we think about health facility and reporting by a health facility, a commercial ERP application does not think that way, it thinks in terms of customer, vendor, supplier, warehouse, 3PL etc. Also, when there is a need to add a new customer, the process is transaction driven (e.g., a new clinic placing an order for health products from the central warehouse (MSD in Tanzania). Epicor will have faced that transaction first, rather than the Facility Registry application. In all likelihood, Epicor will create a new customer account with all the other profile/financial details it needs to capture about that customer and fulfill that order. Resource map and eLMIS will have to sync with Epicor to create that facility code with a subset of data. The assumption that a customer will have to be setup in Resource map first, is impractical, most likely to fail because of inefficient and unreasonable demand it makes.
Now, lets bring in DHIS2 into the picture. DHIS2 treats a facility as an organizational unit. It allows grouping, hierarchical associations (province > districts > councils etc) between facilities. You can enter organizational units through the web site interface or you can import from CSV/XML/JSON files. You can include facility code on the import file. But that field is optional. When you import from a file, DHIS2 internally assign a org id which is not the same as the facility code. If you want to submit data to DHIS2 (e.g., eLMIS submitting data to DHIS2), you'll have to encode the record with orgUnit. DHIS2, currently does not have any built-in mechanism to sync with Facility Registry (Resource map) to ensure that organization codes are in sync with each other. If you develop an additional app to export facility list from Resource map and import it into DHIS2, DHIS2 is still going to assign it's own orgUnit id to that record and it would not accept any data submission that does not encode the data with that Id. A catch-22 situation. Can a solution be found, yes, but will require an upgrade to DHIS2, write some code in Resource Map or additional custom app to act as the translator.
<dataValue orgUnit="Lo423ozVkFw" dataElement="ndLcCJP1R5n" period="201607" value="0"/>
You may ask that, how come eLMIS is able to submit data to DHIS2 currently. It entailed a painful manual process of exporting all orgUnit Ids from DHIS2 into CSV file, manually mapping each of the orgUnit to a corresponding facility code, importing that data into eLMIS in a separate table. When eLMIS generates data for submission to DHIS2, it uses that table to encode data with orgUnit id that DHIS2 expects.
You can validate the assertions made above about DHIS2, at this links on DHIS2 documentation – managing organization units, meta data import, API doc
There could be other mechanism available to build interface with DHIS2 so that it can sync with a Facility Registry, but I relied on what is publicly available through their existing documentation.
- Product Registry. ERP application like Epicor treats product as an inventory item. You can enter it through UI or import it from CSV, XLS, XML, etc format. When we talk about GS1 bar coded product, we're taking about a product encoded with a bar code that is following one of the GS1 bar symbology (link). Scanning a bar code will read only few data points encoded in that bar code such as UPC, batch, serial #, expiration, etc. A typical product record will need additional required field. These are typically handled through GDSN (link). As can be seen from Oracle/Peoplesoft one is, link (I could not find online Epicor doc on this), on page 76, ERP can be configured to read and parse a GS1 bar code. When a new product is encountered, it would be through a receiving transaction at the ERP which will assign an SKU id to that record. eLMIS/OpenLMIS will not be aware of that transaction, until through a separate manual SOP eLMIS/OpenLMIS is notified and some import from CSV file process is triggered. Whether or not a product was GS1 bar coded, would not change this workflow. In fact less than 5% of MSD SKUs are actually going to be GS1 bar coded. That means a practical solution needs to support both GS1 bar code as well as non GS1 bar code or no bar code at all.
If a product registry application, such as Facility Registry (e.g., Resource map) were to be setup, it will face the same challenges. It will be the ERP that will encounter a new product through a receiving transaction first. The requirement to obtain a product/SKU code from 3rd party Product Registry first will prove to be impractical and likely to fail (imagine a scenario where a truck is waiting unload in a rainy day at the MSD warehouse and warehouse is refusing to the receive the shipment because somebody from that 3rd party app have not yet created an SKU for them and there can be dozens of new SKUs to be created in that shipment alone).
Now, let's put DHIS2 into the mix. DHIS2 does not have any concept about product, like the way it had for organizational units. It treats product as dataelement (similar to indicator or variable, a concept coming from statistic application like SPSS, SAS, or R). For example, for Stock-on-hand, DHIS2 will assign a dataElementId. For consumption it will assign a separate dataElementId, even through both data points are for the same product (see DHIS2 documentation on this here).
To DHIS2, health facilities are going to submit data through online forms. When a data submission form is designed in DHIS2, it assigns each data element id for each field/questionnaire on the form such as – open stock, receipts, closing stock, or a question like "Are you pregnant (y/n)"? DHIS2 allows data submission using their API, as long as you encode the data with the data element ids that it assigned when that online form was created. DHIS2, currently does not have any means to use product id that a product registry may assign. All it cares about is what data collection form you designed and what was the internal data element id it assigned. If you don't encode your data submission with that id, it will not accept that submission.
Now, coming back to the original premise of this discussion, that is what standard based interfacing mechanism to develop under OpenLMIS gap project. In my opinion, there is no simple solution. While we should all work towards a standard based approach, but a solution package to be implementable in near terms, needs to allow a phased adoption until everybody comes on-board. This will entail upgrades on respective applications as well.
I would argue that a variant of Option 2+, would be more practical.
How it might work?
Perhaps this way:
OpenLMIS interfaces with Facility Registry (optionally Product Registry), if such an application existing in the country and regularly syncs code with it
If one of those registry does not exists, or the interfacing app like Epicor is not able to communicate with it, OpenLMIS have capability to import from ERP application, newly created customer a/c (or product records) from CSV file. Then it submits that new record to Facility Registry or Product Registry to keep that in sync. This will acknowledge the fact that central warehouse receiving translation initiated the whole workflow and subsequent processes were triggered by that. This will also allow OpenLMIS implementation in other countries who may not have any functioning Facility Registry (or Product Registry), since it can maintain mapping code on it's own, if needed. This also encourages those country to gradually adopt such registries since OpenLMIS will have the capability to import existing records from such registry and send only new facility and new products to the registry. In this later case the registries will take over the mapping function.
Note that this reverse loop (ERP > OpenLMIS > Registry) is only for new facility and new products. For existing facility and products the regular data feed (Registry > OpenLMIS) should be followed.
For Interfacing with DHIS2, there is no mechanism currently documented in their website, that will allow DHIS2 to be standard based, to accept orgUnit (facility code) or dataElementId (product code) to be taken from either OpenLMIS of Facility/Product Registry. In other words, DHIS2 in it's current version is not capable to be standard based, particularly for products. For the short-term the manual process of getting an export of orgUnit and dataElementId from DHIS2 and manually entering those in either OpenLMIS or Facility/Product registry comes to mind. For Craig's question about whether OpenLMIS or OpenLMIS/Reporting service submits data depends on what type of data. If DHIS wants district/regional/six monthly/annual aggregated/summarized indicator data, then OpenLMIS/Reporting service makes better sense. If DHIS2 wants monthly data broken down to individual facility reporting, then OpenLMIS submitting to DHIS2 makes better sense.
In terms of prioritization, OpenLMIS's own capability to maintain code mapping code, ability to interface with Facility Registry, and manual mechanism to setup interface with DHIS2 can be the priority order. Product code mapping within OpenLMIS and ability to interface with product registry perhaps can be taken in later stages. For the short term, ERP can notify OpenLMIS through a report that get emailed to eLMIS admin on a monthly basis listing the newly created product/SKU(Epicor reporting tool has such scheduler/email a report, capability)
The commercial ERP applications has dealt with this set of problems, albeit with a much robust/costly solution package. Links here (X12, Amazon marketplace) discusses about how ERP applications uses X.12 EDI interfaces (or EDIFACT for EU) to exchange/interfaces with other systems through EDI documents. Different transaction uses different EDI documents. For example ERP can send X12 846 EDI advising new inventory or sending EDI 850 or 810 for sales (which may happens to be, to a new health facility). There are commercial 3rd party EDI vendor/translator) tool exists which can be added to the solution package.
Discussion above used Epicor as the ERP straw-man. But when Zambia migrate to Sage/WHExpert, the situation will be comparable (MACS WMS have limited EDI capability). Sage will bring capability similar to Epicor.
My two cents.