Overview and Definition

What is Program Data? A working definition:

We call it "Program Data" when:

a non-programmer would want to change the fields and data collected for implementation-specific needs; and when
OpenLMIS only needs to know WHETHER the data has been collected, but does not need to know the CONTENTS of what data is collected.

In other words, it is data that is not tied to the OpenLMIS system (its business logic, calculations, notifications). The timing of when this data is received can be asynchronous (collected at a different time).

We may decide to pick a different name. Examples: user-defined data, custom data, non-logistics data, non-stock data, un-actionable data.

For OpenLMIS, there is a certain need to ensure program data has been collected during certain workflows, like requisition.

Requirements and Scope

Requirements for OpenLMIS to support program data, some must have and some nice to have:

Requirement	Sub-requirement	Must Have / Nice to Have
Form design - determining what data needs to be captured and how, and designing a form to facilitate that data capture This part of the system should be able to accommodate a user that is mostly non-technical (can use Excel or an interactive form designer tool to design forms)	Basic forms - enough to facilitate basic program data capture A user can define the questions of the form, but the layout is just one question after another	Must Have
	Complex forms - a fully featured form designer. Could potentially include (this list is not exhaustive): Layout - how fields, labels and headers are organized Skip - allowing fields or rows to be skipped Validation - validating input is "correct" Calculations - showing calculations in forms	Nice to Have
Data capture - entering data into and completing a form as part of a workflow This part of the system should be able to accommodate a user that is completely non-technical (simple, straightforward form of entering relevant data; cues of what is incomplete)	Online data capture support (one way sync)	Must Have
	Offline data capture support - easy to go offline, capture data and sync it back to the server once online	Very Nice to Have
	Mobile (tablet or smartphone) data capture - either through a web or native app Note: this is not mutually exclusive with the offline sub-requirement	Nice to Have
	Two way sync	Nice to Have
Reporting on data - taking the data, running reports and doing visualization on it This part of the system should be able to accommodate a user that is mostly non-technical (can use reporting tool to generate reports or interact with tool		Must Have
Integrated "look and feel" - elements of program data look like part of the OpenLMIS system (users do not need to go to a separate system to fulfill program data requirements) This requirement is most important during the data capture process		Nice to Have
Security - ensuring OpenLMIS users can access their program data and cannot access other program data for which they are not authorized	Quick to implement: Within OpenLMIS UI, only see data for your facilities/programs Users need to sign onto another system to enter program data Admin maintains users in 2 systems	Must Have
	Better: Single sign-on (SSO) for OpenLMIS and Program Data system Unified RBAC across OpenLMIS and Program Data system	Nice to Have

Proposal for Program Data Service

In order to create a loose coupling between logistics workflow and program data, there is a need to create a new Program Data microservice. This service is what the rest of the OpenLMIS system (Requisition, Stock Management, Informed Push) would talk to for its program data requirements.

We think that it's sufficient for other services to know whether data has been collected (e.g. to allow submitting a requisition) but not the contents of that data. Therefore we propose a very minimal service that only answers the question "was program data collected for a facility + program + period?"

Of course the program data must be collected, but the other services do not need to know the details of how this happens. Regardless of whether the program data is captured through the OpenLMIS UI, or through an external mobile or web tool, we should mediate all interactions that other services have through this thin service.

A diagram with the interactions:

Open Questions:

Discussion around the natural key and configuration value or document/source ID.
- Natural key for requisition service identified as composite of facility, program and processingPeriod, but this may not be true for other services; may need a document/source ID of program data.
- This natural key may need to be configurable in some way.
Configuration may be needed on the requisition template (does program data need to be checked for, for this program?) - this is true, but this would probably be dealt with in the requisition service; it does not affect the logic in the program data service

Proposal for DHIS2

We propose using DHIS2 as-is to capture data for facilities, programs and periods, because it is a mature product that already supports this use case. We will do a simple connection that allows OpenLMIS to query DHIS2 and determine whether data has been captured or not.

There are a number of upsides to using DHIS2 as a program data solution.

It fulfills many of the requirements, including all of the Must Haves, out-of-the-box:
- Form design - can support basic and some complex forms through a WYSIWYG form designer (under Data set, data set management)
- Data capture - done through Data Entry web app, with some offline support (support for connectivity interruptions measured in seconds/minutes, not hours/days); there is also a mobile web client and an Android app
  - Note: even though the web app does not have full offline support, the fact that many countries with intermittent connectivity are using it, indicates it may be sufficient for program data capture requirements
- Reporting on data
- Security - DHIS2 appears to have a robust RBAC design, similar to OpenLMIS
It has been proven to be usable by country ministries and facilities, and has a community behind it
There are no dependencies on external development; APIs have been released and are available

The weaknesses of this approach are that if we want it integrated into the OpenLMIS UI, we won't get the benefits that come out-of-the-box. Another weakness is that DHIS2 is another system that must be maintained (although often the country already maintains an instance).

MVP using Program Data Service and DHIS2

As a first step to exploring how program data would work in OpenLMIS, an MVP is proposed. It would have:

A DHIS2 instance which would handle form design, data capture and reporting as a separate web app and login
A new Program Data microservice as described in the "Proposal for Program Data Service" section above
- DHIS2 would be the "backend" for the Program Data Service, in order to answer questions about program data completeness
A single DHIS2 "OpenLMIS system" user for integration which has the correct DHIS2 permissions to read data required by OpenLMIS

The new Program Data Service would be a lightweight service which has one API endpoint:

It would handle requests from consumers–Requisition, Stock Management, Informed Push Services
- Input: a program data composite (facility, program, processingPeriod)
- Output: true/false depending on whether that program data is complete
To answer these requests, it would make an external call to DHIS2 to get relevant data (integration piece)
- Authenticate with a DHIS2 "OpenLMIS system" user
- Call to DHIS2 APIs, /api/<version>/dataValueSets
  - Input: a mapped program data composite between OpenLMIS and DHIS2
    - An OpenLMIS Facility maps to a DHIS2 Org Unit
    - An OpenLMIS Program maps to a DHIS2 Program
    - An OpenLMIS Processing Period maps to a DHIS2 Period
  - Output: dataValueSet result, with completeDate if program data is complete)

Benefits to doing this MVP with this new microservice and DHIS2:

It is a low-effort proposal - we think it would take a fraction of the effort of alternative approaches
We get significant functionality out-of-the-box, including reporting and a WYSIWYG form designer, which we would not get from an MVP using alternative approaches
It is a low-risk proposal because:
- We are using a mature tool, so we are not dependent on any external roadmap
- DHIS2 is widely used, so it is a safe choice for the ecosystem
We build the Program Data interface and integration to other services
- The Program Data Service's "backend" (DHIS2) could be swapped out without much waste of effort
- Additionally, no changes to other Services' logic would be necessary, since the Program Data's API would be unchanged
It will cheaply test how painful it is to have Program Data in an external system (integrating in the UI will be only of the largest costs of any approach)

Diagram of the interactions:

Component Diagram for the Program Data Service:

Next Steps

Some of this depends on findings in the MVP. However, a list of potential next steps:

Support for Informed Push - should be straightforward, as it would access the same Program Data endpoint with the same criteria (facility, program, processingPeriod). See interaction diagram above.
If we like DHIS2 and want to invest more in that integration, then:
- Explore single sign-on (SSO) or synchronize users
- Explore integrated "look and feel" - with SSO, it may be possible to put access to DHIS2 in an iframe in the OpenLMIS UI, but effort would be required to give it the same "look and feel"
If we find DHIS2 is unsuitable, then pivot to supporting ODK2 as a "backend"
If we find that there is a dependency on the content of the Program Data, then work on returning basic key-value pairs of program data and not just a flag of program data completeness