Notes/Outcomes from OpenLMIS-ODK Collaboration

This document is public. Anyone can register for an account here to the OpenLMIS wiki by clicking Sign Up at the top right and then contacting Tenly Snow (Deactivated) or Mary Jo Kochendorfer (Deactivated) for full write access after you have signed up.

Day 1

Attendees

  • Waylon Brunette/UW ODK,
  • Mitch/UW ODK, 
  • Clarice Larson/UW ODK, 
  • Jeff Beorse/UW ODK,
  • Niles/UW summer project/VA Tech,
  • Josh Zamor/OpenLMIS Architect,
  • Darius Jazayeri/ThoughtWorks,
  • Brandon Bowersox-Johson/VillageReach,
  • Nick Reid/VillageReach

Agenda

  • ODK Architecture
  • OpenLMIS Scenario Introduction
  • OpenLMIS Working Diagram
  • OpenLMIS Integration Details
  • Specify MVP Details


ODK v2 Architecture

  • Survey is the v2 replacement for ODK Collect from v1
    • it is 100% webkit
  • Tables is a web view to access data
  • Services is a core that the other components share; it runs a web server on the mobile Android device; Survey and Tables each inject JS ODK libraries into WebKit

The ODK Docker Endpoint:

    • can run in the cloud
    • looks identical to a Mobile ODK handset
    • has an entire copy of the data set; syncs that data set to and from each Mobile ODK handset using that same "application"
      • if the data set gets too large, you could segment into multiple "applications" by kind of form or by geographic region
    • stores full diffs of all data ever in that application
    • is not intended to serve as long-term archival storage of the data submissions
      • later there is a "Missouri" project with UC Berkeley that may build a long-term data archival piece, but not in next 3 months

Work-In-Progress Diagrams

 

Integration Diagram

Initial integration plan from Day 1 Before Lunch:

Use Case / Workflow Scenario

  1. Admin creates and distributes an ODK Form (e.g., HIV ARV Regimens form for monthly programmatic reporting)
  2. User logs in (to _____)
  3. User fills out survey
  4. User goes online and completes/submits survey
  5. System makes submission read-only
  6. OpenLMIS polls for changes
  7. OpenLMIS reads the feed of surveys submitted, indexes it, and stores its URI, also may archive the submission in OpenLMIS long-term archival storage
  8. User logs in to OpenLMIS web UI (may be a different user or supervisor)
  9. User views Requisition can see whether ODK survey is complete (optionally may be able to see the survey submission)
  10. User approves Requisition

Discussion of This Workflow

  • OpenLMIS ODK form distributed
    • ODK form gets full list of facilities/programs/periods from OpenLMIS
    • Question: What does an implementer have to do get linked properties (ie Programs, Facilities, ect) then it MUST only show the information an end user has access to
  • User logs in to ODK Mobile
    • ODK uses Spring security through LDAP interface
    • Docker endpoint needs secure ldap endpoint w/ldap username/password and a group prefix
      • Using group prefix is used to filter the amount of logins that are being exposed
    • On ODK user login from server, get roles and groups
      • User stays logged in with server until bad sync
  • User fills out ODK form
  • User submits/completes/sync survey
    • Warning: Form validation might change between user completing a form and updating 
  • ODK System makes the form read only
    • Question: ODK allows for incomplete forms to be updated, and we need to decide where a form is marked "complete"
      • OpenLMIS processing period might make it so a user can't make changes to a form
  • OLMIS Polls changes
  •  User views requisition and sees evidence of form
  • User approves requisition  

MVP - first draft

MVD = Min. Viable DemoMVP = Min. Viable ProductLater
Mobile data collection in ODK

Pick list data in ODK comes from OLMIS

(ie Facilities, Programs, Periods)

ODK & program data used in other services
(i.e., other parts of OpenLMIS might start using ODK
as a mobile-friendly UI to collect data) 
Status of form submission visible in OLMIS

Web based data collection (ie ODK inside OLMIS UI)

  • No browser installs
  • Not pixel perfect
  • Same form definition from ODK
Integrated look and feel for ODK + OLMIS

ODK Dockerized within OLMIS deploymentLong term storing and using ODK Data in OLMIS

ODK Configuration in OLMISScaling performance (i.e. 4,000+ facility list)

XLSX Form UploadODK Form Builder within OLMIS

(The MVP got refined for Day 2.)

Break-out: Authentication Roles/Rights/Supervisory Nodes

Josh and Mitch discussed:

  • OpenLMIS concept is tied to a particular program and facility ID, potentially.
  • ODK roles are "global" and give widespread access.
  • Idea: need lookup tables with lists of facilities, and a column in that table with read-only access to that information so you can see it. We would show users a dropdown list—which facility do you want to enter program data for? Another lookup table would have Program IDs and facility IDs, and another read-only group. That would allow you to choose which program ID you are filling out your data for. Then yet another table is a reference table with facility ID, program ID, and list of retro-virals or whatever data items you are trying to collect for. That last one does not need any read-only restriction because you can only get there by answering the first two questions. Then we display a table where you enter data for those first two.
  • The challenge is: how does this map back to the supervisory node "rights" of the OpenLMIS security model.
  • Waylon: we could do other approaches too. In the static table it could have a list of every facility that user is assigned to, and they would just be able to see that, and everyone else would be blocked.
    • Brandon: can the user just link this up on the OpenLMIS UI by saying "I just made a submission" and paste a document ID or pick from a pick list of which ones they just submitted?
    • Waylon: the list table is being generated by OpenLMIS. There is a lot of detail to dive in to.


Break-out: UI Integration Approaches

Nick and Waylon discussed:

  • Nick: We have a game-plan for pulling in files with Bower, wrapping those into OpenLMIS UI, so the OpenLMIS UI basically includes ODKv2 Survey.
  • Hard Questions: 
    • Where are we storing the form objects and form data?
  • We can add the ODK Survey application into OpenLMIS. That should be quick and fun to put together.
  • Discussed running things inside versus outside the web browser.
  • Nick: Simplest way to get this running is to store the data as an intermediary in OpenLMIS web services.
    • Maybe we would run the ODK web services up inside OpenLMIS in the cloud.
    • Each user has a SQLite database. Easy to work with PostgreSQL too.
  • Waylon: Nick is here to help tell us and shape this future.
  • Thursday topic: Waylon and Mitch show something (JDBC connection to a SQLite).

Day 1 Wrap-Up

What went well? What did we learn? What do we need to improve on for Thursday?

  • Went Well:
    • Had some good conversations.
    • Started at high level.
    • Diagram before lunch is a good achievement. We also got good definitions of MVD/MVP/Later done.
  • To Improve:
    • Need to figure out what our goal and our demo/outcome for Thursday is.
    • Need to work on the lists of todos. Not sure how we get into the next level of gritty details.
    • Figuring out mapping of security roles is the big unknown.
      • But maybe that can wait for the MVP (not the MVD), or for the Later phase? We don't need this for the demo.
    • Avoid the rabbit holes. Or only go into the ones we want to go into.
    • Start Thursday with the discussion of what we still need to go into deeper.
      • Josh: It would help to go deeper into the roles and responsibilities of each component in the diagram.
    • 9am start time Thursday (it's okay with everyone including Nick).

Day 1 Parking Lot

This is an ongoing list of questions and conversations from our working sessions that need to be resolved.

  • Authentication
    • Make OpenLMIS speak LDAP? Add OAuth2 to ODK?
  • Form retrieving linked property values
    • linking facility IDs, program and periods
  • When to make rows read only
  • Mapping facility to ODK docker endpoint and support independent docker endpoints
  • Deployment
    • OpenLMIS Reverse Proxy and RAML
  • Break-outs:
    • Set up ODK, build a form
    • How OpenLMIS uses RAML
    • OpenLMIS UI stack

Day 2

"Minimum Integration" (revised)

Use Case (revised)

Josh wrote this revised scenario (in his first comment below) to capture the minimum integration use case:

  • there is an Anti-retroviral (ARV) program which needs to collect some basic Regimen data alongside the Requisitioning process for ARVs.  This Regimen data is tabular, capturing the number of patients receiving treatment using a particular "cocktail" of ARV drugs.  Ideally the number of columns and rows is variable at the time of the Requisition to capture variances in available drug combinations and changes in KPI's the program would like to see (e.g. a set at one point might be patients starting, patients on regimen for more than X months, etc).  However to start it should be enough to say there is one form with a set # of columns and rows for every facility.
  • A Requisition (non-emergency) is submitted on a defined Schedule (usually quarterly).  Each time a Requisition for the ARV program is to be submitted by a Facility, the ARV Program Data form must also be submitted.  The ARV Program Data form may be submitted before the Requisition is initiated possibly, but not after the Requisition is submitted.
  • The ARV form must be able to define a basic "I'm complete" so that the Requisition knows that the ARV program data form has been completed prior to the Requisition being submitted.
  • from the Requisition Service, we'll want to ask of the integration: is the ARV form for X Facility, of Y Processing Period and Z Program complete?  And receive a yes/no answer.
  • The person that is allowed to fill out the ARV Program Data form, must be definable using OpenLMIS' RBAC structure, which includes Supervisory Nodes, Role Assignments, Programs, Requisition Groups, Home Facility, etc.  This means that it must be possible to give an OpenLMIS user the permission to fill in the ARV Program Data form at a facility (which might be the person's home facility or that person might be a supervisor of that facility)
  • the ARV program data must be programmatically lockable.  This form is locked once the Requisition is submitted.  Or it could just be that the form is locked after the first completion.  Either way, after it's locked, it may not be removed or altered.

Minimum Integration Requirements (revised)

Josh wrote this and it replaces the MVP/MVD ideas from Day 1:

To say we've accomplished an ODK integration with OpenLMIS we need to achieve the following requirements:

  • Form needs to be able to collect tabular data
  • ODK integration has a rich mapping so that the OpenLMIS RBAC mechanism to identify which OpenLMIS authenticated users may submit the program data form, and which may not.
  • Providing a basic understanding of when that form is "complete"
  • Demonstrating the identification of a program data form by at least a few OpenLMIS dimensions:  Program, Facility, Processing Period.
  • Allowing for forms to be locked programmatically
  • Deployable with OpenLMIS and reachable from OpenLMIS Services.


Permissions

Task List

TaskComponentDetailsStatus as of Oct 2017
Minimum Integration
ODK Endpoint working in OpenLMISODK

(requires OpenLMIS team help to understand RAML, etc)

  • Docker compose or Swarm
  • deployment instructions
  • RAML? (maybe)
ODK team spun up ODK endpoint in Docker, Li Lin shared code and documentation
RBAC Magic StringifierOpenLMISconverts 'dimensions' of facility/program/period in OpenLMIS into a string to use an a group/role name in ODKOpenLMIS team released permission string feature in OpenLMIS 3.2.0
Create ODK2 templatesboth

with dimensions/reference data


OAuth2 integrationODK

OpenLMIS ODK program data connector serviceOpenLMIS
  • locator
  • indexer
  • can query ODK to fetch data with dimensions and check completeness
  • can lock submissions

What this achieves: Can submit data from Android device, can see in OpenLMIS whether it is submitted.
But we have solved the hard parts—auth, reference data lists, deployment.
Future Version
UI for web client data entryOpenLMISwithout this, Android is required for ODK program data entry
UI for data viewing/reportingOpenLMISto view the program data that was submitted in the OpenLMIS web UI
UI for ODK2 administrationOpenLMISwithout this, technical skills are required for an implementor to set up the ODK-OpenLMIS integration
Support multiple program data forms per programOpenLMISwithout this, there is at most 1 program data form per program
Performance/Scaling for the endpoint and connector serviceboth

Long-term storageOpenLMISstore submissions inside OpenLMIS in long-term archive (because the storage inside the ODKv2 ecosystem is transient)
Apply ODK beyond program dataOpenLMISother parts of OpenLMIS (apply this integration to other domains such as re-supply, stock management, etc)
Push notification from endpoint via webhookbothwhen new ODK data is available/synced

What this achieves:

  • Easier to use
  • Scalable
  • Re-usable throughout OpenLMIS


User Story

Architecture Diagram

Source: https://www.lucidchart.com/publicSegments/view/92aca91c-4b7d-4d7d-80e1-9437f76c892a/image.pdf

Showcase

Agenda

  1. Introductions
  2. Goals
  3. User Story - Nick
  4. Architecture Diagram - Josh
  5. Task List - Brandon
  6. Future Version / Vision for Next - Brandon
  7. Q & A
  8. Next Steps

Showcase Notes

(see comments inline on task list above)

Next Steps:

  • estimation
  • seeking out what is dual-purpose / re-usable in OpenLMIS; what would VillageReach team do anyway
  • UW team is moving ahead with some of the ODK platform changes
  • timeline - uncertain for VillageReach team; July is the earliest available collaboration for UW team

Final Wrap-Up

What went well? What did we learn? What do we need to improve on for our future collaboration?

  • Went Well:
    • Getting specific / diving in
    • RBAC details
    • Learned about ODK
  • To Improve:
    • Figure out our shared goals (the people attending the collaboration should be part of defining the shared goals in advance)
    • a speaking conch (or similar to help our conversations)
    • getting stakeholder feedback sooner (we wish the input from Jake and Mary Jo came a day earlier)
    • understand the UI integration
    • A/V setup with projector and microphones for remote participation
    • donuts


OpenLMIS: the global initiative for powerful LMIS software