NiFi Training Notes

Open questions:

Notes

• Reporting stack process groups

  • Reference data connector

  • Requisition connector

  • Materialized views - refreshes the first two connectors, basis for indicator queries

    • Waits three minutes (hardcoded setting) on startup

  • Postgres db schema is created in init db, not in Nifi

Using the Nifi UI

• Currently most of the db connection pool settings are hardcoded (except the db password which is in the .env file), but there is a ticket to automate settings these from the env file

• Right click within process group > variables > specify your list of variables for later reference using NiFi language

  • We should write a ticket to make the user-client property configurable

• Nifi uses user-based token (using administrator user); we'll need to design intended use for service-based token and service account/API key tokens; reporting stack likely should use API key tokenshttps://openlmis.atlassian.net/browse/OLMIS-3861

• Each OpenLMIS API endpoint is converted to table using joltspec

• @Antonate Maritim (Unlicensed) will write a a ticket for how to handle paginated responses from the API. Note this ticket may not be necessary to take on as @Mateusz Kwiatkowski mentioned working on something to have page size = ~2B

Requisitions connector

• UpdateAttribute modified date to beginning of Unix epoch to get all requisitions on startup

• Runs every day at midnight

• cronmaker.com to generate cron expressions

• Trigger uses modified date of yesterday

• Requisition jolt transform splits requisitions and their line items into four tables

NiFi versioning

• Shows registry clients

• Right click on a connector, version, show local changes

2019-05-07

FHIR

• Convert requisitions in JSON to FHIR formats

• Industry standard format for interoperability

• Originally using Nifi between OpenLMIS and DHIS2

• OpenLMIS create data into FHIR, which then can be used by multiple sources, one is DHIS2

• DHIS2 Integration Software Requirements Specification

• What is a Measure

  • FHIR resource that tracks measure quantity, e.g. beginning balance, holds all the attributes of the thing that you are measuring

• MeasureReport

  • A bunch of Measures, create a MeasureReport from all of them

• Using a service-based token for getting FHIR resources

• @Wambere (Unlicensed) to add the required customizations / configurations for FHIR integration to documentation

• Using just the Measure (FHIR) version provided by the FHIR server

• Assuming that only "our" Measures are on the FHIR server, we should document that assumption somewhere

• Getting FTAPs by administrator user, and assumes all products we care about are associated with facility type of health center

• Getting Measures only runs once, but should need to be run again each time the products list are updated, can only do this manually in the Nifi UI

• Using AttributesToCSV processor to see why DetectDuplicate is failing

• Processors can be renamed to have their purposes more clear (without opening them to look at config)

• Getting all the FHIR facilities the processor currently uses facility name which is not unique, this should use facility ID or a unique ID instead

• Getting locations, when evaluating JSON, all geo zones would have to be entered in properties for processor, which will not scale for many geo zones

• Get location name assumes district level and region level

• Get location id for requisition has hardcoded location names (demo data of Malawi and Mozambique)

• Get requisitions api needs to be updated to only return certain statuses (currently returns all)

• Generate Measure and Measure Reports process groups may not be generated correctly on Nifi startup; to ensure they are corrected generated, have to do it in Nifi UI after startup

• Stock status is hardcoded

• Are emergency requisitions reported in the Measure report? Should they?