NiFi Training Notes

Open questions:

Notes

• Reporting stack process groups
  • Reference data connector
  • Requisition connector
  • Materialized views - refreshes the first two connectors, basis for indicator queries
    • Waits three minutes (hardcoded setting) on startup
  • Postgres db schema is created in init db, not in Nifi

Using the Nifi UI

• Currently most of the db connection pool settings are hardcoded (except the db password which is in the .env file), but there is a ticket to automate settings these from the env file
• Right click within process group > variables > specify your list of variables for later reference using NiFi language
  • We should write a ticket to make the user-client property configurable
• Nifi uses user-based token (using administrator user); we'll need to design intended use for service-based token and service account/API key tokens; reporting stack likely should use API key tokens OLMIS-3861 - Getting issue details... STATUS
• Each OpenLMIS API endpoint is converted to table using joltspec
• Antonate Maritim (Unlicensed) will write a a ticket for how to handle paginated responses from the API. Note this ticket may not be necessary to take on as Mateusz Kwiatkowski mentioned working on something to have page size = ~2B

Requisitions connector

• UpdateAttribute modified date to beginning of Unix epoch to get all requisitions on startup
• Runs every day at midnight
• cronmaker.com to generate cron expressions
• Trigger uses modified date of yesterday
• Requisition jolt transform splits requisitions and their line items into four tables

NiFi versioning

• Shows registry clients
• Right click on a connector, version, show local changes

2019-05-07

FHIR

• Convert requisitions in JSON to FHIR formats
• Industry standard format for interoperability
• Originally using Nifi between OpenLMIS and DHIS2
• OpenLMIS create data into FHIR, which then can be used by multiple sources, one is DHIS2
• DHIS2 Integration Software Requirements Specification
• What is a Measure
• • FHIR resource that tracks measure quantity, e.g. beginning balance, holds all the attributes of the thing that you are measuring
• MeasureReport
  • A bunch of Measures, create a MeasureReport from all of them
• Using a service-based token for getting FHIR resources
• Wambere (Unlicensed) to add the required customizations / configurations for FHIR integration to documentation
• Using just the Measure (FHIR) version provided by the FHIR server
• Assuming that only "our" Measures are on the FHIR server, we should document that assumption somewhere
• Getting FTAPs by administrator user, and assumes all products we care about are associated with facility type of health center
• Getting Measures only runs once, but should need to be run again each time the products list are updated, can only do this manually in the Nifi UI
• Using AttributesToCSV processor to see why DetectDuplicate is failing
• Processors can be renamed to have their purposes more clear (without opening them to look at config)
• Getting all the FHIR facilities the processor currently uses facility name which is not unique, this should use facility ID or a unique ID instead
• Getting locations, when evaluating JSON, all geo zones would have to be entered in properties for processor, which will not scale for many geo zones
• Get location name assumes district level and region level
• Get location id for requisition has hardcoded location names (demo data of Malawi and Mozambique)
• Get requisitions api needs to be updated to only return certain statuses (currently returns all)
• Generate Measure and Measure Reports process groups may not be generated correctly on Nifi startup; to ensure they are corrected generated, have to do it in Nifi UI after startup
• Stock status is hardcoded
• Are emergency requisitions reported in the Measure report? Should they?