Apache Nifi is used as the middle layer between OpenLMIS v3 and the reporting system (Kafka, postgres and Superset). Nifi is responsible for data extraction from OpenLMIS v3, normalizing it into the appropriate schema and storing it in the reporting stack's permanent storage PostgreSQL. Nifi queries the OpenLMIS APIs on a regular basis to extract information, it checks the reporting database to ensure the information exists and adds it if necessary. Additionally, Nifi is used to create Kafka topics, which are immutable permanent storage for all transactions that are performed by Nifi.

This user guide focuses on the first run experience for developers and will be expanded over time to support end users. It provides an overview of Apache Nifi and a step by step process for standing up the services.

Starting Apache Nifi

Clone the OpenLMIS-ref-distro GitHub repository into your local developer environment. We assume your environment already has Docker installed.

git clone https://github.com/openlmis/openlmis-ref-distro

Once cloned update your settings.env file as is a standard practice.

Enter the reporting directory and run docker-compose (Currently, there is a bug around scalyr socket not being available, so you may need to run this command multiple times).

cd openlmis-refi-distro/reporting
docker-compose up --build

After 2 or 3 minutes open your web browser and navigate to http://localhost:8080/nifi/ and you will be presented with the Apache Nifi home screen.

Integrating with the Nifi Registry

The Nifi docker container automatically integrates with the OpenLMIS Nifi Registry. You can view this integration by clicking on the hamburger menu in the top right corner of Nifi>Controller

Services and then clicking the Nifi Registry Tab.

The most straightforward way to start a flow is by right clicking the processor group or processor and clicking "Start". You can navigate into a processor group by double clicking the grey processor group. Once there, you can see a number of flows defined for a particular template. The screenshot below shows a nested processor group within the existing processor group and a number of processors. You can right click any processor (white background) to start or stop it.

How to view a queue

Nifi queues flow files as they move between processors. in the graphic above, you can see the right processor is named "GenerateFlowFile". This processor type starts a new flowfile and can be manually triggered by starting that processor.The link between that processor and the next step in the flow is a "success" relationship. Because the next processor is stopped, the flow file will queue until we stat the "Split Facilities and Programs" processor.

You can see that 1 flow file is in the queue. Right click on that queue and click "List Queue". You can empty the queue in the same menu by clicking "Empty queue".

The next screen shows all of the items in the queue. You can view the flow file attributes and content by clicking the info button on the left side.