Who this is for / How to use this

...

• OpenLMIS v3
  • TODO: what should we link here ?
• OpenLMIS v3 reporting stack
• Kafka - Along with NiFi, Kafka is used to drive the Casper pipeline. Each change to the v2 database becomes a message on a Kafka topic, and NiFi outputs transformed data as messages to other Kafka topics
• Debezium - Streams data from eLMIS into Kafka
• Superset - The data visualization web app that is the user interface for the OpenLMIS v3 reporting stack
• AWS - The Casper demo is entirely hosted on Amazon's cloud services, managed by Terraform
• Grafana - We run this application for monitoring the Casper pipeline to view statistics about Kafka and NiFi

...

1. Set up your AWS access keys. 

   $ export TF_VAR_aws_access_key_id=$AWS_ACCESS_KEY_ID
   $ export TF_VAR_aws_secret_access_key=$AWS_SECRET_ACCESS_KEY

2. Go to one of the three subdirectories of the openlmis-deployment/provision/terraform/casper/ directory (e.g. v2/ for the server with eLMIS and the pipeline).

   $ cd openlmis-deployment/provision/terraform/casper/v2/

3. Prepare terraform (this creates the .terraform directory):

   $ terraform init

4. Start up the resources defined in the current folder. Or, if they are already running (e.g. if you are using the VillageReach AWS account), apply changes from any edited files. This command will ask for confirmation before actually making changes:

   $ terraform apply

5. You should be able to check that the newly created resources are working by pinging them, even though no applications are deployed yet e.g.:

   $ ping casper-elmis.a.openlmis.org

...

To understand better what's going inside NiFi, visit http://casper-elmis.a.openlmis.org:8080/nifi and explore the processors that are set up there; double-click on a process group to navigate inside, and use the bar at the bottom to navigate back up (reading this NiFi documentation will provide good background). Some of the processes in NiFi look like the image below (I've zoomed out here to hide extraneous details). Each piece of data that comes in to NiFi will follow paths defined by the arrows between the processors. In the example in the image, the arrows going in to the LogAttribute processor are only taken when a processor encounters an error; the path that each data piece will hopefully take in this image is ... → "Remove V2 Attrbutes and Set Defaults" → "Replace Schema" → "Save IDs to Attributes" → ... . Each arrow in NiFi is actually a queue; if a processor is working more slowly than data is coming in to it, the processors before it can still do work until that queue is full.

Anchor
transforms transforms

Transformations in NiFi

...

Each eLMIS table has a corresponding Kafka topic that starts with original, as specified in the Postgres connector. The NiFi processor is configured to monitor that Kafka topic and consume any new messages streamed to that topic, with the key data put into the FlowFile's kafka.key attribute, and the value data put into the FlowFile's content. (NiFi represents each piece of "User Data" as a FlowFile.)

...