Research: Nifi Registry for Version Controlling Flows

Status

DONE

DescriptionResearch spike to test the Nifi Registry as a potential system for version controlling and auto-loading Nifi flows
Due date
July 5th Showcase
Owner

This page documents the research completed to test the Nifi Registry as a version controlling mechanism for Nifi flows. The OpenLMIS reporting system relies heavily on Nifi to stand up the schema for reports and begin the integration. "Registry—a subproject of Apache NiFi—is a complementary application that provides a central location for storage and management of shared resources across one or more instances of NiFi and/or MiNiFi." (Source) Nifi has a mechanism for exporting templates and we have been using this in a manual process.

The current manual process is as follows:

  • The developer creates a flow in their developmentenvironment
  • The developer exports the flow as a template and downloads it
  • The template is given a version number manually
  • The template is added to a local git repository and committed
  • The commit is pushed to GitHub and a Pull Request is initiated
  • The commit is reviewed and merged with master

Nifi Registry as a Service

This test assumes that the OpenLMIS community will run a single instance of the Nifi Registry as a service that supports multiple implementations. The Nifi Registry will be run alongside other services like the continuous integration services and demo servers. Developers working on the reporting systems will be given independent permissions to modify and create flows for OpenLMIS. Anyone downloading OpenLMIS will be able to read public flows. Furthermore, this could be extended to support implementation specific flows for things like integrations with third party systems.

Prove that the Nifi Registry Works with Nifi

Steps:

  1. Stand up Nifi Registry
  2. Create a new bucket for testing
  3. Connect Nifi to the registry
    1. Add Trust Store and download key
  4. Create a template of the existing OpenLMIS - OpenSRP integrations and push it to the Nifi Registry
  5. Add the template to the registry
  6. Change the template and add a new version

Challenges to this Approach

  • Nifi Registry requires separate credentials and access controls than GitHub. This will need to be maintained by the core OpenLMIS team.
  • Nifi Registry does not currently store passwords, so we need a separate mechanism for loading in passwords once the flows are loaded in.
  • The core OpenLMIS team would have to maintain a server and pay for hosting of that server.
  • Server Authorization, like what's currently setup on nifi-registry.ona.io is not supported in the Nifi registry. Instead, we need to research login groups in Nifi and the Nifi registry.


OpenLMIS: the global initiative for powerful LMIS software