Set up the reporting stack step-by-step

Overview

Below steps are explaining how to deploy NiFi and Superset services from the local machine, as it’s easier to make any adjustments than using Jenkins at this point. If there are any errors encountered during the deployment take a look at the forum post. Additionally, the more detailed configuration was described in Stand-up and Configure the Reporting Stack.

Any changes in the NiFi Flows will require to update the template files in the repository. Section Updating the XML templates explains how to do it.

Preliminaries

Set up the AWS EC2 instance (as described in Configuration of the reporting AWS instance)
Get the domain name (to simplify the instruction we’ll refer to it as reporting.openlmis.org) for the AWS EC2 instance as soon as possible. Otherwise, some of the steps might have to be repeated.
Download the deployment script and configuration sample.
Download the ref distro reporting directory.
Run the init-env.sh script to automatically generate the settings.env file and propagate all necessary variables.

Steps to Execute

1. Prepare the Docker certificates

Add your local machine to authorized_hosts (step 2 from this instruction) - in order to be able to execute the scripts
On your local machine create docker-machine (step 3 from this instruction) with generic-IP-address set to the domain name (reporting.openlmis.org) and with the target machine name set instead of name1 (to simplify the instruction we’ll refer to it as name1)
Copy directory ~/.docker/machine/machines/name1 via SSH to the EC2 instance, to the same directory:
scp -r ~/.docker/machine/machines/name1 ubuntu@reporting.openlmis.org:/home/ubuntu/.docker/machine/machines/name1
Enter the instance via SSH, and edit file /home/ubuntu/.docker/machine/machines/name1/config.json - the paths to docker machines might be different as it was generated in a different environment - it should be adjusted to EC2

2. Deploy the initial Nifi & Superset

Execute the services/deploy_services.sh script (from deployment)
Log in to the NiFi (reporting.openlmis.org/nifi)
Verify if the process groups can be executed properly (Right mouse button → stop & start)

As the initial XML files might contain some variables like URLs or credentials (that are automatically updated/replaced during the redeploy) it’s recommended to export the initially loaded Process Groups once to avoid any confusion. Section Updating the XML templates explains how to do it.

3. Connect OpenLMIS with the Superset

Create an OpenLMIS user (with email and any roles that might be used in the reports) that will be used by the Superset. In the instruction, a superset is used as its username (using a UI for this is a convenient way).
Configure OpenLMIS to create an OAuth Client for superset. Add the following lines in settings.env of OpenLMIS v3 (not reporting stack’s settings.env). PS: client ID should be any value different from the word “superset” and should match what is configured in reporting stack’s settings.env.
SUPERSET_CLIENT_ID=<ClientID> SUPERSET_CLIENT_SECRET= SUPERSET_REDIRECT_URI=<Superset URL>/oauth-authorized/openlmis
Log in to the OpenLMIS as a superset to verify if it’s working. The user should be able to access the Superset reports in the Reports tab.

4. Set up the Jenkins job

Create a job for the deployment of the NiFi & Superset with the following script (might needs some adjustments in directories) to execute:
cp ./credentials/reporting/settings.env ./distro/reporting/settings.env cd ./deployment/reporting_env/services ./deploy_services.sh ../../../distro

Updating the XML templates

Any changes made on the Flows will be lost when the NiFi is redeployed if the XML templates won’t be updated. Below steps describes the update process:

Create new templates of changed Process Group (Right mouse button -> Create template)
Download the templates and update those XML files in the repository (available in ‘Templates’ option in the main hamburger menu)