$customHeader
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Next »

Overview

There are two major services to be deployed in order to make the services work: nifi-registry and superset. The nifi-registry should be deployed first. Below steps are explaining how to deploy both services from the local machine, as it’s easier to make any adjustments than using Jenkins at this point. If there are any errors encountered during the deployment take a look at the forum post. Additionally, the more detailed configuration was described in Stand-up and Configure the Reporting Stack.

Note that any changes in the NiFi Flows will require to update the backup files in the configuration repository. Section 4. Redeploy the NiFi registry explains how to do it manually.

Preliminaries

  • Set up the AWS EC2 instance (as described in Configuration of the reporting AWS instance)

  • Get the domain name (to simplify the instruction we’ll refer to it as reporting.openlmis.org) for the AWS EC2 instance as soon as possible. Otherwise, some of the steps might have to be repeated.

  • Download the deployment scripts and configuration sample.

  • Download the ref distro reporting directory sample.

Steps to Execute

1. Prepare the Docker certificates

  1. Add your local machine to authorized_hosts (step 2 from this instruction) - in order to be able to execute the scripts

  2. On your local machine create docker-machine (step 3 from this instruction) with generic-IP-address set to the domain name (reporting.openlmis.org) and with the target machine name set instead of name1 (to simplify the instruction we’ll refer to it as name1)

  3. Copy directory ~/.docker/machine/machines/name1 via SSH to the EC2 instance, to the same directory:

    scp -r ~/.docker/machine/machines/name1 ubuntu@reporting.openlmis.org:/home/ubuntu/.docker/machine/machines/name1

  4. Enter the instance via SSH, and edit file /home/ubuntu/.docker/machine/machines/name1/config.json - the paths to docker machines might be different as it was generated in a different environment - it should be adjusted to EC2

2. Deploy the initial NiFi registry

  1. Remove all .properties files from directory config/services/nifi/scripts/preload/process-groups/OpenLMIS (from distro) if there are any

  2. Execute script nifi-registry/deploy_nifi_registry.sh (from deployment) to deploy the nifi-registry

  3. Wait until the NiFi registry is accessible (reporting.openlmis.org:18080/nifi-registry)

  4. Create a new bucket on the NiFi registry UI

3. Deploy the initial Nifi & Superset

  1. Execute the services/deploy_services.sh script (from deployment)

  2. Log in to the NiFi (reporting.openlmis.org/nifi)

  3. On the NiFi upload templates from XML files (directory nifi-flows in distro)

  4. Create a new template by dragging the ‘Template’ block and selecting proper files

  5. Start version controls for all Process Groups (PPM -> Version -> Start version control)

  6. For each template create .properties file in config/services/nifi/scripts/preload/process-groups/OpenLMIS/ (in distro) which should information about its versions. The information can be taken from the NiFi registry (reporting.openlmis.org:18080/nifi-registry/explorer/grid-list), as in the example:
    For Process Group:


    The file should contain:

    bucketIdentifier=903a5789-7791-46d3-b24d-b8c404116b60
flowIdentifier=52cff1a6-a488-4d12-bf64-6e05a334b4ce
flowVersion=1

  7. Redeploy the NiFi - the variables for the imported templated are set in the preload script

  8. Create new templates of each Process Group (PPM -> Create template)

  9. Download the templates and update those XML files in the repository (available in ‘Templates’ option in the main hamburger menu)

4. Redeploy the NiFi registry

  1. Adjust and use below script to prepare dump of the NiFi-registry in order to persist the changes made in NiFi flows:

    export DOCKER_TLS_VERIFY="1" &&
export DOCKER_HOST="<domain>:2376" &&
export DOCKER_CERT_PATH="<certs path>" &&
NIFI_REGISTRY_IMAGE="apache/nifi-registry:0.2.0" &&
BACKUP_DEST="." &&

NIFI_REGISTRY_ID=$(docker ps -a -q --filter ancestor=$NIFI_REGISTRY_IMAGE) &&
docker cp $NIFI_REGISTRY_ID:/opt/nifi-registry/nifi-registry-0.2.0/database $BACKUP_DEST &&
docker cp $NIFI_REGISTRY_ID:/opt/nifi-registry/nifi-registry-0.2.0/flow_storage $BACKUP_DEST &&
echo "Backup completed"

    Please note, that all values in the first part of the script might need adjustments. Even NIFI_REGISTRY_IMAGE might change over time.

  2. Place both folders with dump data (database and flow_storage) to the nifi-registry-dump directory (in credentials repository)

  3. Execute script deployment/nifi-registry/deploy_nifi_registry.sh to redeploy the registry

  4. Verify if the registry is working properly and contain the data (reporting.openlmis.org:18080/nifi-registry/explorer/grid-list)

5. Connect OpenLMIS with the Superset

  1. Create an OpenLMIS user (with email and any roles that might be used in the reports) that will be used by the Superset. In the instruction, a superset is used as its username (using a UI for this is a convenient way).

  2. Create OAuth User for the Superset in the OpenLMIS database, with adjusted clientsecret (OL_SUPERSET_PASSWORD) and redirecturi (SUPESET_URL) values. You can use and adjust the following SQL:

    INSERT INTO auth.oauth_client_details
(clientid, accesstokenvalidity, additionalinformation, authorities, authorizedgranttypes, autoapprove, clientsecret, refreshtokenvalidity, redirecturi, resourceids, "scope", webserverredirecturi)
VALUES('superset', NULL, NULL, 'TRUSTED_CLIENT', 'authorization_code', NULL, <OL_SUPERSET_PASSWORD>, NULL, '<SUPERSET_URL>/oauth-authorized/openlmis', 'hapifhir,notification,pcmtintegration,cce,auth,requisition,referencedata,report,stockmanagement,fulfillment,reference-instance-ui', 'read,write', NULL);

  3. Log in to the OpenLMIS as a superset to verify if it’s working. The user should be able to access the Superset reports in the Reports tab.

6. Set up the Jenkins jobs

  1. Create a job for the deployment of the NiFi registry with the following script (might needs some adjustments in directories) to execute:

    cp ./credentials/reporting/settings.env ./deployment/reporting_env/nifi-registry/settings.env &&

if [ -d ./credentials/reporting/nifi-registry-dump/ ]; then
  cp -r ./credentials/reporting/nifi-registry-dump/* ./deployment/reporting_env/nifi-registry/config/nifi-registry/
fi

cd ./deployment/reporting_env/nifi-registry &&

./deploy_nifi_registry.sh

  1. Create a job for the deployment of the NiFi & Superset with the following script (might needs some adjustments in directories) to execute:

    cp ./credentials/reporting/settings.env ./distro/reporting/settings.env
cd ./deployment/reporting_env/services

./deploy_services.sh ../../../distro
  • No labels
$customFooter