DevOps

Docker Compose setup

Running questions

• Why is reporting stack incorporated into ref distro? Why is it not in the main docker compose?

• Note to create a recommended topology for reporting stack

Reporting stack pieces

• Nifi

  • Open source by Apache, data ingestion, sophisticated ETL

  • Connect to OpenLMIS API, get data, preprocess data, then put it into reporting stack

  • Could get sources from anywhere, OpenLMIS, DHIS2, etc.

• Superset

  • Also by Apache, data visualization

  • Indicator queries, build charts

• Postgres

• Nifi → Postgres ← Superset

Nifi registry

• Nifi works with entities called process groups, flow of how to process data

  • Essentially a bunch of XML files

  • This is how to version control Nifi flows

• Nifi registry docker compose file separate from reporting stack

  • One registry for multiple deployments, why it is in the openlmis-deployment repository

Zookeeper/Kafka

• Don't use it right now

• For multiple instances of Nifi

• Tied in with Kafka

• @Jason Rogena (Unlicensed) to exclude Kafka and Zookeeper from documentation touchups he's doing

Reporting docker-compose file

• Zookeeper/Kafka - not used

• Scalyr - log analyzer

• Consul - register Nifi and Superset as running services, service discovery

• Nginx - reverse proxy, uses consul-template to pick up services

  • Potential conflict if both reporting and main docker compose are running on the same instance because of port conflicts

  • Also for managing authentication and SSL config

• Assistive containers

  • config-container - config folder, similar to service-configuration in main ref-distro docker compose

    • Also hits consul API and registers Nifi and Superset

    • Has a bunch of API statements, if those need to be customized, docker-compose file would have to be directly modified

  • db-config-container - db folder

    • No real reason why it is separate from config-container

• Some use settings.env file

• Nifi - under onaio

  • Initially because we needed OAuth support, but now no longer the case

  • Now can move back to apache Nifi image

Reporting .env file

• Nifi passwords are here, but not client ids or usernames

• Nginx basic auth username and pw are for clients needing to access Nifi through Nginx

Nifi registry docker-compose file

• In deployment repository

• Also has a backup container to back up to s3

Spinning up reporting stack

• docker-compose up --build --scale scalyr=0 --scale kafka=0 --scale zookeeper=0 - to build images and not bring up scalyr or kafka

• How nginx proxy passes, checks host header, if nifi, then redirects to nifi; if superset, then redirects to superset

• Superset image gets built each time we spin up reporting stack

  • This was not incorporated into CI process, but there is no obstacle that keeps it from being done

Reporting Stack Persistence

• How to load data into Nifi from a Nifi registry, persisting nifi flows for a nifi instance

  • Folder reporting/config/services/nifi

  • Process groups are versioned, each separately

  • For configuring load, need bucket id and flow id

  • Property file for nifi registries - preload/registries

    • Registry client name is used by process groups, unique, used by process groups

  • Property files for flows/process groups - preload/process-groups

  • When creating a new process group, config automatically uploaded and then persisted and backed up

• How to persist a nifi registry itself

  • Not in ref distro, but deployment repository, folder deployment/reporting_env

  • Two volumes, one for flow storage, one for db

  • Registry database has nifi metadata

  • Back up flow files to git, version control

  • Back up registry database (as well as flow files) to s3

  • When backup container comes up, gets backup files/db from s3, so it must come up before nifi-registry container

  • Backup cron schedule in .env file, default is once a minute

    • Check local files vs. s3 remote files, if local is newer, copy to s3

2019-05-07

How to do Provisioning / Deployment

• Shared UAT instance; shared registry, nifi, superset, etc.

• Not using AWS dashboard

• EC2 instance, terminate SSL certs on load balancer, so we can just create a new server and attach to load balancer

• Guidance on EC2 size?

• Need to update recommended deployment topology and requirements to include reporting stack

• Terraform, definition of state of servers, its maintenance (creating/destroying resources); "infrastructure as code"

  • These states are stored on s3

  • Can automatically create resources based on resource definition

  • Creates servers, load balancers, bringing up AWS stuff

• Ansible

  • Configuring inside the server itself; installing docker and other stuff, etc.

  • Kind of like sshing into the server

• Jenkins

  • Automated service deployment, git clone, docker-compose up

  • Use certs installed on Jenkins to connect to docker daemon on remote host

• Could define in terraform how to limit access from only the Jenkins server

• nifi-registry folder actually is a server that contains shared registry, nifi and superset

• The .terraform folder is a snapshot of the infrastructure generated, not checked into source control

  • Also backed up to s3 bucket

• Reference variables in main.tf with prefix "var."

• Can have multiple tfvars files, scanned and evaluated alphabetically in the folder

• Can also have default values in variables.tf

• Start with terraform init, then terraform plan (generates a .terraform folder), then terraform apply (which executes then saves)

• How does terraform connect with AWS to init and generate plan?

  • Need to install AWS CLI first

• The modules folder shows how it actually creates the resources

  • The local exec section (in compute.tf) runs the ansible playbook

Superset Persistence

• Superset Dockerfile

  • pip install a package called superset-patchup aka ketchup

  • Made modifications to superset itself

    • Want users to have role-based permissions in superset

    • Automatically create and grant role

• Superset config python file

  • Making changes to superset config

  • Import CustomSecurityManager

  • Using OAuth, because authentication with OpenLMIS

  • OLMIS_Gamma role

  • Users can self register

  • Allow-from uat.openlmis.org, need to customize setting

• Superset-patchup

  • Adding OAuth

  • Creating custom roles

  • Redirect URLs