Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Status

IN PROGRESS

Stakeholders

Josh Zamor Craig Appl (Unlicensed) Clay Crosby (Unlicensed) Peter Lubell-Doughtie (Unlicensed)

Outcome
Due date
Owner
Craig Appl (Unlicensed)

Background

The new reporting platform requires a number of components that each run independently. The team needs to determine a structure for the code repository to ensure we follow practices that are architecturally sound. This document provides a central location for the discussion and focuses on documenting the decisions we made to arrive at the structure. This document was informed by this thread on the Openlmis-dev forum

The following components will be addressed:

  • Apache Nifi
  • Kafka
  • Druid
  • (Optional) Relational Datastore - This is dependent on the implementation needs. We need to decide if OpenLMIS wants to support a relational database for smaller implementations.
  • Apache SuperSet
  • Apache Zookeeper

Each of the components need to be able to be stood up with a docker-compose command, they need to support CI/CD and eventually allow for version controlled templating. This document includes a proposed GitHub Repository structure with links to Docker hub containers that can be used.

Proposed Structure

We propose to create one repository for each component of the reporting stack:

  • openlmis-ref-distro-reporting
    • Description: This repository allows implementers to setup and run the reporting reference distribution independently from the openlmis reference distribution. An independent reporting reference distribution is recommended because it is a separate platform independent from the microservices paradigm, it is not required for a core implementation and should not be run on the same production machines as OpenLMIS. This GitHub repository will act as a central location for all reporting services, including maintaining a docker-compose file and any sample settings.env files that are platform dependent.
  • openlmis-reporting-nifi
    • Description: This repository will contain all code related to running a production quality, versioned Nifi environment. This repository will include Nifi specific templates and API calls to auto-load those templates when the docker container is running.
    • Official Website: https://nifi.apache.org/
    • Docker Container(s):
  • openlmis-reporting-kafka
  • openlmis-reporting-druid
  • openlmis-reporting-superset
    • Description: This repository will contain all code related to running a production quality, versioned SuperSet environment. This repository will include scripts that load in dashboard templates through the SuperSet API.
    • Official Website: https://superset.apache.org/
  • openlmis-reporting-zookeeper
    • Description: Apache Zookeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. This repository will contain all code related to running a production quality version of Apache Zookeeper.
    • Official Website: https://zookeeper.apache.org/
    • Docker Container(s):

OpenLMIS Reporting Repository Template

The OpenLMIS team maintains an OpenLMIS Service Template GitHub repository to support the standardization of all repositories across the GitHub organization. We need to identify if these templates are appropriate for each of these reporting components. This analysis was performed on 7 May 2018 and may have changed since.

Current repository template structure:

  • config
    • checkstyle
      • checkstyle.xml
    • pmd
      • ruleset.xml
  • consul
    • config.json
    • package.json
    • registration.js
  • src
    • integration-test/...
    • main
    • test/...
  • .gitignore
  • Dockerfile
  • ERROR_HANDLING.md
  • FLYWAY.md
  • LICENSE
  • LICENSE-HEADER
  • LICENSE-HEADER.md
  • NEW_SERVICE.md
  • README.md
  • STYLE-GUIDE.md
  • TESTING.md
  • build.gradle
  • build.sh
  • docker-compose.builder.yml
  • docker-compose.erd-generation.yml
  • docker-compose.override.yml
  • docker-compose.yml
  • documentation.gradle
  • gradle.properties
  • package.json
  • registration.gradle
  • settings.gradle
  • sync_transifex.sh

Notes on new reporting template structure:

  • To start, we will not have the following:
    • config directory - the config directory will likely change. The current config directory contains a code style checker and maven build style checks, which are both based on java principles. We likely don't need that for this configuration.
    • consul directory - we are not registering with consul
    • src directory - this likely won't include java code, more likely templates
    • FLYWAY.md - we likely won't have flyway migrations
  • We will likely need to add the following:
    • templates - this directory will include templates for auto loading information in to the service. This is applicable for Kafka topics, Nifi templates and SuperSet dashboards.

Discussion Topics

  • What monitoring systems are used by OpenLMIS?
    • How should each service interact with each monitoring system?
  • Each of these docker containers focus on deploying single node clusters. This is fine for development environments, but is not appropriate for production where we would expect the ability to horizontally scale each cluster. How can we account for this as OpenLMIS?
  • How should we handle translations? Do they only need to be accounted for in SuperSet?
  • Do we need to depend on gradle for any build processes? How can we load in templates through the API for each service? For example, this can be done with shell scripts.










  • No labels

OpenLMIS: the global initiative for powerful LMIS software