Technical Values & Approach

Project goal:  Deliver v3 reports to users of v2 / eLMIS in Tanzania quickly - without a "full" upgrade.

• Pipelines allow us to break the problem down into structured independent data flows.

• Everything is repeatable

  • Infrastructure as Code

  • (Re-)Running Pipelines

  • Experimentation

• Nothing is manual, nothing is a one-off  - Automate
  
  

  • From environment setup to deployment, we put everything we can in code.

  • We never "just do a one-time tweak to the database/API/app/etc"

  • Backups

• Resilient

  • Not a "big bang" migration, it will need to function continuously for weeks, months, perhaps even a year or longer.

  • During lifetime eLMIS and OpenLMIS v3 will have releases, data models will change without notice.

  • Source and destination software releases should slow new data from flowing, however both systems should stay available.

• Data is usable in near-real time, a user should expect their data once entered to be available in seconds to minutes, not tens of minutes, hours or days.

• Source system should be unaware and unaffected

  • We must demonstrate that eLMIS availability and performance can't be noticeably effected.

  • We must convince eLMIS developers that they do not need to coordinate releases with us, merely notify us of what's changing and by when.

Tech Stack:  Tech Stack

• Deployed via Terraform, Ansible and Docker.

  • Terrraform & Ansible to provision environments.

  • Ansible to manage deployed application.

  • Docker as deployment unit and runtime.

• Streaming architecture centered around Kafka

  • Decouple source and destination systems.

  • Native support for pipelines.

  • Kafka is one of the top leading technologies - built in support for high-performance, repeatable pipelines, and resilient log-architecture.

• Change Data Capture using Debezium

  • Debezium works off the same internal PostgreSQL mechanism as replication - eLMIS database should be unaware and unaffected by it's presence.

  • Change Data Capture allows us to capture changes made directly to a database as they're made

  • We have direct access to the source data & it's schema, including changes to either.

  • Resilient against network outages - if the network is down, then Debezium simply will queue until it's restored.

• Monitoring using Prometheus and Grafana

• Control using ???

  • Simple (e.g. REST) interface for starting/stopping pipeline

  • Clear a pipeline's destination, and re-run pipeline (e.g. clear a table in v3, re-run the pipeline with a new transformation phase).

• Simple Transformations using Nifi (or perhaps Java / Python)

  • Simple transformations may be chained together as needed, and are more testable.

  • Nifi is provisional:

    • Nifi is a tool we've been learning

    • Nifi Processors are a known commodity for building transformation stages

    • Nifi has been one of the more difficult tools to use, and it's more suited for building entire pipelines that are run one-off, rather than simple transformations.

    • We don't have a good method for testing Nifi - individual processors.

• Schema Registry (optional)

  • Reduces size of messages in Kafka for database changes, improving performance.

  • Optional as it's mostly a performance enhancement.