Reporting - Data Ingestion and Analytic Data Schemas

Target release3.4
Epic OLMIS-4637 - Getting issue details... STATUS
Document status
DRAFTED
PriorityHigh
eLMIS StatusUnimplemented
OpenLMIS StatusUnimplemented
Ona Craig Appl (Unlicensed)
OpenLMIS -
JSI -

Goals/Scope

Custom reports and, non-custom reports, new reports, and any report in Superset, will need to have the data it depends on available in a database connected to Superset.

Background

Each report requires different sets data to be built, but most of these sets of data overlap will overlap in some way or other.

We can split ingestion the following dimensions

  • Source component (e.g. Requisitions, Stock)
  • Report (e.g. Stock Imbalance, Order Summary)

When we ingest the data and push it into an analytic database we can store it a number of alternative formats (and in multiple simultaneous formats)

  • Replicate in API
  • Replicate structure in underlying databases
    • Could include replicating a structure from a previous version of OLMIS
  • Structure per report
  • Determine higher-level commonalities between repeats and store in those formats

Assumptions

User Stories

#TitleUser StoryLabelImportanceNotesJIRA Tickets


Pull data from APIs







Push data into raw storage (exact same format as when received from API)







Decide on analytic database structure







Push data into analytics databases (this is taken from our raw storage not API)





Dependencies

DescriptionLink


Jira Epics

OLMIS-4637 - Getting issue details... STATUS

OLMIS-4901 - Getting issue details... STATUS

Open Questions

Below is a list of questions to be addressed as a result of this requirements document:

#QuestionOutcomeStatus

Gap Estimation Notes 4/12/2018

  • Blocks all reports created in Superset
    • With the caveat that it can be broken down into pieces of ingestion