Reporting - Data Ingestion and Analytic Data Schemas
Target release | 3.4 |
|---|---|
Epic | |
Document status | drafted |
Priority | High |
eLMIS Status | Unimplemented |
OpenLMIS Status | Unimplemented |
Ona | @Craig Appl (Unlicensed) |
OpenLMIS | - |
JSI | - |
Goals/Scope
Custom reports and, non-custom reports, new reports, and any report in Superset, will need to have the data it depends on available in a database connected to Superset.
Background
Each report requires different sets data to be built, but most of these sets of data overlap will overlap in some way or other.
We can split ingestion the following dimensions
Source component (e.g. Requisitions, Stock)
Report (e.g. Stock Imbalance, Order Summary)
When we ingest the data and push it into an analytic database we can store it a number of alternative formats (and in multiple simultaneous formats)
Replicate in API
Replicate structure in underlying databases
Could include replicating a structure from a previous version of OLMIS
Structure per report
Determine higher-level commonalities between repeats and store in those formats
Assumptions
User Stories
# | Title | User Story | Label | Importance | Notes | JIRA Tickets |
|---|---|---|---|---|---|---|
|
| Pull data from APIs |
|
|
|
|
|
| Push data into raw storage (exact same format as when received from API) |
|
|
|
|
|
| Decide on analytic database structure |
|
|
|
|
|
| Push data into analytics databases (this is taken from our raw storage not API) |
|
|
|
|
Dependencies
Description | Link |
|---|---|
Jira Epics
https://openlmis.atlassian.net/browse/OLMIS-4637
https://openlmis.atlassian.net/browse/OLMIS-4901
Open Questions
Below is a list of questions to be addressed as a result of this requirements document:
Question | Outcome | Status |
|---|
Gap Estimation Notes 4/12/2018
Blocks all reports created in Superset
With the caveat that it can be broken down into pieces of ingestion