Reporting - Data Ingestion and Analytic Data Schemas
- Peter Lubell-Doughtie (Unlicensed)
- Mary Jo Kochendorfer (Deactivated)
- Craig Appl (Unlicensed)
Goals/Scope
Custom reports and, non-custom reports, new reports, and any report in Superset, will need to have the data it depends on available in a database connected to Superset.
Background
Each report requires different sets data to be built, but most of these sets of data overlap will overlap in some way or other.
We can split ingestion the following dimensions
- Source component (e.g. Requisitions, Stock)
- Report (e.g. Stock Imbalance, Order Summary)
When we ingest the data and push it into an analytic database we can store it a number of alternative formats (and in multiple simultaneous formats)
- Replicate in API
- Replicate structure in underlying databases
- Could include replicating a structure from a previous version of OLMIS
- Structure per report
- Determine higher-level commonalities between repeats and store in those formats
Assumptions
User Stories
| # | Title | User Story | Label | Importance | Notes | JIRA Tickets |
|---|---|---|---|---|---|---|
Pull data from APIs | ||||||
Push data into raw storage (exact same format as when received from API) | ||||||
Decide on analytic database structure | ||||||
Push data into analytics databases (this is taken from our raw storage not API) |
Dependencies
| Description | Link |
|---|---|
Jira Epics
OLMIS-4637 - Getting issue details... STATUS
OLMIS-4901 - Getting issue details... STATUS
Open Questions
Below is a list of questions to be addressed as a result of this requirements document:
| # | Question | Outcome | Status |
|---|
Gap Estimation Notes 4/12/2018
- Blocks all reports created in Superset
- With the caveat that it can be broken down into pieces of ingestion
OpenLMIS: the global initiative for powerful LMIS software