Goal: Kickoff technical vision, ask questions.
Agenda:
- Overview project, goals
- Overview thoughts on technical vision, identify where the highest priority questions/risks are
- AOB
Project and goals (incl technical)
- Get v3 reports working in country, where the source system is v2.
- Deliver, in-code (repeatable and testable), an ETL pipeline that can pull data from v2 and get it into the v3 reporting stack.
- Deliver a working system that has 0 bespoke modifications to v3, so that the v3 components can still be on the continual upgrade path
- Deliver a streaming system, so that the source system has no discernible impact, and that the pipeline's transformation stages may be repeated at-will.
- Achieve a 0 modification goal to v2 source system
Technical vision
- Leverage the "reporting stack" to form the basis of this data migration pipeline.
- Move the reporting stack back toward a streaming (lambda) architecture (w/ Kafka yet again)
- Introduce Debezium for Change Data Capture, also to move back toward a streaming architecture, and to eliminate (nearly) load on source v2 system.
Risks
- Reporting stack learning curve
- Access to v2 production data, access to discern semantics behind structure
- Introducing Debezium
- Solve for aggregate root problem
- Stream join
- Aggregate root table
- Solve for aggregate root problem
- Reporting stack doesn't achieve robustness level we'd need