2020-03-10 Sprint Transition

Date

Mar 12, 2020

Participants / Internal Stakeholders

@Felimone Amone Junior
@Matthew Kumbuyo (Deactivated)
@Chongsun Ahn (Unlicensed)
@Christine Lenihan
@Ben Leibert
@Sebastian Brudziński
@Aleksandra Ciesińska
@Mateusz Kwiatkowski

Retrospective

It’s great that @Chongsun Ahn (Unlicensed) was able to finish the data pump so quickly.
@Aleksandra Ciesińska focused on supporting OpenLMIS-Angola, while @Mateusz Kwiatkowski concentrated on the longterm goal of introducing an aggregated approval page. Chongsun’s ticket was therefore, understandably, the only one were were able to close.

Showcase

The data pump syncs/streams data between the original requisition service and the new batch-requisition one. Our database image now includes Debezium, which uses Kafka to stream changes. You can currently see Kafka’s state, along with the data that it’s streaming, via port 8000. The port is publicly accessible at the moment, though can be essentially closed via AWS' security group and/or load balancer’s settings. There's no guarantee of the order in which streaming messages will be received. There can thus be foreign-key violations/exceptions if data arrives out of order and is attempted to be inserted into the new service accordingly. To handle this, we use multiple connectors configured to retry their insertions.

Confluent is a platform which has Zookeeper and several other things we're using. Kafta connect's port is 8083.

Kafka doesn’t just stream data - it streams schema changes as well. Although this allows it to automatically update schema within the target database, it has no way of knowing whether and how indexes should be applied within the updated tables. We therefore don’t rely on automatic schema updates, and instead manually make them via migrations.

Planning

The UAT is still slated for the 17th and 18th.
The primary ways in which the dev team can currently support the UAT is by focusing on https://openlmis.atlassian.net/browse/SELV3-148 , https://openlmis.atlassian.net/browse/SELV3-147, and https://openlmis.atlassian.net/browse/SELV3-150.
can be considered done.

Action Items

@Ben Leibert will create a ticket dedicated to restricting access to Kafka’s streaming data.

@Felimone Amone Junior will refine the users/roles defined with the CSV files given that they currently contain overlap.

@Felimone Amone Junior will provide CCE configuration. (Defining it is manual and very labor intensive process. Because our configuration tool doesn't currently support the upload of CCE configuration, the dev team will likely have to write a script to use whatever Felimone provides. Having the configuration several days prior to the UAT will thus be necessary to ensure that we can load it in time.)