Requisition Batch Approve service design meeting (Kafka + Debizium)

Attendees

• @Mateusz Kwiatkowski

• @Klaudia Pałkowska (Deactivated)

• @Aleksandra Ciesińska

• @Chongsun Ahn (Unlicensed)

• @Wesley Brown

• @Sebastian Brudziński

• @Josh Zamor

• @Ben Leibert

Notes

• Project Casper - streaming architecture to sync OLMIS v1/v2 with OLMIS v3

• 3 parts:

  • source system (requisition db)

  • Kafka

  • target system (new Requisition Batch service)

• https://casper-elmis.a.openlmis.org

• https://casper-elmis.a.openlmis.org:8000

• Debizium is required to be installed on requisition db

• topics in Kafka

• Kafka connect - a wrapper around Kafka for defining connectors, config has a connection to the source database, whitelist contains all tables that we want to watch, those would correspond to topics in Kafka

• Kafka and Kafka connect are established for reporting stack so we can use its docker-compose as an example

• feed vs debizium

• would be nice to include 3 services (kafka, kafka connector and zookeeper) into the OpenLMIS docker-compose file

• it would be nice to have a service that data would be streamed into and could be used for demo purposes

Next steps

• when we could move those services into core

• write tickets

1. add kafka and others to OpenLMIS Ref Distro.

2. start building new service.

3. stream the changes out of Requisition service.

4. stream data into batch service (aggregate on write).

5. remove kafka and others from reporting stack, as it's now in ref distro.

https://gitlab.com/openlmis/casper-pipeline

https://gitlab.com/openlmis/casper-pipeline/-/tree/master

https://openlmis.atlassian.net/wiki/spaces/OP/pages/507806293