Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Date

Participants / Internal Stakeholders

Retrospective

Showcase

The data pump syncs/streams data between the original requisition service and the new batch-requisition one. Our database image now includes Debezium, which uses Kafka to stream changes. You can currently see Kafka’s state, along with the data that it’s streaming, via port 8000. The port is publicly accessible at the moment, though can be essentially closed via AWS' security group and/or load balancer’s settings. There's no guarantee of the order in which streaming messages will be received. There can thus be foreign-key violations/exceptions if data arrives out of order and is attempted to be inserted into the new service accordingly. To handle this, we use multiple connectors configured to retry their insertions.

Confluent is a platform which has Zookeeper and several other things we're using. Kafta connect's port is 8083.

Kafka doesn’t just stream data - it streams schema changes as well. Although this allows it to automatically update schema within the target database, it has no way of knowing whether and how indexes should be applied within the updated tables. We therefore don’t rely on automatic schema updates, and instead manually make them via migrations.

Planning

Action Items

  • Ben Leibert will create a ticket dedicated to restricting access to Kafka’s streaming data.

  • No labels