We completed which will help production data sets, however for demo data we would still iterate over all the demo data records on the first start as they don't have snapshots yet. Is a slow startup still / more an issue? With the increased size of the demo data that we're adding, and thoughts to have millions of records, do we think we have an issue?
When Ref Distro is started, DB usage is sustaining 100% CPU for > 5 minutes
Quick update that I'm looking at Change Data Capture (with Debezium) as an approach to data pumps:
Still very much in exploration - no decision or course change yet. Though hope to get a sense for 3.6.
Streams changes from database, avoiding multiple writes (one to database, another as a published event)
CDC results in events in Kafka that look like the database, with schemas perhaps this is okay, however the event itself is often not the same higher-level semantic event we might think of (think of an Orderable vs. an Orderable with a Program Orderable).
One potential solution is to have a stream processor, perhaps even the service, which is responsible for making more semantic events for downstream clients.
Begs the question what, if anything, we'd use Nifi for.
(for next time)
DHIS2 followup on how (keeping this on the agenda to discuss the plan prior to implementation, however implementation isn't pressing on this just yet)
It sounds like not (if there is someone in a timezone where this becomes harder to attend because of daylight savings changes, please let us know and we'll try to accommodate)
Audit log initializer
Currently bogged down in CCE alerts under demo data (the developer workflow primarily)
Stock Mgmt screen was slower - not sure if this is from # of new orderables, or start up sequence of audit log initializer
Are we starting ref distro locally?
Some say yes
Only if changes in several services, otherwise push and see in test. Run ref distro to test demo data.
Run ref distro migration changes (working on Jenkins migration test problems) - running it locally.
In UI dev you don't need to run ref distro, the UI dev can work off a deployed cloud instance
Some / many aren't starting ref distro locally
Would be nice to catch build failures before they got pushed (and so would make sense if people weren't starting it locally)
Biggest problems:
time consuming to launch ref distro
configurations (after some months of not running, the changes to setup required debugging / migration)
Changing a single module (service?) - we don't have documentation to show how that's done.
Negative for marketing to new devs / users:
Native developer experience
e.g. IntelliJ idea running tests (would need to disable the login/auth, which might accidentally get committed, for integration tests)
Keep development instance more up to date so that we could build outside of docker (it's more efficient)
NodeJS issue with version 6 and version 9 - dev-ui is version 7, on mac only 6 and 9 are installable.
Complicated setup - settings.env and the profile that's used.
When services haven't all started on login, you'll get an error.
Big machine - Make services more efficient, explore service meshes (though would require big internet), etc
No longer extends base entity due to compound primary key
Foreign keys from other tables sometimes needs to also have a version indicator
Main lessons:
Our tools really don't like this: Hibernate and the ORM impedance mismatch
Composite keys are difficult, if we were to do this again, try to stay away from composite keys (there were a couple other potential solutions but couldn't get those to work)
Action Items
Josh Zamor to make dev forum post on pain points (and do this again on next TC - timebox but keep it going)
which will help production data sets, however for demo data we would still iterate over all the demo data records on the first start as they don't have snapshots yet. Is a slow startup still / more an issue? With the increased size of the demo data that we're adding, and thoughts to have millions of records, do we think we have an issue?