Javers log initializer - do not iterate over all items

Description

While https://openlmis.atlassian.net/browse/OLMIS-2534#icft=OLMIS-2534 made some improvements to the initialization of Javers tables, we should still consider making further improvements to make our startup time better.

Currently the log initializer retrieves all objects (for example requisitions) and iterates over them, initializing the audit log for the entries that do not have it (in a paginated manner):

https://github.com/OpenLMIS/openlmis-requisition/blob/master/src/main/java/org/openlmis/AuditLogInitializer.java

We should avoid iterating over hundreds of thousands of items during startup. There a few ways we can do this:

  • figure out a way to fetch only the items that do not have audit history. From my understanding it's not trivial, that's why it was not implemented - but please make sure you try to figure out a way, this is the cleanest solution possible.

  • consider using spring profiles to enable/disable this process

  • a date based mechanism, that uses timestamps to figure out the newly added items

  • possibly others

In any case, the acceptance criteria is:

  • Services do not iterate over all records for auditable tables at startup

  • Auditable records inserted by demo data or directly into the database will still have their audit logs initialized properly upon startup

Before starting implementation, suggest a solution on the mailing list

Activity

Show:
Mateusz Kwiatkowski
September 6, 2018 at 2:43 PM

Looked into all changed services and it seems it works fine. Moving to Done.

Josh Zamor
May 24, 2018 at 5:49 AM

Moving to Demo data epic as that epic comes with an increase in scale of demo-data for which this must be addressed. Also bumping priority.

Łukasz Lewczyński
August 31, 2017 at 9:21 AM

Note: The javers jv_global_id table constains in the local_id column id of object so for example to connect this table with requisitions we have to execute the following SQL statement:

SELECT * FROM requisition.requisitions r INNER JOIN requisition.jv_global_id g ON CAST(r.id AS varchar) = SUBSTRING(g.local_id, 2, 36) limit 10;
Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Labels

Story Points

Original estimate

Time tracking

2w 2d logged

Sprint

None

Fix versions

Priority

Time Assistant

Created August 26, 2017 at 8:59 AM
Updated October 30, 2018 at 4:44 AM
Resolved September 6, 2018 at 2:44 PM