WIP, Nick Reid (Deactivated) I'm just starting the brainstorming here.

Date

03 Jul 2017

Attendees

Goals

Discuss UI caching, what it means, performance v cost
Approach with Reference Data and Snapshotting which will have the best performance
UI as an App and last mile connectivity

Brainstorm

An idea that's been floating around is to cache large Reference Data lists, Orderables and associated entities, on the client-side device (browser or someday even an app perhaps):

A "working document" we'll define as a Requisition or an Order. It's something that a workflow is built around. For such working documents, we know they will need the details of Orderables, Facilities, Programs, etc.
Such data usually first is defined in Reference Data, however it's not always guranteed to be immutable - ie it may change.
The change is usually slow - Product lists, facilities, etc aren't updated daily or usually not even weekly. Annual cycles or more common SOP.
When every working document comes over the network with this reference data, the network payload quickly outstrips the reasonable bandwidth.

Could the UI store a list of this Reference Data, which it downloads over the network infrequently?

Lets take a Requisition's Line Item Orderables as an example. Here this information looks like:

{
"id" : "8c8026b8-0ae8-4adb-b2ca-bfb0dfb030a5",
    "orderable" : {
      "id" : "d602d0c6-4052-456c-8ccd-61b4ad77bece",
      "productCode" : "C100",
      "fullProductName" : "Levora",
      "netContent" : 84,
      "packRoundingThreshold" : 42,
      "roundToZero" : false,
      "programs" : [ {
        "programId" : "dce17f2e-af3e-40ad-8e00-3496adef44c3",
        "orderableDisplayCategoryId" : "15b8ef1f-a5d6-42dd-95bf-bb68a4504e82",
        "orderableCategoryDisplayName" : "Oral contraceptive",
        "orderableCategoryDisplayOrder" : 6,
        "fullSupply" : true,
        "displayOrder" : 1,
        "pricePerPack" : 5.23
      } ],
      "dispensable" : {
        "dispensingUnit" : "10 tab strip"
      }
    },
    "beginningBalance" : 50,
    "totalReceivedQuantity" : 0,
    "totalLossesAndAdjustments" : -5,
    "stockOnHand" : 45,
    "requestedQuantity" : 0,
    "totalConsumedQuantity" : 0,
    "requestedQuantityExplanation" : "we need more",
    "totalStockoutDays" : 0,
    "pricePerPack" : 5.23,
    "skipped" : false,
    "adjustedConsumption" : 0,
    "previousAdjustedConsumptions" : [ 50 ],
    "averageConsumption" : 25,
    "maxPeriodsOfStock" : 5.00,
    "stockAdjustments" : [ {
      "id" : "120557a7-1354-47b5-9579-8717324a3179",
      "reasonId" : "9e25fd93-d8a0-4d1d-8188-55dd3c408d24",
      "quantity" : 15
    }, {
      "id" : "ce50aa66-9465-4a0a-94c5-e389ecd97d45",
      "reasonId" : "71ee1dc3-5444-4a40-8b17-5676152e2ee1",
      "quantity" : 20
    } ]
  }

Minified this line item is 1.1KB.

Which could be further reduced:

{
"id" : "8c8026b8-0ae8-4adb-b2ca-bfb0dfb030a5",
    "orderable" : {
      "id" : "d602d0c6-4052-456c-8ccd-61b4ad77bece",
      "productCode" : "C100",
      "fullProductName" : "Levora",
      "netContent" : 84,
      "packRoundingThreshold" : 42,
      "roundToZero" : false,
        "orderableCategoryDisplayName" : "Oral contraceptive",
        "orderableCategoryDisplayOrder" : 6,
        "displayOrder" : 1,
        "pricePerPack" : 5.23
        "dispensingUnit" : "10 tab strip"
    },
    "beginningBalance" : 50,
    "totalReceivedQuantity" : 0,
    "totalLossesAndAdjustments" : -5,
    "stockOnHand" : 45,
    "requestedQuantity" : 0,
    "totalConsumedQuantity" : 0,
    "requestedQuantityExplanation" : "we need more",
    "totalStockoutDays" : 0,
    "pricePerPack" : 5.23,
    "skipped" : false,
    "adjustedConsumption" : 0,
    "previousAdjustedConsumptions" : [ 50 ],
    "averageConsumption" : 25,
    "maxPeriodsOfStock" : 5.00,
    "stockAdjustments" : [ {
      "id" : "120557a7-1354-47b5-9579-8717324a3179",
      "reasonId" : "9e25fd93-d8a0-4d1d-8188-55dd3c408d24",
      "quantity" : 15
    }, {
      "id" : "ce50aa66-9465-4a0a-94c5-e389ecd97d45",
      "reasonId" : "71ee1dc3-5444-4a40-8b17-5676152e2ee1",
      "quantity" : 20
    } ]
  }

Minified this version is 964B

Option A: Reference Data is Immutable

With this option, any change to something in Reference Data (e.g. an Orderable, TradeItem, Facility, etc) would be treated as:

mark existing record as archived
create new record (with new UUID), linking back to previous record (if any)

The benefit to this is that if you've downloaded, or store a reference to any Reference Data UUID, you never have to update it. You may have to check if it's archived, and then optionally go and retrieve the new object.

For the UI applied to Requisitions (presume similar to other working documents):

Download the full list of FTAP WITH eagerly fetched Orderables to Programs (5000 items x 3 deep)
With a HEAD request to the respective endpoints, discover if this list has changed, if it has, download the new list (infrequent).
Any change means this list gets bigger and bigger. If everything changed every year, after 3 years this list is 15,000 items x 3 deep.

Option B: Variant of A, however pull product list based on an "as of" date or range

This variant of A attempts to limit the ever-growing Reference Data lists present in Option A by placing a time-window of interest in front of those lists.

For the UI applied to Requisitions, we might make the modification:

Download the full list of FTAP WITH eagerly fetched Orderables to Programs (5000 items x 3 deep) AS OF the the date June 5th UTC. This will exclude all records that aren't active any longer as of June 5th.
- /api/facilityTypeApprovedProducts?asOf=20170605

Option C: Working Document Snapshot is cached

The working document's Service takes a full snapshot of the Reference Data it uses (Orderables, facilities, etc). With this option, Reference Data is mutable, it may change at any time. The Requisition Service (the working documents service) would take a snapshot of that Reference Data, and store it locally forever for all of the Requisition's it has.

For the UI applied to Requisitions:

Download the Requisition's Snapshot cache of Line Item Orderables (X items where X is the number of FTAP (Orderables x Program x Type) that have been used in all requisitions, columns targetted to the ones the Requisition document needs)
When a Requisition is initiated, the list from Ref Data FTAP is fetched, and Requisition's Snapshot of FTAP (down through the Orderable defintion) is placed in the snapshot "dictionary". If nothing has changed, nothing needs to be added.
What would such a Requisition RESTful endpoint look like?
- /api/requisition/{id}/orderables
  - HEAD

Discussion items

Time	Item	Who	Notes

OpenLMIS

2017-0X-X TBD UI Caching Meeting notes