Define Requirements for Requisition Data Refresh

Description

We need to define the requirements around updating requisition data in the reporting stack. Ideally updates will increment to prevent us needing to drop and reload the table every time we want to update data (which is very resource intensive at scale).

We want to understand:

  • How to identify updated requisitions via the API (and which API)

  • How frequently to refresh the data (weekly, daily, real-time)

This will determine the LOE for configuring automatic data refreshes.

Activity

Show:
Clay Crosby
September 19, 2018, 8:19 AM

joshzamor [4 days ago]
@Clay: Agreed and we all (gap teams) agreed we needed it back in early Spring, however I’m not sure we said which team was responsible for making it happen, do you know?

joshzamor [4 days ago]
I know I’m down for supporting whichever team is going to get it done, so my question is focused really on: “which team”

Clay [< 1 minute ago]
In talking with @Craig Appl on this yesterday, websub is a T-shirt size Large task for us, which we do not have the bandwidth to complete by the 3.5 release

For now, we propose adding a modifiedDate to the requisitions API, so we can fetch all requisitions with a modifiedDate (datetime) greater than the last datetime we fetched requisitions

I think it would be to Team Parrot or Mind the Gap to make this addition to the API

For the 3.6 release, we can investigate moving to a websub paradigm

Clay Crosby
September 13, 2018, 12:46 PM

@here for the reporting stack, we want to incrementally update the requisitions table. How do we identify requisitions which have changed since the last time we fetched data? Are there any pertinent fields we can use in the API?

ngraczewski [4 hours ago]
Requisition model has a modifiedDate, would that be enough?

Clay [4 hours ago]
is there a way to pass modifiedDate as a parameter so we can fetch requisitions with a modifiedDate greater than the last date we pinged the API?

Clay [4 hours ago]
we’re going to need to do a similar thing for reference data and stock cards as well

ngraczewski [4 hours ago]
I don't see such option in the requisition API, but that should be reasonably easy to add, I suppose. What do you think @sbrudzinski?
CC @joshzamor

Clay [3 hours ago]
along these lines @joshzamor, perhaps now would be the ideal time to look into an ATOM feed or websub for all LMIS microservices in general — we will have the same needs around Reference Data and eventually stock cards, cce, etc. — but I’m not sure if the team has the bandwidth to set that up for the 3.5 release

if not, we can use NiFi to fetch data using the suggested new modifiedDate parameter for requisitions, and we can just do a full refresh (drop and reload) for reference data nightly or weekly

is modifiedDate date or datetime? (edited)

ngraczewski [3 hours ago]
It's date time.

Done

Assignee

Clay Crosby

Reporter

Clay Crosby

Labels

None

Story Points

1

Epic Link

Sprint

None

Fix versions

Affects versions

Priority

Major