NFR - Refinement

1 Profiles and Working Data Definitions
2 Performance
- 2.1 Matrics and benchmarks
  - 2.1.1 First Contentful Paint (FCP)
  - 2.1.2 Largest Contentful Paint (LCP)
  - 2.1.3 First Input Delay (FID)
- 2.2 Server Hardware Profile
3 Test Scenario and response times
- 3.1 Scalability
- 3.2 Availability
- 3.3 Network Usage
- 3.4 Notes

Profiles and Working Data Definitions

OpenLMIS is designed to be used in environments with differing levels of connectivity and by users that will use different sizes of working data. e.g. off-line or in-process stored locally and / or that a user works with at any given time. Focusing on the biggest set of data we provide the desired results for those smaller chunks of stored data.

The table below shows data volumes for each implementation and for the current test seed data on performance server.

Data name / Implementation	Angola	Gambia	Malawi	Mozambique (Selva)	Max	Performance Test Server	Proposed (old)	Long-Term Proposal (old)

Data name / Implementation	Angola	Gambia	Malawi	Mozambique (Selva)	Max	Performance Test Server	Proposed (old)	Long-Term Proposal (old)
Facilities	3280	155	1639	2021	3280	2734	4,000	10000
Facility Operators	0	0	6	2	6	2
Facility Type Approved Products	29900	15119	25972	721	29900	20293
Facility Types	21	11	24	20	24	6	10	15
Geographic Levels	4	4	4	4	4	4
Geographic Zones	3432	51	38	2017	3432	205
Orderable Display Categories	10	17	42	4	42	12
Orderables	9884	10054	11128	180	11128	10234
Processing Periods	58	44	393	102	393	434
Processing Schedules	4	2	2	3	4	6
Program Orderables	9931	7149	11522	192	11522	10236
Programs	12	7	11	2	12	8	10
Requisition Group Program Schedules	219	56	332	342	342	1012
Requisition Groups	171	8	135	342	342	510
Roles	11	14	23	12	23	18
Stock Adjustment Reasons	0	0	71	0	71	0
Supervisory Nodes	346	8	174	342	346	511
Supply Lines	218	56	174	342	342	1007
Trade Items	4536	4526	3908	30	4536	22
Users	681	496	2042	618	2042	1209	1000	5000
Valid Destination Assignments	8056	1667	21688	57055	57055	0
Valid Reason Assignments	2526	1070	3163	529	3163	0
Valid Source Assignments	8645	1699	1953	49446	49446	0
Other							Full-supply: 500 Non-full supply: 3,000	Full-supply: 2,000 Non-full supply: 10,000

Performance

Matrics and benchmarks

Operation	Success (p90)	Needs Improvement	Fail (p90)
GET	<= 1.5 seconds	1-5 to 2.5 seconds	>= 2.5 seconds
POST	<= 2 seconds	2 to 3 seconds	>= 2.5 seconds
PUT	<= 2.5 seconds	2-5 to 4 seconds	>= 2.5 seconds
DELETE	<= 1.5 seconds	1-5 to 2.5 seconds	>= 2.5 seconds
Complex Tasks	<= 3 seconds	3 to 5 seconds	>= 2.5 seconds

This is a proposition for the requests. Based on this we have to match it into the specific flow. Note that it focuses on single request to detect any bottlenecks and doesn’t give info on a whole flow which can contain few requests at the same time.

For flow web performance we use:

Metric	Good	Needs Improvement	Poor
First Contentful Paint (FCP)	<=1.8 seconds	1.8 to 3.0 seconds	>= 3.0 seconds
Largest Contentful Paint (LCP)	<=2.5 seconds	2.5 to 4.0 seconds	>= 4.0 seconds
First Input Delay	<=100 ms	100 to 300 ms	>= 300 ms

First Contentful Paint (FCP)

Measures when the first visible element appears on the screen, giving users a sense that the page is loading.

Largest Contentful Paint (LCP)

To provide a good user experience, sites should strive to have Largest Contentful Paint of 2.5 seconds or less. To ensure you're hitting this target for most of your users, a good threshold to measure is the 75th percentile of page loads, segmented across mobile and desktop devices.

First Input Delay (FID)

FID measures the time from when a user first interacts with a page (that is, when they click a link, tap on a button, or use a custom, JavaScript-powered control) to the time when the browser is actually able to begin processing event handlers in response to that interaction.

Server Hardware Profile

This requires to be defined again. We have to gather information about each instance of OpenLMIS we have and check if this is still relevant: link to recommendations

Test Scenario and response times

Use Case/Scenario	Test Parameters	Relevant Working Data	Release v3.18.0				Proposed
Use Case/Scenario	Test Parameters	Relevant Working Data	p90 (ms)	Error Rate	FCP	LCP	p90 (ms)	Error Rate	FCP	LCP
Initiate Requisition	Concurency: 1 Ramp-up time: - Iterations: 10 Hold-for: 2m	Performance Test Seed Data 12.2024	1350	0	-	-	<= 2 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Save (sync) Requisition	Concurency: 1 Ramp-up time: - Iterations: 10 Hold-for: 2m	Performance Test Seed Data 12.2024	617	0	-	-	<= 2 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Submit Requisition	Concurency: 1 Ramp-up time: - Iterations: 10 Hold-for: 2m	Performance Test Seed Data 12.2024	727	0	-	-	<= 2 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Authorize Requisition	Concurency: 1 Ramp-up time: - Iterations: 10 Hold-for: 2m	Performance Test Seed Data 12.2024	1457	0	-	-	<= 2 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Approve Requisition	Concurency: 1 Ramp-up time: - Iterations: 10 Hold-for: 2m	Performance Test Seed Data 12.2024	9512	0	-	-	<= 2 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Batch Requisition Approval	Concurency: 1 Ramp-up time: - Iterations: 10 Hold-for: 2m	Performance Test Seed Data 12.2024	-	0	-	-	<= 2 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Convert to Order (one)	Concurency: 1 Ramp-up time: - Iterations: 10 Hold-for: 2m	Performance Test Seed Data 12.2024	6353	0	-	-	<= 2 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Convert to Order (multiple)	Concurency: 1 Iterations: 1	Performance Test Seed Data 12.2024	2220	0	-	-	<= 3 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Filter performance on the convert to order page	Concurency: 1 Iterations: 1	Performance Test Seed Data 12.2024	1590	0	-	-	<= 1.5 seconds	0.1%	<=1.8seconds	<=2.5 seconds
View Requisition	Concurency: 1 Iterations: 1	Performance Test Seed Data 12.2024	364	0	0.790	2.503	<= 1.5 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Filter performance	Concurency: 1 Iterations: 1	Performance Test Seed Data 12.2024	594	0	0.343	0.343	<= 1.5 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Get Period For Initiate	Concurency: 1 Ramp-up time: - Iterations: 10 Hold-for: 2m	Performance Test Seed Data 12.2024	249	0	-	-	<= 1.5 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Get Supplying Depot	Concurency: 1 Ramp-up time: - Iterations: 10 Hold-for: 2m	Performance Test Seed Data 12.2024	332	0	-	-	<= 1.5 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Delete Requisition	Concurency: 1 Ramp-up time: - Iterations: 10 Hold-for: 2m	Performance Test Seed Data 12.2024	480	0	-	-	<= 2 seconds	0.1%	<=1.8seconds	<=2.5 seconds
Reject Requisition	Concurency: 1 Ramp-up time: - Iterations: 10 Hold-for: 2m	Performance Test Seed Data 12.2024	6431	0	-	-	<= 2 seconds	0.1%	<=1.8seconds	<=2.5 seconds

Endpoints that do not meet response time criteria:

Approve Requisition
Convert To Order (one) - Similarly, for multiple orders at once, it may be misleading that they were not tested under the same load as for a single order.
Reject Requisition

Scalability

Availability

Network Usage

Notes

How the system handles peak performance? How it recovers?
How do we maintain performance in life cycle of the application?
How to automate the process to fetch information from CI/CD into final report sheet?
Link to google sheets to store data from tests per release: https://docs.google.com/spreadsheets/d/1Ny5X0dfzAWNYU8kzJ2rdTFpqPs83RpMdyuH3O6ML2Gk/edit?usp=sharing
Link to data volumes and requisition usage from logs: https://docs.google.com/spreadsheets/d/1-NCfeuv2WZRxmoTu4KR6JFK684ybtWOxa6N8rC7yAhA/edit?usp=sharing