...

The sample column presents the number of iterations/resources. The memory limit presents if memory limit has been set for FHIR servers and how big it was. The mem column presents how much memory was used during the testby the docker container before and after all tests - that is why values are the same in all result tables. The avg column presents the average time of waiting for a response from all request. The p90 column presents times of 90% of all results (in this case 900 requests). Result times in columns are in milliseconds. 

...

URL: ${fhir-server-url}/Location
Method: POST
Headers:
  Content-Type: application/json
Body:
  {
    resourceType: "Location",
	alias: [ "N007" ],
	identifier: {
	  { system: "urn:ietf:rfc:3986", value: "https://test.openlmis.org/api/programs/418bdc1d-c303-4bd0-b2d3-d8901150a983" },
	  { system: "urn:ietf:rfc:3986", value: "https://test.openlmis.org/api/facilityTypes/ac1d268b-ce10-455f-bf87-9c667da8f060" },
	  { system: "urn:ietf:rfc:3986", value: "https://test.openlmis.org/api/facilityOperators/9456c3e9-c4a6-4a28-9e08-47ceb16a4121" },
	  { system: "https://test.openlmis.org", value: ${identifier_value} }
	},
	name: "Lurio, Cuamba",
	position: {
	  longitude: 36.852509
	  latitude: -14.78975
	},
	physicalType: {
	  coding: [
	    { system: "http://hl7.org/fhir/location-physical-type", code: "si", display: "Site" }
	  ]
	},
	partOf: {
	  reference: "https://test.openlmis.org/api/Location/9b8cfb5a-217a-4261-a64f-16ca06ae79fa"
	},
	status: "active"
  }

Response times

Update the location

Note: When an X test is being executed, there are X - 1 resources in the FHIR server. For the first resource update, the FHIR server will contain a single resource.

...

URL: ${fhir-server-url}/Location
Method: PUT
Headers:
  Content-Type: application/json
Body:
  {
    resourceType: "Location",
	alias: [ "N007" ],
	identifier: {
	  { system: "urn:ietf:rfc:3986", value: "https://test.openlmis.org/api/programs/418bdc1d-c303-4bd0-b2d3-d8901150a983" },
	  { system: "urn:ietf:rfc:3986", value: "https://test.openlmis.org/api/facilityTypes/ac1d268b-ce10-455f-bf87-9c667da8f060" },
	  { system: "urn:ietf:rfc:3986", value: "https://test.openlmis.org/api/facilityOperators/9456c3e9-c4a6-4a28-9e08-47ceb16a4121" },
	  { system: "https://test.openlmis.org", value: ${identifier_value} }
	},
	name: "Lurio, Cuamba (Updated)",
	position: {
	  longitude: 36.852509
	  latitude: -14.78975
	},
	physicalType: {
	  coding: [
	    { system: "http://hl7.org/fhir/location-physical-type", code: "si", display: "Site" }
	  ]
	},
	partOf: {
	  reference: "https://test.openlmis.org/api/Location/9b8cfb5a-217a-4261-a64f-16ca06ae79fa"
	},
	status: "active"
  }

Response times

Get a location by an id

Note: When an X test is being executed, there are X - 1 resources in the FHIR server. For the first resource search, the FHIR server will contain only one resource.

...

URL: ${fhir-server-url}/Location/${location_id}
Method: GET
Headers:
  Accept: application/json

Response times

Search single location by an identifier

...

Search single location by an identifier

Note: When an X test is being executed, there are X - 1 resources in the FHIR server. For the first resource search, the FHIR server will contain only one resource.

...

URL: ${fhir-server-url}/Location?identifier=https://test.openlmis.org|${identifier_value}
Method: GET

Response times

Search locations

Note: When an X test is being executed, there are 1000 resources in the FHIR server. The search endpoint returns only 10 resources.

Request body

URL: ${fhir-server-url}/Location
Method: GET

Response times

Search locations

Note: When an X test is being executed, there are as many resources as the number in the sample column in the FHIR server. The search endpoint returns only 10 resources.

Request body

URL: ${fhir-server-url}/Location
Method: GET

Response times

Search locations by name

Note: When an X test is being executed, there are 1000 resources there are as many resources as the number in the sample column in the FHIR server. The search endpoint returns only 10 resources.

...

URL: ${fhir-server-url}/Location?name:contains=Lurio
Method: GET

Response times

Search locations by partOf

...

Search locations by partOf

Note: When an X test is being executed, there are 1000 resources there are as many resources as the number in the sample column in the FHIR server. The search endpoint returns only 10 resources.

...

URL: ${fhir-server-url}/Location?partof=https://test.openlmis.org/api/Location/9b8cfb5a-217a-4261-a64f-16ca06ae79fa
Method: GET

Response times

Search locations by status

...

Search locations by status

Note: When an X test is being executed, there are as many resources as the number in the sample column in the FHIR server. The search endpoint returns only 10 resources.

...

URL: ${fhir-server-url}/Location?status:exact=active
Method: GET

Response times

Conclusions

In all attempts, the Hearth FHIR server has better performance than the HAPI FHIR server. Creation and update actions take a lot of more time in the HAPI FHIR than in the Hearth. The search actions have better results but still, Hearth server is faster about 4 times. I noticed that the HAPI FHIR server caches results (not know for how long) for search requestsboth FHIR servers are very fast because all response times are lower than 200 milliseconds. The hearth is significantly faster for creation and update operations - all response times are lower or equals 20 milliseconds. For read and search operations, both FHIR servers are on par and generally, their times are sub-50ms for p90.

HAPI FHIR server requires more memory on the start but also during tests the memory usage is significantly higher over the Hearth server.

The backing data store characteristics (e.g. durability, consistency, memory, etc.) were not considered in tests.

Performance tests and FHIR servers were on the same machine so there are some notable limitations to this analysis:

• network latency was not accounted,
• variations in the host OS were not accounted for,
• test scenarios are a limited to a basic load test that should be refined when the real world is better understood

• read and search operations are likely highly impacted as the test, especially for HAPI FHIR server, is against Locations that are fully cached:

  Code Block

  hapi_1 | 2018-09-05 13:42:12.190 [http-nio-8080-exec-7] INFO c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:151] Search entity marked as finished hapi_1 | 2018-09-05 13:42:12.217 [http-nio-8080-exec-5] INFO c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:301] Reusing search f36b5bb4-d58c-4836-83fe-448225372c85 from cache hapi_1 | 2018-09-05 13:42:12.228 [http-nio-8080-exec-5] INFO c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:151] Search entity marked as finished hapi_1 | 2018-09-05 13:42:12.245 [http-nio-8080-exec-8] INFO c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:301] Reusing search f36b5bb4-d58c-4836-83fe-448225372c85 from cache hapi_1 | 2018-09-05 13:42:12.250 [http-nio-8080-exec-8] INFO c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:151] Search entity marked as finished hapi_1 | 2018-09-05 13:42:12.265 [http-nio-8080-exec-1] INFO c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:301] Reusing search f36b5bb4-d58c-4836-83fe-448225372c85 from cache hapi_1 | 2018-09-05 13:42:12.269 [http-nio-8080-exec-1] INFO c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:151] Search entity marked as finished hapi_1 | 2018-09-05 13:42:12.288 [http-nio-8080-exec-4] INFO c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:301] Reusing search f36b5bb4-d58c-4836-83fe-448225372c85 from cache

  
  

Overall these current performance metrics are within the expected needs of an FHIR server that is apart of OpenLMIS for interoperability.

Performance results

The following zip file contains all result data of all attempts. Those results are also available in the following repository.