Overview

The following document contains information about FHIR servers performance: Hearth and HAPI FHIR. Tests contain create, update and read actions on a location resource. Each section presents the request structure and response times. To execute tests, the Taurus tool has been used. The following machine has been used for tests:

Memory: 7.7 GiB
Processor: Intel Core i7-6700HQ 2.60GHz x 8
Graphics: GeForce GTX 960M/Pcie/SSE2
OS: Ubuntu 18.04.1 LTS (64-bit)

The following variables have been set and used to avoid duplication in tests. Some of them are used in request bodies.

fhir-server-url: http://localhost/fhir # Hearth
                 http://localhost/hapifhir/baseDstu3  # HAPI FHIR
scenario-iterations: # how many times a scenario/test will be executed
scenario-concurrency: 1 # how many users will execute the given scenario/test
identifier_value: # this value is generated for each location resource so searching by identifier always returns a single resource
location_id: # set by a FHIR server when a resource is created

The sample column presents the number of iterations/resources. The memory limit presents if memory limit has been set for FHIR servers and how big it was. The mem column presents how much memory was used by the docker container before and after all tests - that is why values are the same in all result tables. The avg column presents the average time of waiting for a response from all request. The p90 column presents times of 90% of all results (in this case 900 requests). Result times in columns are in milliseconds.

Create a new location

Note: When an X test is being executed, there are X - 1 resources in the FHIR server. For the first resource creation, the FHIR server will not contain any resources.

Request body

URL: ${fhir-server-url}/Location
Method: POST
Headers:
  Content-Type: application/json
Body:
  {
    resourceType: "Location",
	alias: [ "N007" ],
	identifier: {
	  { system: "urn:ietf:rfc:3986", value: "https://test.openlmis.org/api/programs/418bdc1d-c303-4bd0-b2d3-d8901150a983" },
	  { system: "urn:ietf:rfc:3986", value: "https://test.openlmis.org/api/facilityTypes/ac1d268b-ce10-455f-bf87-9c667da8f060" },
	  { system: "urn:ietf:rfc:3986", value: "https://test.openlmis.org/api/facilityOperators/9456c3e9-c4a6-4a28-9e08-47ceb16a4121" },
	  { system: "https://test.openlmis.org", value: ${identifier_value} }
	},
	name: "Lurio, Cuamba",
	position: {
	  longitude: 36.852509
	  latitude: -14.78975
	},
	physicalType: {
	  coding: [
	    { system: "http://hl7.org/fhir/location-physical-type", code: "si", display: "Site" }
	  ]
	},
	partOf: {
	  reference: "https://test.openlmis.org/api/Location/9b8cfb5a-217a-4261-a64f-16ca06ae79fa"
	},
	status: "active"
  }

Response times

FHIR server	Sample	Memory limit	1st attempt			2nd attempt			3rd attempt			4th attempt			5th attempt
FHIR server	Sample	Memory limit	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90
Hearth	1 000	no limit	-	3	4	-	3	4	-	3	5	-	3	5	-	3	4
HAPI FHIR	1 000	no limit	-	105	172	-	107	176	-	106	174	-	106	176	-	105	172
Hearth	10 000	no limit	158MiB/118MiB	2	4	175MiB/165MiB	2	4	172MiB/136MiB	3	4	-	-	-	-	-	-
HAPI FHIR	10 000	no limit	992MiB/1.399GiB	103	176	982MiB/1.442GiB	70	129	957MiB/1.357GiB	102	172	-	-	-	-	-	-

Update the location

Note: When an X test is being executed, there are X - 1 resources in the FHIR server. For the first resource update, the FHIR server will contain a single resource.

Request body

URL: ${fhir-server-url}/Location
Method: PUT
Headers:
  Content-Type: application/json
Body:
  {
    resourceType: "Location",
	alias: [ "N007" ],
	identifier: {
	  { system: "urn:ietf:rfc:3986", value: "https://test.openlmis.org/api/programs/418bdc1d-c303-4bd0-b2d3-d8901150a983" },
	  { system: "urn:ietf:rfc:3986", value: "https://test.openlmis.org/api/facilityTypes/ac1d268b-ce10-455f-bf87-9c667da8f060" },
	  { system: "urn:ietf:rfc:3986", value: "https://test.openlmis.org/api/facilityOperators/9456c3e9-c4a6-4a28-9e08-47ceb16a4121" },
	  { system: "https://test.openlmis.org", value: ${identifier_value} }
	},
	name: "Lurio, Cuamba (Updated)",
	position: {
	  longitude: 36.852509
	  latitude: -14.78975
	},
	physicalType: {
	  coding: [
	    { system: "http://hl7.org/fhir/location-physical-type", code: "si", display: "Site" }
	  ]
	},
	partOf: {
	  reference: "https://test.openlmis.org/api/Location/9b8cfb5a-217a-4261-a64f-16ca06ae79fa"
	},
	status: "active"
  }

Response times

FHIR server	Sample	Memory limit	1st attempt			2nd attempt			3rd attempt			4th attempt			5th attempt
FHIR server	Sample	Memory limit	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90
Hearth	1 000	no limit	-	5	6	-	5	7	-	5	7	-	5	7	-	4	5
HAPI FHIR	1 000	no limit	-	115	183	-	119	187	-	114	182	-	114	183	-	115	184
Hearth	10 000	no limit	158MiB/118MiB	12	19	175MiB/165MiB	13	20	172MiB/136MiB	13	20	-	-	-	-	-	-
HAPI FHIR	10 000	no limit	992MiB/1.399GiB	112	179	982MiB/1.442GiB	76	143	957MiB/1.357GiB	114	181	-	-	-	-	-	-

Get a location by an id

Note: When an X test is being executed, there are X - 1 resources in the FHIR server. For the first resource search, the FHIR server will contain only one resource.

Request body

URL: ${fhir-server-url}/Location/${location_id}
Method: GET
Headers:
  Accept: application/json

Response times

FHIR server	Sample	Memory limit	1st attempt			2nd attempt			3rd attempt			4th attempt			5th attempt
FHIR server	Sample	Memory limit	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90
Hearth	1 000	no limit	-	3	4	-	3	4	-	3	4	-	3	4	-	2	3
HAPI FHIR	1 000	no limit	-	5	9	-	6	10	-	5	9	-	5	9	-	5	9
Hearth	10 000	no limit	158MiB/118MiB	6	10	175MiB/165MiB	6	10	172MiB/136MiB	7	10	-	-	-	-	-	-
HAPI FHIR	10 000	no limit	992MiB/1.399GiB	4	8	982MiB/1.442GiB	4	7	957MiB/1.357GiB	4	7	-	-	-	-	-	-

Search single location by an identifier

Note: When an X test is being executed, there are X - 1 resources in the FHIR server. For the first resource search, the FHIR server will contain only one resource.

Request body

URL: ${fhir-server-url}/Location?identifier=https://test.openlmis.org|${identifier_value}
Method: GET

Response times

FHIR server	Sample	Memory limit	1st attempt			2th attempt			3th attempt			4th attempt			5th attempt
FHIR server	Sample	Memory limit	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90
Hearth	1 000	no limit	-	4	6	-	4	6	-	5	7	-	4	6	-	4	5
HAPI FHIR	1 000	no limit	-	16	24	-	17	26	-	17	25	-	17	25	-	17	25
Hearth	10 000	no limit	158MiB/118MiB	17	29	175MiB/165MiB	17	29	172MiB/136MiB	18	30	-	-	-	-	-	-
HAPI FHIR	10 000	no limit	992MiB/1.399GiB	14	20	982MiB/1.442GiB	10	19	957MiB/1.357GiB	13	19	-	-	-	-	-	-

Search locations

Note: When an X test is being executed, there are as many resources as the number in the sample column in the FHIR server. The search endpoint returns only 10 resources.

Request body

URL: ${fhir-server-url}/Location
Method: GET

Response times

FHIR server	Sample	Memory limit	1st attempt			2th attempt			3th attempt			4th attempt			5th attempt
FHIR server	Sample	Memory limit	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90
Hearth	1 000	no limit	-	3	4	-	3	5	-	3	4	-	3	5	-	3	4
HAPI FHIR	1 000	no limit	-	19	25	-	17	20	-	20	28	-	20	27	-	22	29
Hearth	10 000	no limit	158MiB/118MiB	3	4	175MiB/165MiB	3	4	172MiB/136MiB	3	4	-	-	-	-	-	-
HAPI FHIR	10 000	no limit	992MiB/1.399GiB	19	24	982MiB/1.442GiB	20	25	957MiB/1.357GiB	17	21	-	-	-	-	-	-

Search locations by name

Note: When an X test is being executed, there are as many resources as the number in the sample column in the FHIR server. The search endpoint returns only 10 resources.

Request body

URL: ${fhir-server-url}/Location?name:contains=Lurio
Method: GET

Response times

FHIR server	Sample	Memory limit	1st attempt			2th attempt			3th attempt			4th attempt			5th attempt
FHIR server	Sample	Memory limit	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90
Hearth	1 000	no limit	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
HAPI FHIR	1 000	no limit	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
Hearth	10 000	no limit	158MiB/118MiB	12	13	175MiB/165MiB	12	14	172MiB/136MiB	12	14	-	-	-	-	-	-
HAPI FHIR	10 000	no limit	992MiB/1.399GiB	19	24	982MiB/1.442GiB	19	23	957MiB/1.357GiB	19	24	-	-	-	-	-	-

Search locations by partOf

Note: When an X test is being executed, there are as many resources as the number in the sample column in the FHIR server. The search endpoint returns only 10 resources.

Request body

URL: ${fhir-server-url}/Location?partof=https://test.openlmis.org/api/Location/9b8cfb5a-217a-4261-a64f-16ca06ae79fa
Method: GET

Response times

FHIR server	Sample	Memory limit	1st attempt			2th attempt			3th attempt			4th attempt			5th attempt
FHIR server	Sample	Memory limit	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90
Hearth	1 000	no limit	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
HAPI FHIR	1 000	no limit	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
Hearth	10 000	no limit	158MiB/118MiB	11	12	175MiB/165MiB	11	13	172MiB/136MiB	11	13	-	-	-	-	-	-
HAPI FHIR	10 000	no limit	992MiB/1.399GiB	20	25	982MiB/1.442GiB	19	23	957MiB/1.357GiB	18	23	-	-	-	-	-	-

Search locations by status

Note: When an X test is being executed, there are as many resources as the number in the sample column in the FHIR server. The search endpoint returns only 10 resources.

Request body

URL: ${fhir-server-url}/Location?status:exact=active
Method: GET

Response times

FHIR server	Sample	Memory limit	1st attempt			2th attempt			3th attempt			4th attempt			5th attempt
FHIR server	Sample	Memory limit	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90	mem	avg	p90
Hearth	1 000	no limit	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
HAPI FHIR	1 000	no limit	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
Hearth	10 000	no limit	158MiB/118MiB	12	13	175MiB/165MiB	11	13	172MiB/136MiB	12	14	-	-	-	-	-	-
HAPI FHIR	10 000	no limit	992MiB/1.399GiB	20	26	982MiB/1.442GiB	18	23	957MiB/1.357GiB	17	20	-	-	-	-	-	-

Conclusions

In all attempts, both FHIR servers are very fast because all response times are lower than 200 milliseconds. The hearth is significantly faster for creation and update operations - all response times are lower or equals 20 milliseconds. For read and search operations, both FHIR servers are on par and generally, their times are sub-50ms for p90.

HAPI FHIR server requires more memory on the start but also during tests the memory usage is significantly higher over the Hearth server.

The backing data store characteristics (e.g. durability, consistency, memory, etc.) were not considered in tests.

Performance tests and FHIR servers were on the same machine so there are some notable limitations to this analysis:

network latency was not accounted,
variations in the host OS were not accounted for,
test scenarios are a limited to a basic load test that should be refined when the real world is better understood

read and search operations are likely highly impacted as the test, especially for HAPI FHIR server, is against Locations that are fully cached:

hapi_1                   | 2018-09-05 13:42:12.190 [http-nio-8080-exec-7] INFO  c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:151] Search entity marked as finished
hapi_1                   | 2018-09-05 13:42:12.217 [http-nio-8080-exec-5] INFO  c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:301] Reusing search f36b5bb4-d58c-4836-83fe-448225372c85 from cache
hapi_1                   | 2018-09-05 13:42:12.228 [http-nio-8080-exec-5] INFO  c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:151] Search entity marked as finished
hapi_1                   | 2018-09-05 13:42:12.245 [http-nio-8080-exec-8] INFO  c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:301] Reusing search f36b5bb4-d58c-4836-83fe-448225372c85 from cache
hapi_1                   | 2018-09-05 13:42:12.250 [http-nio-8080-exec-8] INFO  c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:151] Search entity marked as finished
hapi_1                   | 2018-09-05 13:42:12.265 [http-nio-8080-exec-1] INFO  c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:301] Reusing search f36b5bb4-d58c-4836-83fe-448225372c85 from cache
hapi_1                   | 2018-09-05 13:42:12.269 [http-nio-8080-exec-1] INFO  c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:151] Search entity marked as finished
hapi_1                   | 2018-09-05 13:42:12.288 [http-nio-8080-exec-4] INFO  c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:301] Reusing search f36b5bb4-d58c-4836-83fe-448225372c85 from cache

Overall these current performance metrics are within the expected needs of an FHIR server that is apart of OpenLMIS for interoperability.

Performance results

The following zip file contains all result data of all attempts. Those results are also available in the following repository.