Expected Differences: vrMIS vs. SELV

Indicator

Expected Difference

Impacts Migrated Data from vrMIS?

Impacts Data After SELV Launch?

Explanation

Validation Approach

for Future Tests

Health Units Reporting

Should almost always be the same. SELV might be slightly less than vrMIS.

Yes, revised metric

The business rules is that a HU has reported if the HU was visited, or if routine data is reported. The two systems implement this in different ways:

vrMIS counts a HU as reporting if the HU was synched. The facility may have been visited, or not visited. In vrMIS, if you don't have any data you don't have to enter anything at all.
In SELV you have to enter data for each facility (for reporting to be accurate). SELV counts a HU as reporting if the facility was visited, or if it was not visited and some data was entered on the child coverage screen.

In most cases these two approaches will yield the same results, but if a facility was synched in vrMIS, but there was no data on the child coverage screen (all "N/R"), it will "count" as reported in vrMIS but not in SELV.

Identify the HU that shows as "Reported" in vrMIS but not SELV. Look in vrMIS and check if data for the facility was synched, but no data on child coverage screen. I

Tetanus Used

SELV will be higher than vrMIS

Yes, revised metric

vrMIS wasn't calculating this correctly. It should be the sum of all 1st and 2-5 doses administered to all groups of women.

Verify one health center manually

PCV Supplies Delivered and Full Deliveries

SELV will be less than vrMIS

Yes, revised metric

vrMIS had mistaken information on vial size (10 doses per vial instead of 2 doses per vial)

PCV Vial Size Info:

http://apps.who.int/iris/bitstream/10665/90378/1/WHO_IVB_13.09_eng.pdf

vrMIS PCV Supplies Delivered / 10 * 2 = SELV PCV Supplies Delivered

Child Coverage Rate

SELV will be higher than vrMIS

Yes, revised metric

vrMIS was erroneously including cases where no coverage data was reported in the overall coverage rate calculation. When no coverage data was reported, the target group number was still included in the overage coverage rate calculation. For example, if there are three HUs (note the row in pink is the source of the error):

	Doses Administered	Target Group for facility	Coverage Rate (vrMIS)	Coverage Rate (Correct)
HU1	10	100	10%	10%
HU2	null	100	0%	NULL
HU3	0	100	0%	0%
		Summary Coverage Rate for all 3 Health Units	10 + 0 / 300 = 3%	10 + 0 / 200 = 5%

Hand calculate correct coverage rate in excel and compare to Argentina's data

Child & Adult Coverage Rate

SELV will be much lower than vrMIS. This is only for data entered directly into SELV (NOT migrated from vrMIS)

No

The target group in SELV is the ANNUAL target rate, not the monthly target rate. This will be fixed in the next release.

Hand calculate the correct coverage rate and compare with vrMIS. Note that even after hand correcting the line item above (SELV higher than vrMIS) will still apply:

0-11 mo doses / (target group / 12)

Child & Adult Coverage Target

SELV may be 1 higher than vrMIS

Yes, revised metric

Yes, revised Metric

vrMIS always rounded child/adult coverage target down (floor function). Vidya checked and it should be a round, not a floor. This has been changed in the migrated data and going forward.

If same or 1 more than vrMIS, test passes.

PCV product deliveries

SELV values will be 0 for reports before August 2013 (? tbc)

Yes, revised metric

Need to check with Vidya, but no data was reported for that time period.

Check the date of the report.

% target group covered (? ) by EPI

1 point difference between values

Yes, revised metric

There is a rounding convention that differs between vrMIS and SELV. (Explanation to be expanded.)

ISA (on EPI Inventory Screen/Load Amounts Screen)

Vial ideal quantity in SELV may be 1 higher than vrMIS value for vial sizes > 1

No

Yes

SELV rounds up when converting from doses to vials, vrMIS rounds down.

Example: ISA doses is 124. Vial size is 10. vrMIS says # of vials for a full delivery is 12. SELV says # of vials for a full delivery is 13.

If SELV is within 1 of vrMIS, test passes.

ISA, Full Deliveries, Child Coverage Target - PCV10

ISA in SELV will be ~25% lower than vrMIS

No

Yes

vrMIS erronously was calculating PCV10 targets at 4 doses per year, not 3. Changed to 3 in SELV. This impacts the child coverage targets and the ideal quantity on the EPI Inventory and View Load Amounts screen

If SELV is ~25% lower than vrMIS targets, test passes.

ISA, Full Deliveries - Measles

vrMIS has a minimum number of doses of 50 (5 vials). SELV has no minimum.

No

Yes

vrMIS had a minimum number of measles doses in the ISA calculation, but no one knows why. We removed this in SELV. Thus in some cases the Ideal Quantity in vrMIS will be 5, but in SELV will be 1, 2, 3, or 4.

If vrMIS is 5 and SELV is less, test passes. For values >=5, vrMIS will be one more than SELV (see related rounding issue)

Full Deliveries Results for South-Niassa

In Tableau, there will be between a 10% and 20% under-reporting of Full Deliveries for some months for South Niassa, which will roll upwards to affect all-province wide calculations as well.

Yes

No

At some point in vrMIS, the facility ISA overrides for a few facilities in South-Niassa was set to Null. This meant that the ISA for these was non-existent. vrMIS concluded that any HU visited was then a full delivery, because it did not have an ISA to compare the delivered+existing stock amount to. Tablueau ignores those facilities that has a Null ISA value.

This does not impact data after SELV launches, because there are no facility overrrides of ISA as yet. If at some point in the future the overrides happen, it falls on the HSG team in Seattle and in Moz to ensure that no Null (or 0) ISA values are entered.

If performing facility overrides of ISA in SELV, ensure that no ISA is written as Null or 0 and that always a numerical value is uploaded.

Penta Wastage Rate Calculation (Open Vials)

In vrMIS, the open vial wastage rate calculations for Penta are much bigger than the calculations with the same data seen in Tableau.

Yes

No

In vrMIS, there was only one data field to enter open vials of vaccines, without distinguishing whether it was a 1-dose vial that was open or a 10-dose vial. vrMIS was coded to assume that all Penta vaccines opened were 1-dose vial, but from experience, it is known that that is incorrect, it should have been 10-dose vials. Hence the large open vial wastage rate for Penta in vrMIS. In Tableau, this error has been corrected, with all historical data for Penta open vials now coded as 10-dose vials. Going forward, this ambiguity will not be an issue because in SELV you can specify the vial size in the open vial column.

Ensure that vial size is specified and clear for open vials data column.