Copenhagen Vaccine Module Prioritization Workshop Report

Where: Copenhagen, UN City

When: May 2-4, 2017

Table of Contents

Overview

With 27 attendees, this 3-day workshop had three objectives: (1) Create a shared understanding of the purpose of the Vaccine module; (2) Prioritize module features and the associated user stories for vaccine development; and (3) Generate enthusiasm from participants to continue engagement with the work even after workshop. The facilitators and VillageReach team generally agreed at our closing meeting that the workshop was successful in achieving these objectives; feedback evaluation forms completed by participants matched this observation, with all responses indicating "very satisfied" or "satisfied" with meeting objectives.

A significant amount of effort was put into planning this workshop, which was paid off with a strong event with very involved break-out sessions and lively dialogue. The event was noteworthy for strong engagement (with closed laptops over 3 days) and a low attrition rate from attendees. Most importantly, this event lays a strong path forward for the OpenLMIS team to propel development of the vaccine module ahead.

^{(photos from Josh Zamor album}^{; photo by Alice Fu)}

What's Next?

Information collected at the event is being cataloged into the OpenLMIS wiki and will be distilled into roadmap plans and software assignments after input from the OpenLMIS governing committees. The team will follow up on key open items and leverage the expertise of the group and networks to further refine the user stories. See the Key Topics and Outcomes section for more details on next steps.

In addition, a regional user-centered design workshop will be held in West Africa in June 2017. This User Design Workshop will take place the last week of June 2017 (3 days to be confirmed during the week of June 26-30) in Dakar, Senegal. This workshop will be an opportunity for VillageReach and the OpenLMIS software development team to collect feedback from country EPI staff on specific user needs and for participants to work together within the design framework to inform the optimal user experience for the OpenLMIS vaccine functionality system.

While the Copenhagen prioritization workshop focused on understanding high-level system functionality needs, the User Design Workshop will conduct activities focused on designing appropriate user interfaces (UI) for the end users of the software. Without incorporating user needs into the design of the OpenLMIS tool, the system will not be successful in improving existing EPI logistics processes. In order to address user needs and ensure relevancy for a variety of country users, VillageReach will conduct this user design workshop in Dakar, Senegal to work with country-level users to co-create the user experience for vaccine functionality. VillageReach will facilitate the workshop in French and will lead participants through various design activities over the course of the three days. The outcome of this 3-day workshop will be a set of design recommendations, mock-ups, and user personas (idealized system users) that will directly inform development of the user interface for the OpenLMIS vaccine functionality. For more information on the Senegal workshop you can check on the Senegal User Design Confluence page as planning is ongoing: Senegal User Design Workshop

In addition to the Senegal workshop, the OpenLMIS team is also looking into smaller engagements in English with specific country teams to collect similar feedback. These would also take place in summer 2017.

^{(photos from Josh Zamor album}^{; photo by Josh Zamor)}

Key Topics Discussed and Outcomes

The following section is for the community to think through and determine the best way to address and follow up going forward to ensure the sucess of adoption of OpenLMIS in supporting the management of the vaccine supply chain.

Sustainability of OpenLMIS: Participants raised questions on the OpenLMIS business model and long-term sustainability of OpenLMIS. We arranged a special all-group session to discuss this on Day 3. Kaleb Brownlow (Gates) and Brian Taliesin (PATH) lead the discussion. Brian highlighted the Digital Health Initiative (DHI) program being launched by PATH. Kaleb introduced the Governance Committee and ongoing discussion around potential business models by the community and specifically at VillageReach.
- Attendees suggested many creative ideas: a revenue model where countries sell their OpenLMIS stock data to drug manufacturers; sliding-scale pricing modeled off of GAVI drug pricing where countries get free use or pay for OpenLMIS depending on their level of development; and more traditional open source dual licensing revenue models.
- Next Steps: VillageReach, led by Jake Watson, is planning a meeting at the end of May to continue the discussions around sustainability of the OpenLMIS community.
Programmatic Data Collection: Throughout the workshop the participants raised the importance of collecting and using programmatic data alongside of logistical data to manage the program and supply chain effectively. The team expressed the importance of highlighting what data is needed at which point in the various workflows to ensure the appropriate technological solution is implemented. There were many conversations around integrating with DHIS2 and other tools to collect the data. At this point, the team is focused on gathering the needs and understanding the end user value. It is clear that whatever solution is chosen (integration or direct collection into OpenLMIS), it needs to be seamless and easy for EPI staff to enter and to obtain the data they need when they need it.
- Next Steps: The OpenLMIS team will continue to document and discover what data is needed when and the best way for it to be captured. Currently the OpenLMIS team is actively working on technical options for Program Data (e.g. https://openlmis.atlassian.net/browse/OLMIS-2323).
OpenLMIS 0.9, 1.x, 2.x and VIMS Forking: The OpenLMIS team provided an overview of the history of OpenLMIS and the various forks. However, it is clear that the message can be refined and clarified as many participants had questions and weren't clear on what it meant.
- Next Step: The OpenLMIS team will refine messaging around the different versions and country implementations.
Guinea OpenLMIS Version: JSI raised that Guinea will be one of the next implementers of OpenLMIS. The community discussed staying closely engaged so that Version 3 can incorporate the Guinea requirements and needs as the roadmap is built out. JSI under contract with SIAPS (USAID) will initially implement the eLMIS version of OpenLMIS and eventually upgrade to version 3 with support from BMGF.
Introducing OpenLMIS to Ministries: UNICEF staff raised the question on how how to advocate for OpenLMIS usage or how to know what country situations may be a good fit for OpenLMIS. Chris Wright (JSI) answered that it really comes down to the implementation partners and the technical assistance (TA) to help answer this. He also suggested adopting a "Certified Partners" model to help countries and others know which vendors to trust to do a proper implementation or give good advice. Another suggestion is that OpenLMIS could create a few country profiles showing different examples of how generic countries at different levels of supply chain professionalization can use OpenLMIS.
Vaccine Feature Roadmap: Based on the prioritization, VR will construct a roadmap and share with participants and OpenLMIS community. Participants voted on features they wanted to include within the MVP or identified features they thought were not needed within the MVP. There were a couple key points which will need further exploration and discussion.
- Next Step: Update participants and the Product Committee with outcomes of these discussions.
Regional workshop: The Senegal User Design Workshop will focus on activities around designing appropriate user interfaces (UI) for the end users of the software. This User Design workshop will take place in Dakar, Senegal the week of June 26. VillageReach will facilitiate this workshop in French and will work with country-level participants to co-create the user experience for vaccine functionality. The outcome of this 3-day workshop will be a set of design recommendations, mock-ups, and user personas (idealized system users) that will directly inform development of the user interface for the OpenLMIS vaccine functionality.
- Next Step: OpenLMIS team will invite participants and conduct the workshop in late June.

Feedback from Attendees

Evaluation forms from attendees submitted in person at the close of the workshop showed they were satisfied that objectives were achieved (n=12 of 20 non-VR attendees).

On the Net Promoter Score question, "How likely are you to recommend the OpenLMIS Vaccine Module for countries/programs you work with," the average score was 7 on a scale of 10; attendees that ranked it higher than 7 expressed that they were likely to recommend OpenLMIS Vaccine Module, while some attendees who were less likely stated that the limited scope of the discussed Minimal Viable Product (MVP) version was the key factor; a couple of attendees who gave a score less than 5 also stated that their score would likely increase after reviewing the final MVP. Graph below shows distribution of Net Promotor scores.

^{(photos from Josh Zamor album}^{; photo by Josh Zamor)}

Agenda

Day 1

Topic

Notes/Presentation

Greetings and introductions. Review of objectives and ground rules for the workshop.

Goals :

Shared understanding of the purpose served by the OpenLMIS Vaccine Module
Prioritization of which features and user stories to develop for the Vaccine Module
Enthusiasm from participants to continue engagement with the work even after workshop

Participants shared what they wanted to get out of the workshop. A couple are highlighted below :

Figuring out where to up GAVI's investments
Incorporating Tanzania's core requirements in OpenLMIS
Understand vision and goals
Common set of requirements
Consensus and alignment on core features

Introduction to the domains and the process of defining the minimal viable product (MVP). The presentation provides the overview of what workflows fall under each domain.

Country Profile Activity

Participants reviewed and commented on the generalized vaccine supply chains for two countries, A and B. The objective was to come to an agreement on what the 'usual' vaccine supply chain looks like to build shared understanding and orient participants for the upcoming efforts in outlining the key workflows and user stories. link to PDFs

Reporting and KPIs small group break out.

- - Stock Management

Attendees: Ryan McWhorter (UNICEF), Dorthe Konradsen (UNICEF), Teresa Ha (UNICEF), Kaleb Brownlow (Gates Foundation)

Facilitator: Brandon Bowersox-Johnson (VillageReach)

- - Forecasting and Estimation

Attendees: Ashraf Islam (JSI), Gemma Orta-Martinez (UNICEF), Jorgen Kofoed (UNICEF), Ian Sliney (Gavi), Josh Zamor (VillageReach)

Facilitator: Chris Wright (JSI)

KPIs:

Forecast Accuracy (National level is a must, lowers levels if resupply is by allocation/push)
Stock at Risk (projected stock outs or expiry alerts by location, antigen, and $ value) (all levels)

Reports:

Forecast per antigen based on any methodology: demographic, session size, historical consumption trend, or a mix of these (e.g. demographic first, adjusted by historic consumption trend to keep it reality based in order to avoid overstock).
1. If using consumption, must adjust for reporting rate from facilities if less than 100%
Forecast accuracy, by level, time period, antigen, volume/quantity, and $ value. Compares forecast to actual consumption (adjusted at national level for reporting rate) for a specific period of time, and presents the data as a percentage (e.g. 93% accuracy means actual consumption is 93% or forecasted consumption). Also displays the $ value of the difference (7% below forecast = $X.00 worth of product(s) not used)
Pipeline report (national level is a must, other levels nice to have if allocation/push system): SOH, future stock position based on AMC rate (including coverage and wastage), and planned inbound shipments.
1. AMC must take into account reporting rate from facilities if less than 100%

- - Resupply

Attendees: Amy Roberts (CHAI), Justin Lorenzon (Field Partners), Fesha Getahun (JSI Ethiopia), Alpha Omari (JSI Tanzania)

Facilitator: Vidya Sampath (VillageReach)

KPIs:

On-Time and In-Full Deliveries (DISC) (to be available for all hierarchies within OpenLMIS – sub-national to district to H
Distribution Costs (note, voted down on Day 2)
# of Requisitions
1. WHAT: # of requisitions in a time period over expected #
  WHY needed: tells you if efficiencies are to be had in your requisition planning (if you're making more requisitions than policy during each supply period, then increase your allotted requisitions or how much you requisition each time, etc).

Reports:

OTIF report, with reasons for mismatch

This is also referred to as Fulfillment Report, which should be dis-aggregated by reason for no or partial fulfillment (reasons such as CCE failure, low stock at resupply level etc)
i. Related to the above, we identified the following information that is needed to get the OTIF indicator:

KPI name

Calculation details

Why needed

Who needs it (& how often)

What else

Order + Item Fill Rate

Item Fill: # of items received/ # of items orderd

Order Fill: average of each line item (qty supplied / qty ordered) expressed as %

To monitor and esure that fulfilment equals order quantities (Assuming order accuracy)

DIVO and SM levels above (frequency matches the re-supply period)

The report showing this indicator should be sites listed ranked by their rate (low to high)

No Visit Reason Report

Details the reasons why a direct delivery visit did not happen to the recipient site

Informs the potential root causes for a missed distribution

DIVO, EPI Program Supervisor (for both, match the distribution periods)

Show as pie chart with reasons breakdown2.

Disitributed Amounts Report - report detailing quantities distributed for a product, from which location, to which location
1. (cross refer with Supply Planning Report - doses issued to a site over a given period, disaggregated by product)
Vaccine Consumption Rate (doses consumed out of CCE/ doses available in CCE) - indicates how good supply planning is and whether allocation/requisition is adequate
Goods in Transit Report (only from sub-national to district, to indicate that large shipment is on its way and details how much of each product)

- - Cold Chain Equipment

Attendees: Mitesh (NexLeaf), Michelle Arnot-Kruger (UNICEF), Matt Morio (PATH), Mike Brison (CHAI), Prakash Vaidyanathan (UNICEF)

Facilitator: Rachel Powers (VillageReach)

Top KPIs:

Functionality Status of CCE (DISC KPI)
1. What it measures: Percentage of cold chain equipment (CCE) operable for vaccine storage of the overall number of commissioned CCE devices in a particular area at a point in time or over a particular time period. CCE is defined as all refrigerators, fixed passive storage devices, and walk-in cold rooms and freezers designated for storing vaccines.
2. Why it's important: This indicator measures operational cold chain equipment to identify risk of inadequate cold storage for maintaining potent vaccines. Used for operational and strategic purposes such as to ensure that vaccines are appropriately stored (e.g. to target deployment of repair/maintenance services) and to plan for replacement (e.g. to identify equipment in need of replacement).
3. Discussion also noted that for this KPI to be most effective, it should be paired with facility information around nonfunctioning CCE so that supervisory action can be taken.
CCE Capacity Available
1. What it measures: How many litres of storage are available in my supply chain, by facility
2. Dependencies: Functional status of CCE, information on specific CCE model capacity
3. Why it's important: This has several important uses. A. Should be used by EPI manager for planning new vaccine introduction. B. Should be used by logisticians to address influx of populations in catchment area and corresponding CCE capacity available (increase/decrease amount as needed) C. Should be used in resupply to understand capacity available at facilities so that excess product is not delivered

Other KPIs discussed:

Temperature Alarms (DISC KPI) - rates of low and high temperature alarms per reporting period

Top Reports:

Inventory Report
1. What: Query from back-end CCE inventory lists. Shows all CCE listed in supply chain and includes information on their: energy source, device name, serial #, installation date, warranty, capacity. This list would basically be a printout report of the combined CCE inventory lists from all facilities in a given area.
2. Why: A huge amount of resources have been spent on surveys of CCE in use in a country. This list would show all of the CCE for each facility and would be filterable pre-printout by facility, model type, energy source, etc.
"Naughty List" of facilities
1. What: A list of chronically poor-performing sites. Report tracks facilities that have experienced "out of range" (too hot or cold) temperatures/status within a certain period of time.
2. Why: Identifies facilities/districts/etc. that are having long-term, non-improving issues so that supervisory actions can be taken. This may include additional maintenance focus, retraining needs, or other supervisory actions.

Other reports discussed:

CCE Holdover Time: How long a CCE device can hold product at temperature without power
CCE Uptime: Percentage of time that the CCE holds product in the 2-8 degree range (or average hours per day within zone)
Min/Max Temperature: Report on min and max temperatures at facility (or average within district?) within a certain period of time
Replacement plan report
Repair parts needed report: Maintenance report for CCE technician to plan for repair trips & supplies
Power alarm: number of power alarms for facility / district within a certain period of time. (Power outages, etc.)

Recap and Feedback

Participants provided their thoughts on what worked or didn't work that day.

Day 2

Topic	Notes/Presentation
Review of the MVP process and objectives for the breakout sessions by domain	See the following notes on the discussions which took place.
Stock Management	Attendees: Ryan McWhorter (UNICEF), Dorthe Konradsen (UNICEF), Teresa Ha (UNICEF) Facilitator: Brandon Bowersox-Johnson (VillageReach) Key Activities discussed Dependencies & Configuration Enter stock data View stock data Alerts and notifications Barcodes Questions
Forecasting	Attendees: Ashra Islam (JSI), Gemma Orta-Martinez (UNICEF), Jorgen Kofoed (UNICEF), Alpha Omari (JSI) Facilitator: Chris Wright (JSI) The group discussed a wide range of user stories, key activities, and scenarios. The following is a brief summary of the two day discussion. Key Take aways: Annual national forecast needs to be entered into OpenLMIS rather than building in the various forecasting logic required, especially because there are different methodologies and sometimes mixed methods for determining forecast. But some information must also be included as text to capture the methodology used, the assumptions, and any constraints that affected the final quantity forecasted (e.g. budget limits or storage capacity). Forecast should be the true "constrained forecast" on which supply plans and forecast accuracy are based. OpenLMIS should enable upload of forecast for all levels if the program follows the traditional WHO methodology. If the annual forecast is used for allocation/push resupply to lower levels (the traditional WHO methodology), this process can be used for either routine or campaigns, since the target-based methodology is the same. However, campaigns are otherwise out of scope for MVP If the annual forecast is used to determine resupply throughout each tier of the supply chain using an allocation/push system, then pipeline views (future stock position based on forecast, current SOH, AMC, and planned resupply) are MVP must haves at each level in order to predict potential stock imbalances that risk stock out or expiry due to overstock anywhere within the system. This will inform resupply decisions (timing and quantities). If lower levels requisition, MVP functionality should provide each user a suggested resupply quantity based on: Max - (closing) SOH = qty to order. This is simple and takes into account any losses/adjustments due to transfers, expiry, damage, etc. However, this will work only if the MVP Max stock level is a dynamic figure (configuration) shown as months of stock (MOS) or weeks of stock (WOS), in which: MOS = AMC (over most recent # months) X resupply period (e.g. 3 months) + # weeks safety stock (should include buffer for fluctuations in demand and lead-time stock consumed between order and receipt).
Cold Chain Equipment (CCE)	Attendees: Mitesh (NexLeaf), Michelle (UNICEF), Matt (PATH), (Josh (VillageReach) for part of the day) Facilitator: Rachel Powers (Deactivated) (VillageReach) The group focused on the key activities: Create CCE catalog Add/View CCE inventory (specific CCE devices per facility) Capture Temperature & Functionality status Remote Temperature Monitoring (RTM) Alarms & Notifications Available Storage Capacity Repair & Maintenance
Re-supply	Attendees: Amy Roberts (CHAI), Justin Lorenzon (Field Partners), Fesha Getahun (JSI Ethiopia), Kaleb Brownlow (Gates Foundation) Facilitator: Vidya Sampath (VillageReach) The group focused on the key activities: View critical information in order to plan re-supply (prioritized list of elements in orange) Review SOH or Requested Quantities of the facility(ies) to resupply Set the 'fill'/'resupply' quantities (potentially) Print a pick pack list Confirm shipment amounts Generate an ePOD Notify the destination Receive stock (into the destination site/CCE, including if picking up products from level above) Reject stock
Team review of day's discussion and MVP scoping	Core team (VillageReach, Brian Taliesin, Chris Wright) reviewed the activities and stories within each domain. An initial line of the MVP was drawn.

Day 3

Topic	Notes/Presentation
Review and recap	Day 2 review and recap Participants took turns, global cafe/poster sessions style, to visit each domain's white board and quiz the facilitators on the discussions from the small group the previous day and the reasoning behind the core team's MVP line decision for that domain.
Prioritization	Participants voted on which user stories should be included in the Minimal Viable Product. They did this by placing their stickers on the user stories which fell below the line. If participants down-voted items that had been marked above the line, they were given an extra sticker to vote something up from below the line. Updates on the outcomes and final minimal viable product will be communicated to the community once analysis and estimations are done. The following photos demonstrate what the voting process looked like. The OpenLMIS team is currently working on digitizing the effort and conducting a more detailed estimation of level of effort in relation to the time and resources available.
Group discussion on sustainability, governance, and program data capture.	A majority of participants attended a break-out discussion on select 'bike rack' items: Sustainability, Governance, and Program Data capture. Kaleb Brownlow (Deactivated) started the conversation describing the governance structure and purpose of the committee. Explained how the community views OpenLMIS as a product and moving towards change and standards. There was good conversation on the long-term sustainability of OpenLMIS. Folks discussed the following topics: Spin off entity for OpenLMIS community/development What is the minimum support package. What does that look like? Cost and resources needed for ongoing support of the community and its members What types of business models could work? Brian Taliesin introduced the Digital Health Initiative with PATH. How do you create the environment where global goods are sustainable/successful? Key mechanisms of DHI: Pooled funding mechanism Global goods repository (what does this mean?) Community support Advocacy support A lively conversation ensued around program data collection and inclusion in OpenLMIS: The following opinions and points were raised by the group: It is very important that the SMT and DVDMT are replicated if country EPI are going to take up the OpenLMIS Desire to support current systems used in country and look for integration opportunities Important to feed data into the WHO reporting standards Country needs vary and we should take care of where they are in their system maturity Discussion on integration with DHIS2 both at the global and country level Mary Jo Kochendorfer (Deactivated) proposed an approach of identifying all the key programmatic data points and where in the workflows the data is used by end-users and when the data is consumed/reviewed by users. We would like to identify the key data points and when the data is needed and to whom (and in what way). The key approach is to identify the scope and value to determine the best way to approach the technical implementation. Josh Zamor conducted the other concurrent session, attended by Fesha Getahun and Kalkidan Kassahun, on the OpenLMIS technical committee as well as ways in which contributors can add code to the OpenLMIS software.
Overview of ways participants can engage with the OpenLMIS community.	The last side highlights key discussion items which will need follow up and continued discussion.
Note of thanks and feedback forms

Save