/
Copenhagen Vaccine Module Prioritization Workshop Report

Copenhagen Vaccine Module Prioritization Workshop Report

Aug 02, 2017

Where: Copenhagen, UN City

When: May 2-4, 2017 

Table of Contents

Overview

With 27 attendees, this 3-day workshop had three objectives: (1) Create a shared understanding of the purpose of the Vaccine module; (2) Prioritize module features and the associated user stories for vaccine development; and (3) Generate enthusiasm from participants to continue engagement with the work even after workshop. The facilitators and VillageReach team generally agreed at our closing meeting that the workshop was successful in achieving these objectives; feedback evaluation forms completed by participants matched this observation, with all responses indicating "very satisfied" or "satisfied" with meeting objectives.

A significant amount of effort was put into planning this workshop, which was paid off with a strong event with very involved break-out sessions and lively dialogue. The event was noteworthy for strong engagement (with closed laptops over 3 days) and a low attrition rate from attendees. Most importantly, this event lays a strong path forward for the OpenLMIS team to propel development of the vaccine module ahead.

(photos from Josh Zamor album; photo by Alice Fu)

What's Next? 

Information collected at the event is being cataloged into the OpenLMIS wiki and will be distilled into roadmap plans and software assignments after input from the OpenLMIS governing committees. The team will follow up on key open items and leverage the expertise of the group and networks to further refine the user stories. See the Key Topics and Outcomes section for more details on next steps.

In addition, a regional user-centered design workshop will be held in West Africa in June 2017. This User Design Workshop will take place the last week of June 2017 (3 days to be confirmed during the week of June 26-30) in Dakar, Senegal. This workshop will be an opportunity for VillageReach and the OpenLMIS software development team to collect feedback from country EPI staff on specific user needs and for participants to work together within the design framework to inform the optimal user experience for the OpenLMIS vaccine functionality system.

While the Copenhagen prioritization workshop focused on understanding high-level system functionality needs, the User Design Workshop will conduct activities focused on designing appropriate user interfaces (UI) for the end users of the software. Without incorporating user needs into the design of the OpenLMIS tool, the system will not be successful in improving existing EPI logistics processes. In order to address user needs and ensure relevancy for a variety of country users, VillageReach will conduct this user design workshop in Dakar, Senegal to work with country-level users to co-create the user experience for vaccine functionality. VillageReach will facilitate the workshop in French and will lead participants through various design activities over the course of the three days. The outcome of this 3-day workshop will be a set of design recommendations, mock-ups, and user personas (idealized system users) that will directly inform development of the user interface for the OpenLMIS vaccine functionality. For more information on the Senegal workshop you can check on the Senegal User Design Confluence page as planning is ongoing: Senegal User Design Workshop

In addition to the Senegal workshop, the OpenLMIS team is also looking into smaller engagements in English with specific country teams to collect similar feedback. These would also take place in summer 2017. 

(photos from Josh Zamor album; photo by Josh Zamor)

Key Topics Discussed and Outcomes

The following section is for the community to think through and determine the best way to address and follow up going forward to ensure the sucess of adoption of OpenLMIS in supporting the management of the vaccine supply chain. 

  • Sustainability of OpenLMIS: Participants raised questions on the OpenLMIS business model and long-term sustainability of OpenLMIS. We arranged a special all-group session to discuss this on Day 3. Kaleb Brownlow (Gates) and Brian Taliesin (PATH) lead the discussion. Brian highlighted the Digital Health Initiative (DHI) program being launched by PATH. Kaleb introduced the Governance Committee and ongoing discussion around potential business models by the community and specifically at VillageReach. 

    • Attendees suggested many creative ideas: a revenue model where countries sell their OpenLMIS stock data to drug manufacturers; sliding-scale pricing modeled off of GAVI drug pricing where countries get free use or pay for OpenLMIS depending on their level of development; and more traditional open source dual licensing revenue models. 

    • Next Steps: VillageReach, led by Jake Watson, is planning a meeting at the end of May to continue the discussions around sustainability of the OpenLMIS community. 

  • Programmatic Data Collection: Throughout the workshop the participants raised the importance of collecting and using programmatic data alongside of logistical data to manage the program and supply chain effectively.  The team expressed the importance of highlighting what data is needed at which point in the various workflows to ensure the appropriate technological solution is implemented.  There were many conversations around integrating with DHIS2 and other tools to collect the data. At this point, the team is focused on gathering the needs and understanding the end user value. It is clear that whatever solution is chosen (integration or direct collection into OpenLMIS), it needs to be seamless and easy for EPI staff to enter and to obtain the data they need when they need it.

    •  Next Steps: The OpenLMIS team will continue to document and discover what data is needed when and the best way for it to be captured. Currently the OpenLMIS team is actively working on technical options for Program Data (e.g. https://openlmis.atlassian.net/browse/OLMIS-2323).

  • OpenLMIS 0.9, 1.x, 2.x and VIMS Forking: The OpenLMIS team provided an overview of the history of OpenLMIS and the various forks. However, it is clear that the message can be refined and clarified as many participants had questions and weren't clear on what it meant.

    • Next Step: The OpenLMIS team will refine messaging around the different versions and country implementations.

  • Guinea OpenLMIS Version: JSI raised that Guinea will be one of the next implementers of OpenLMIS. The community discussed staying closely engaged so that Version 3 can incorporate the Guinea requirements and needs as the roadmap is built out. JSI under contract with SIAPS (USAID) will initially implement the eLMIS version of OpenLMIS and eventually upgrade to version 3 with support from BMGF. 

  • Introducing OpenLMIS to Ministries: UNICEF staff raised the question on how how to advocate for OpenLMIS usage or how to know what country situations may be a good fit for OpenLMIS. Chris Wright (JSI) answered that it really comes down to the implementation partners and the technical assistance (TA) to help answer this. He also suggested adopting a "Certified Partners" model to help countries and others know which vendors to trust to do a proper implementation or give good advice. Another suggestion is that OpenLMIS could create a few country profiles showing different examples of how generic countries at different levels of supply chain professionalization can use OpenLMIS. 

  • Vaccine Feature Roadmap: Based on the prioritization, VR will construct a roadmap and share with participants and OpenLMIS community. Participants voted on features they wanted to include within the MVP or identified features they thought were not needed within the MVP. There were a couple key points which will need further exploration and discussion. 

    • Next Step: Update participants and the Product Committee with outcomes of these discussions.

  • Regional workshop: The Senegal User Design Workshop will focus on activities around designing appropriate user interfaces (UI) for the end users of the software. This User Design workshop will take place in Dakar, Senegal the week of June 26. VillageReach will facilitiate this workshop in French and will work with country-level participants to co-create the user experience for vaccine functionality. The outcome of this 3-day workshop will be a set of design recommendations, mock-ups, and user personas (idealized system users) that will directly inform development of the user interface for the OpenLMIS vaccine functionality.

    • Next Step: OpenLMIS team will invite participants and conduct the workshop in late June.

Feedback from Attendees

Evaluation forms from attendees submitted in person at the close of the workshop showed they were satisfied that objectives were achieved (n=12 of 20 non-VR attendees).

On the Net Promoter Score question, "How likely are you to recommend the OpenLMIS Vaccine Module for countries/programs you work with," the average score was 7 on a scale of 10; attendees that ranked it higher than 7 expressed that they were likely to recommend OpenLMIS Vaccine Module, while some attendees who were less likely stated that the limited scope of the discussed Minimal Viable Product (MVP) version was the key factor; a couple of attendees who gave a score less than 5 also stated that their score would likely increase after reviewing the final MVP. Graph below shows distribution of Net Promotor scores.

(photos from Josh Zamor album; photo by Josh Zamor)

Agenda

Day 1 

Topic

Notes/Presentation

Topic

Notes/Presentation

Greetings and introductions. Review of objectives and ground rules for the workshop.

Goals :

  • Shared understanding of the purpose served by the OpenLMIS Vaccine Module

  • Prioritization of which features and user stories to develop for the Vaccine Module

  • Enthusiasm from participants to continue engagement with the work even after workshop

Participants shared what they wanted to get out of the workshop. A couple are highlighted below :

  • Figuring out where to up GAVI's investments

  • Incorporating Tanzania's core requirements in OpenLMIS

  • Understand vision and goals

  • Common set of requirements

  • Consensus and alignment on core features

Introduction to the domains and the process of defining the minimal viable product (MVP). The presentation provides the overview of what workflows fall under each domain.

Country Profile Activity

Participants reviewed and commented on the generalized vaccine supply chains for two countries, A and B. The objective was to come to an agreement on what the 'usual' vaccine supply chain looks like to build shared understanding and orient participants for the upcoming efforts in outlining the key workflows and user stories. link to PDFs

Reporting and KPIs small group break out.

Attendees: Ryan McWhorter (UNICEF), Dorthe Konradsen (UNICEF), Teresa Ha (UNICEF), Kaleb Brownlow (Gates Foundation)

Facilitator: Brandon Bowersox-Johnson (VillageReach)



Attendees: Ashraf Islam (JSI), Gemma Orta-Martinez (UNICEF), Jorgen Kofoed (UNICEF), Ian Sliney (Gavi), Josh Zamor (VillageReach)

Facilitator: Chris Wright (JSI)

KPIs:

  1. Forecast Accuracy (National level is a must, lowers levels if resupply is by allocation/push)

  2. Stock at Risk (projected stock outs or expiry alerts by location, antigen, and $ value) (all levels)

Reports: 

  1. Forecast per antigen based on any methodology: demographic, session size, historical consumption trend, or a mix of these (e.g. demographic first, adjusted by historic consumption trend to keep it reality based in order to avoid overstock).

    1. If using consumption, must adjust for reporting rate from facilities if less than 100%

  2. Forecast accuracy, by level, time period, antigen, volume/quantity, and $ value. Compares forecast to actual consumption (adjusted at national level for reporting rate) for a specific period of time, and presents the data as a percentage (e.g. 93% accuracy means actual consumption is 93% or forecasted consumption). Also displays the $ value of the difference (7% below forecast = $X.00 worth of product(s) not used)

  3. Pipeline report (national level is a must, other levels nice to have if allocation/push system): SOH, future stock position based on AMC rate (including coverage and wastage), and planned inbound shipments.

    1. AMC must take into account reporting rate from facilities if less than 100% 

Attendees: Amy Roberts (CHAI), Justin Lorenzon (Field Partners), Fesha Getahun (JSI Ethiopia), Alpha Omari (JSI Tanzania)

Facilitator: Vidya Sampath (VillageReach)

KPIs:

  1. On-Time and In-Full Deliveries (DISC) (to be available for all hierarchies within OpenLMIS – sub-national to district to H

  2. Distribution Costs (note, voted down on Day 2)

  3. # of Requisitions

    1. WHAT: # of requisitions in a time period over expected #
      WHY needed: tells you if efficiencies are to be had in your requisition planning (if you're making more requisitions than policy during each supply period, then increase your allotted requisitions or how much you requisition each time, etc).

Reports:

 

Attendees: Mitesh (NexLeaf), Michelle Arnot-Kruger (UNICEF), Matt Morio (PATH), Mike Brison (CHAI), Prakash Vaidyanathan (UNICEF)

Facilitator: Rachel Powers (VillageReach)

Top KPIs:

  1. Functionality Status of CCE (DISC KPI)

    1. What it measures: Percentage of cold chain equipment (CCE) operable for vaccine storage of the overall number of commissioned CCE devices in a particular area at a point in time or over a particular time period. CCE is defined as all refrigerators, fixed passive storage devices, and walk-in cold rooms and freezers designated for storing vaccines.

    2. Why it's important: This indicator measures operational cold chain equipment to identify risk of inadequate cold storage for maintaining potent vaccines. Used for operational and strategic purposes such as to ensure that vaccines are appropriately stored (e.g. to target deployment of repair/maintenance services) and to plan for replacement (e.g. to identify equipment in need of replacement).

    3. Discussion also noted that for this KPI to be most effective, it should be paired with facility information around nonfunctioning CCE so that supervisory action can be taken.

  2. CCE Capacity Available

    1. What it measures: How many litres of storage are available in my supply chain, by facility

    2. Dependencies: Functional status of CCE, information on specific CCE model capacity

    3. Why it's important: This has several important uses. A. Should be used by EPI manager for planning new vaccine introduction. B. Should be used by logisticians to address influx of populations in catchment area and corresponding CCE capacity available (increase/decrease amount as needed) C. Should be used in resupply to understand capacity available at facilities so that excess product is not delivered

Other KPIs discussed:

  1. Temperature Alarms (DISC KPI) - rates of low and high temperature alarms per reporting period

Top Reports:

  1. Inventory Report

    1. What: Query from back-end CCE inventory lists. Shows all CCE listed in supply chain and includes information on their: energy source, device name, serial #, installation date, warranty, capacity. This list would basically be a printout report of the combined CCE inventory lists from all facilities in a given area.

    2. Why: A huge amount of resources have been spent on surveys of CCE in use in a country. This list would show all of the CCE for each facility and would be filterable pre-printout by facility, model type, energy source, etc.

  2. "Naughty List" of facilities

    1. What: A list of chronically poor-performing sites. Report tracks facilities that have experienced "out of range" (too hot or cold) temperatures/status within a certain period of time.

    2. Why: Identifies facilities/districts/etc. that are having long-term, non-improving issues so that supervisory actions can be taken. This may include additional maintenance focus, retraining needs, or other supervisory actions.

Other reports discussed:

  1. CCE Holdover Time: How long a CCE device can hold product at temperature without power

  2. CCE Uptime: Percentage of time that the CCE holds product in the 2-8 degree range (or average hours per day within zone)

  3. Min/Max Temperature: Report on min and max temperatures at facility (or average within district?) within a certain period of time

  4. Replacement plan report

  5. Repair parts needed report: Maintenance report for CCE technician to plan for repair trips & supplies

  6. Power alarm: number of power alarms for facility / district within a certain period of time. (Power outages, etc.)

Recap and Feedback

Participants provided their thoughts on what worked or didn't work that day.

OpenLMIS: the global initiative for powerful LMIS software