India is a huge country. Creating solutions for the world’s second largest population comes with its own set of challenges that need to be dealt with through capacity building, scale operations and interoperable frameworks that work seamlessly for citizens across the economy. Developing digital infrastructure to drive innovation and to build applications for multiple use-cases is fundamental for its success, acceptance and usage. The shared vision of DIVOC is to enable countries to digitally orchestrate large scale programs with an ability to allow the centre and states to operate in a federal structure to manage the roll out.

Usually, any large scale vaccination program has the following 3 key components:

  1. the supply chain which entails the logistics, storage, distribution, availability, etc.
  2. capacity building for on-ground support which would include trained front-line health personnel; and
  3. rollout and certification followed by continuous monitoring and feedback cycles.

In many cases, while there are options for the first 2 components, most countries lack any kind of digital infrastructure for supporting the third. This is where DIVOC plays a vital role in helping countries move from the traditional paper-based systems to a new-age digital system. Traditional systems such as those used in malaria or the yellow fever vaccination programs are meant for a small subset of the population. But for vaccination programs that are to be administered across the country’s larger population, one would need a systemic approach to build a platform that is resilient to react to the dynamic needs of the citizens the platform supports. This is exactly what DIVOC addresses.

Some foreseen technical challenges –

  • Large scale vaccination platform requires multiple components e.g. field orchestration, certificate generation, monitoring, etc., some of which not only need to work independently but also seamlessly.
  • Almost every component would need to be scalable, be it vaccine availability, vaccination, certificate generation and verification. Hence, scalability had to be baked into the design from the beginning.
  • As the platform is used at national level where the usage of the app is very high, it should always be available. Hence, the components had to be designed in such a way that they are independently scalable and fault-tolerant.
  • The vaccination program needs to be conducted even in remote/low network areas, so the platform needs to have capability to reliably work offline without losing data.
  • The platform needs to capture PII details of every individual who gets vaccinated, which is almost the entire adult (and maybe children too) population of the country. Hence, data security of information is crucial and has to be trustworthy.
  • Vaccination programs, typically, are managed by the government for public health. Hence, there is an expectation that the platform be compliant with WHO specifications.

Based on what countries may require in the coming year to run vaccination programs, DIVOC was architected with an approach to allow for a system to

  1. Work at scale (e.g. 10 million vaccinations / day).
  2. Be elastic (from 10,000 vaccinations to 10 million vaccinations / day).
  3. Be a digital-first design (not about codifying a paper-based system).
  4. Be agile and responsive given the dynamic nature of such large scale programs.
  5. Provide for real-time observability — to allow for a deeper and better understanding of what’s working and what’s not working so that the policy makers can make the necessary changes without any lag.

In summary, DIOVC is a platform to orchestrate the complexity within an ecosystem of diverse players to administer a vaccine program across a country successfully.

DIVOC has 3 key modules which can be used independently or can be integrated with other external systems.

A. Orchestration Module — this is for capturing details about the vaccine, health care facilities, vaccinators, recipients and matching time availability across facilities to potential recipients. The dynamic data driven orchestration is primarily possible from –

  1. Registries and Control Parameters — which is used to define the actors (who) and the activity (where and when) involved in the program.
  2. Telemetry driven analytics — real time observability of what is happening on the ground.
  3. Operational and Effectiveness Feedback — to be able to improve and redefine the control parameters in the first point.

B. Certification Module — this generates the digitally verifiable certificate based on the details captured from the Orchestration Module. The digital certificate is designed to

  • Conform to WHO guidelines which are based on the W3C credential specification.
  • Verify authenticity of the certificate via a digitally signed QR code.
  • Be available to citizens either on their smartphones or as a printable version of the same (for those without smartphones).
  • Support multilingual templates.
  • Integrate with health lockers or other digital certificate lockers.
  • Facilitate easy distribution of certificates via various channels.
  • Receive feedback from citizens using one-click link via the QR code.
  • Provide one-click mechanism via the QR code to manage distribution of authentic information.

C. Feedback Module — this allows recipients of the vaccine to provide one-click immediate feedback and receive authentic preliminary information and guidance based on the symptoms.

Technology Architectural principles

Scalability

Scalability in terms of supporting a varying number of certificates getting issued; large countries like India clocked a maximum of 20 million certificates a day and smaller deployments required a couple of thousands a day. Also supporting elasticity to effectively use resources as most vaccination programs used in day time, infrastructure during off-hours scales down to save the resources.

Security

Certificates usually carry a tag of authenticity and credibility with them. In case of paper certificates, it is easy to forge the document or the signatures, but in the case of a digital certificate, maintaining the same level of authenticity and credibility was important. This was done using cryptographic digital certificates. To support offline verification of certificates, DIVOC uses a public and private key based cryptography. Similar technology is often used in blockchain.

Modularity

There are 5 key components in DIVOC — orchestration, registration, certificate generation, certificate verification and feedback. Each of these components was built as an independent set of services to support easy integration and sharing of certificates. Each of these services was designed in a generic manner to make it usable across a wider range of use-case that countries might have. For example, when international travel started to open up across the globe, there would be a need to accommodate and verify certificates as per that country’s travel guidelines. This is where the modular design helps to plug-in the different components as Lego blocks. This kind of modularity also allowed countries to use either one of the components based on their need and deploy the same in a short period of time. It also allowed for easy integration with the current systems that countries were using.

Distributed System

Data can be managed with local centres, it needn’t be connected to one central system. This allows for continuity in the vaccination process and the workflow, even if one of the components / centre has lost connectivity. Each component of the system runs in a distributed fashion. This kind of distributed service allows a country to run deployments across different states at the same time.

Reusability

DIVOC is made of small reusable components so that when used in different countries with different contexts, one should be able to quickly stitch together the different components. From the ground up, it was thought of as a set of components that would come together to form the solution.

Compliance with globally accepted standards

Keeping a modular architecture design in mind also allowed for interoperability and compliance with international standards such as that of W3C, DDCC and OAuth based security exchange between third party APIs. This enabled integration with other actors within the digital healthcare system of that country.

Offline Capability

Given the sketchy connectivity in some of the remote areas, DIVOC had to have the capability to support when devices were offline. The key activity in such large scale programs is to vaccinate individuals, thus technology being a hindrance due to lack of connectivity should not be a reason to allow for workflow continuity. Hence, a lot of attention was given to design a system to support such offline scenarios. While offline, vaccination and recipient data can be recorded and then synced with the main database once online.

Adaptability of the Certificate design

The adapter component in the micro-services architecture allowed a country to generate different certificate based on the state / district’s needs. Swapping this component with an existing one to be able to render in the local script of that country was also possible without much redevelopment effort.

Architecture Diagram

Challenges and how did DIVOC team overcome them

  1. No available standards — the only reference was the 2006 WHO guidelines on the international vaccination certificate which was mainly used for paper based forms (yellow fever / yellow card). There wasn’t anything in the digital format. These paper based forms lacked the authenticity and verification part which is now possible with the digital certificates.
  2. Competing health related protocols were not Covid-19 specific. The existing core immunisation certificate was limited to only one type of certificate. Divoc is designed to adapt to international specifications and the team worked with international authorities such as WHO to build the specifications. For more details please refer to —
    https://www.who.int/publications/i/item/WHO-2019-nCoV-Digital_certificates-vaccination-2021.1
  3. Representing a Verifiable Credential (VC) as a QR Code in the certificate was consuming large space because of the size of the VC. There were usability issues to read the large sized QR codes. And hence we had to introduce compression on VC so we can reduce the size to be within 1KB.

A. Tech Stack used in this open-source Digital Public Good (DPG)

B. The approach taken to handle country scale vaccination campaigns –

  • Each component was developed as a micro service so that it can be built, deployed and scaled independently.
  • Following APIs had to handle high requests/sec
    1. Vaccination API had to handle more than 1 million requests per hour. This API would be called every time a person gets vaccinated at a vaccination centre. In order to handle this, the platform uses an asynchronous processing approach, allowing the platform to handle a large number of concurrent requests. When this API is called, a message would be enqueued on the message broker (kafka topic partitioned by mobile number). Concurrent consumers would then pick up and process this information. There are also different types of consumers (running in parallel) interested in processing this information, e.g. sign & record the vaccination fact to persistent storage, analytics, update distributed cache, notifications etc. Necessary error handling is in place to handle any failures while processing the message by any consumer.
    2. Certificate download API also had to handle large amounts of concurrent requests. To tackle this, Divoc uses the caching technique, wherein it would place certificate data (in json format) in cache. In order to keep cache size under check, it would store only those users’ certificates who got vaccinated in the last one week. Certificate generation consumer consumes vaccination message from message broker and generates vaccination certificate as json blob. The certificate data would be stored in persistence storage and also placed in cache (redis).
    3. Vaccination slots availability API is also heavily used. To handle this platform again relies on in-memory cache where curated information about available vaccination slots is stored.
  • ElasticSearch is used to handle any search functionality e.g. searching vaccination centres by pincode, searching specific vaccination centres / hospitals etc.

C. Security

  • OAuth 2.0 protocol is used to secure APIs
  • JWT is issued to authenticated users; each JWT token will have only required and necessary claims which can restrict what operations user can perform
  • Platform uses keycloak as authentication server
  • Any PII information sent to registry is masked

D. Production deployment configuration

  • Platform uses a kubernetes cluster with an auto-scaling feature.
  • The number of pods would vary between 5 to 20 on any given day
  • Platform can be deployed on a public cloud or internal data centre

E. Other notes

  • Platform uses json schema to handle any data customization (e.g. address details) from country to country
  • Performance testing was done using Gatling to make sure the platform can handle expected load.
  • Platform uses OpenAPI spec encouraging API first approach.

DIVOC, true to its vision, can enable countries to orchestrate large scale programs beyond the realm of health to benefit a large number of people — a truly powerful tool for future generations.

Contributors:
Dileep Bapat
Tejash JL
Samarth GR
Tejas Varade
Jayalaxmi
Nikita Oliver
Kshitij Sawant
Rohit Bansal
Sai Sathvik