Replace ViralAI With Internal Solution

[Buwujiu]

The issue

ViralAI is tasked with producing lineage calls for the clinical data, which the iMS portal ingests for public usage.
However ViralAI seems not to be working as expected recently.

Tasks

1. iMS portal should ingest the lineage calls produced by ViralAI.

• pedigree gets the csv (generated by ViralAI) from google storage, and updates iMS data using song and maestro.
• this process starts every Tuesday and Friday nights, when we produce a "release bundle", which ViralAI uses to produce the calls.
• after iMS ingests those calls, all samples submitted by "release" date should have lineage name in the explorer.

2. After iMS portal ingests the latest lineage calls, we will replace ViralAI with an in-house workflow that reproduces the same lineage calls. The benefits of replacing are: 1. faster at making the lineage calls public. 2. fewer 3rd party dependencies.
   - this is more of a devops task, not so much dev work.
   - @justincorrigible and @leoraba to provide a proposal, based on which we will evaluate effort and timeline.

[justincorrigible]

Lineage Call processing proposal

general overview

SFU's script produces a csv with the same data that comes from ViralAI
This script consumes the release bundle -> could consume directly from Arranger/Elasticsearch for efficiency using the following SQON:

{
  "content": {
    "fieldName":"analysis.lineage_analysis.lineage_name",
    "value": ["__missing__"]
  },
  "op":"in"
}

Full integration steps

1. deploy the new lineage call script

• workflow repo
• create docker image. who does this?
• account for pangolin updates

2. Figure out how to trigger this new pod

• ideally, kafka upon new submissions (maestro when done indexing)
• when pangolin updates, it needs "all data" in case there's definition changes.
• http endpoint doable with cron, as a fallback approach

3. publish file to s3 bucket

• ideally, update pedigree to consume data from our S3 buckets
• publish to GStorage, as a fallback approach

4. Pedigree consumes the data

• ideally GStorage sends a ping to portal-gateway. trigger options to research
• which in turn sends a kafka message that scales pedigree up.
• keep pedigree on a cron-job, as a fallback approach

[Buwujiu]

Justin J's proposing to store the lineage calls here: https://github.com/bfjia/Duotang_LineagePipeline/tree/main/latest

[Buwujiu]

TO get a better estimate from Justin R on timeline.

[Buwujiu]

Justin will focus on this later this week

[Buwujiu]

Justin R and Justin J tested this in dev with fake data, working as expected. Next step is to test properly in DEV using real prod data, Edmund and Justin J can help.

[Buwujiu]

lineage consumption works, we are ready to replace viral AI.