Add derived dataset provenance to the ro-crate

[jswelling]

This version of build_crate_from_dataset.py produces a fairly complete rocrate and croissant pair for derived datasets. For primary datasets, the croissant.json is fairly complete but the provenance in the rocrate is very incomplete. Also, for primary datasets with thousands of files, both json files will be unwieldy.

Call the routine as:

env AUTH_TOK=<valid token> python build_crate_from_dataset.py -o <output-dir> HMxxx.xxxx.xxx

This will produce a file HMxxx.xxxx.xxx_crate.zip in the given output directory. Supported options include:

--debug/-d for debugging information
--include-all-files to include all files for a derived dataset, not just data product/quality control files

To validate the result, cd to that directory and do:

rocrate-validator validate HBMxxx.xxxx.xxx_crate.zip
unzip HBMxxx.xxx.xxx_crate.zip
mlcroissant validate --jsonld croissant.json

The first command validates the ro-crate json; the second validates the croissant.

Some limitations of this version:

• For primary datasets, the ro-crate does not appropriately represent the provenance for primary datasets, but the croissant information does.
• The program references the UUID endpoint for file information.
• Only a subset of possible EDAM codes are supported. An unknown EDAM code will produce a warning.
• Inferring MIME types from EDAM codes is not always possible. The code should be expanded to include the file extension.
• Version numbers for some things (python, CWL, Airflow) are hard-coded.

[jpuerto-psc]

no. processed datasets get their own DOIs at publication time.

[jswelling]

I haven't found those DOIs in the entity-api data?

[jpuerto-psc]

Apologies, I'm wrong here, yes -we'd want to link back to the primary datasets for their DOIs. Asking around to see the reasoning for this

[jswelling]

On the croissant side, the place this is going is in the "citeAs" reference. If someone hits the DOI URL for the parent dataset, with the current structure of the Portal the relevant info for the derive dataset will also appear. This seems semantically iffy but not actually incorrect. If I naively substitute the DOI of the parent on the rocrate side, it will be tagged as the base identifier and as "sameAs" the derived dataset. That seems incorrect to me. So I propose to swap it in on the Croissant side, where it will avoid a warning, but to leave things as they are on the rocrate side.

[jpuerto-psc]

perhaps there is something we can pull out of the protocols.io information? Not sure that there is much else that we can throw in here otherwise.

[jswelling]

Yes, it can be built from protocols.io and the ancestors data. It's not ideal, because one has to treat things like running a sample through a microtome as a software operation because of the limitations of current workflow standards, but we could indeed built it. It just took me so long to get the derived dataset provenance to pass validation that I ran out of time!

[jpuerto-psc]

Maybe there's value to this down the line, but definitely not necessary for MVP. Seems more of a research-y topic.

[jpuerto-psc]

this might break for snare-seq data since it has multiple parent datasets

[jswelling]

I tried to be careful to keep things correct for multi-parent data, but I have not yet tested it against a published dataset of that type.

[jswelling]

Update: the croissant side does break for snare-seq data.

[jswelling]

Still pending:

• Do we use the parent DOI for the Croissant entry for derived datasets that have no DOI? If so, what do we use for snare-seq?
  Deferred for the future:
• snare-seq fails.
• Build proper provenance for primary datasets, perhaps using protocols.io data.