Tess import

[paulzierep]

Add TESS training material importer with per-tool training ID outputs

Creates a new tess-import/ importer that fetches all training materials from the TESS API (tess.elixir-europe.org) and produces per-tool training material ID files for the RSEc content repo.

What it does:

• Paginates through all training materials on TESS (~3900)
• Fetches detailed metadata for each material (title, description, DOI, EDAM topics/operations, resource type, source node, keywords)
• Extracts tool names from external_resources where type == "tool" and URL points to bio.tools
• Maps tool names to data/{bt_id}/ directories and writes data/{bt_id}/{bt_id}.tess.json containing only a list of training material IDs
• Saves full entries to imports/tess/{id}.tess.json
• Supports --test [N] flag to limit materials for testing
• Uses parallel detail fetching (5 workers) for performance
• Outputs stats: materials processed, unique tools, resource type distribution (tutorial vs webinar etc.), and source node distribution (ELIXIR nodes)

Usage:

python3 tess-import/tess-import.py
python3 tess-import/tess-import.py --test 100

Format of per-tool files (e.g. data/fastqc/fastqc.tess.json):

["5451", "5432", "5389"]

Format of tess files (e.g. imports/tess/5451.tess.json):

{
    "authors": [
        "Diana Chiang Jurado"
    ],
    "contact": null,
    "contributors": [
        "Leonid Kostrykin",
        "Daniela Schneider",
        "Saskia Hiltemann",
        "Enis Afgan",
        "Diana Chiang Jurado",
        "Marius van den Beek"
    ],
    "date_created": null,
    "date_published": "2026-06-15",
    "date_updated": null,
    "description": "## Abstract\n\nManually scoring histological staining across dozens of images is time-consuming and subjective. Two researchers looking at the same slide may reach different conclusions about the amount of staining. Computational automated quantification solves this problem: it applies the same criteria to every image, produces a numeric result, and scales to large datasets without additional effort.\n\n\n## About This Material\n\nThis is a Hands-on Tutorial from the GTN which is usable either for individual self-study, or as a teaching material in a classroom.\n\n\n## Questions this  will address\n\n - How can I quantify the percentage of stained area in histological images?\n - How does color deconvolution separate individual stain components from brightfield microscopy images?\n - How can I apply this workflow to IHC stained tissue sections?\n\n\n## Learning Objectives\n\n- Apply color deconvolution to separate stain channels in histological images\n- Extract and isolate the stain channel of interest (e.g. DAB)\n- Apply automatic thresholding to distinguish stained from unstained regions\n- Calculate the percentage of positively stained area relative to total tissue area\n- Interpret quantitative staining results across multiple images",
    "difficulty_level": "beginner",
    "doi": null,
    "id": 5491,
    "keywords": [
        "Bioimaging",
        "Histology",
        "Imaging",
        "Microscopy",
        "pathology"
    ],
    "learning_objectives": "- Apply color deconvolution to separate stain channels in histological images\n- Extract and isolate the stain channel of interest (e.g. DAB)\n- Apply automatic thresholding to distinguish stained from unstained regions\n- Calculate the percentage of positively stained area relative to total tissue area\n- Interpret quantitative staining results across multiple images",
    "licence": "CC-BY-4.0",
    "mapped_tools": [
        "galaxy",
        "galaxy_image_analysis",
        "imagej"
    ],
    "nodes": [],
    "operations": [],
    "prerequisites": " * FAIR Bioimage Metadata\n * Introduction to Galaxy Analyses\n * REMBI - Recommended Metadata for Biological Images \u2013 metadata guidelines for bioimaging data",
    "resource_type": [
        "e-learning"
    ],
    "scientific_topics": [
        "Imaging"
    ],
    "source": "TESS",
    "status": "Active",
    "syllabus": null,
    "target_audience": [
        "Students"
    ],
    "title": "Quantitative Analysis of Histological Staining Using Color Deconvolution",
    "tools": [
        "galaxy_image_analysis",
        "galaxy",
        "imagej"
    ],
    "url": "https://tess.elixir-europe.org/materials/quantitative-analysis-of-histological-staining-using-color-deconvolution"
}

[mihai-sysbio]

Thanks @paulzierep , look cool! Is this ready to be reviewed - and would you know who best to tag?

[paulzierep]

@mihai-sysbio imo this is ready for review, maybe @hmenager can check if the logic makes sense (ids in data folder, full metadata in dataset folder), or I can present in the next meeting, toimorrow :) @arash77 maybe you can check the code logic, worked for me locally - the question would be how to process it downstream for the atlas, so that we can see for each tool, which training is using it, the training should include at least the link and name I guess

[arash77]

@mihai-sysbio imo this is ready for review, maybe @hmenager can check if the logic makes sense (ids in data folder, full metadata in dataset folder), or I can present in the next meeting, toimorrow :) @arash77 maybe you can check the code logic, worked for me locally - the question would be how to process it downstream for the atlas, so that we can see for each tool, which training is using it, the training should include at least the link and name I guess

Checked the logic, works locally.

Matches repo conventions. CI is red from pre-existing lint/format issues on main, not this PR.

On the atlas question: data/<bt_id>/<bt_id>.tess.json = material IDs for that tool; imports/tess/<mid>.tess.json = full record incl. url + title. So "which training uses a tool + link + name" = join the two. We keep per-tool files as small ID-lists; the Atlas resolves them at build time (same as galaxy).

Minor: requirements.txt lists boltons but it's unused — matches the other importers, harmless.