updating Add Dataset for links

.github/ISSUE_TEMPLATE/new_dataset.yaml

@@ -46,7 +46,7 @@ body:
     id: url
     attributes:
       label: --URL--
-      description: Please provide a URL where the file can be downloaded. A link ending in the file name is split into the directory it is served from and the name; a link that names no file - a Google Drive `uc?export=download&id=` link, or anything else with a query string - is kept whole, and then the file name has to be given below.
+      description: Please provide a URL where the file can be downloaded. A link ending in the file name is split into the directory it is served from and the name; a link that names no file - a Google Drive `uc?export=download&id=` link, or anything else with a query string - is kept whole, and then the file name has to be given below. A Google Drive share link, `file/d/<id>/view` or `open?id=<id>`, works too - it is rewritten to the download link that serves the file.
       placeholder: ex. https://zenodo.org/records/15490547/files/smallPtychography.hspy
     validations:
       required: true
@@ -62,10 +62,10 @@ body:
     id: checksum
     attributes:
       label: --Checksum--
-      description: Please provide the checksum of the dataset file for verification purposes. Include the type of checksum (e.g., md5, sha256) followed by the checksum value.
+      description: The md5 of the file, as `md5:<32 hex chars>`. Optional. Filled in automatically on the pull request by downloading the file; for a very large file, use the Add Dataset form's file picker instead. The size is filled in the same way.
       placeholder: ex. md5:df9376d5c020a23f0f7f51cfe79f303f
     validations:
-      required: true
+      required: false
   - type: textarea
     id: description
     attributes:

.github/scripts/fill_download_fields.py

@@ -0,0 +1,108 @@
+"""Fill in the ``checksum`` and ``size_bytes`` an index entry is missing.
+
+Run by ``.github/workflows/fill_download_fields.yml`` on a pull request that
+touches ``emdatabase/index/``. The docs form and the issue form both let those
+two fields be blank - a contributor cannot be asked to md5 a 100 GB file by hand
+- while every entry needs both, so the file is downloaded here and whatever is
+missing is computed from it by
+:func:`~emdatabase.new_dataset.fill_download_fields`.
+
+A file is only written when something was filled in, and only after it passes
+:func:`~emdatabase.metadata.validate_document` - the same check the test suite
+and ``emdatabase.new_dataset`` run. A link that answers with ``text/html``
+served a page rather than the file; nothing is written for it and the run exits
+non-zero.
+"""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+import yaml
+
+from emdatabase.metadata import (
+    INDEX_DIR,
+    NON_DATASET_FILES,
+    dataset_files,
+    validate_document,
+)
+from emdatabase.new_dataset import build_document, fill_download_fields, write_document
+
+
+def index_files(index_dir: Path | None, paths: list[Path]) -> list[Path]:
+    """Every dataset YAML to fill in: the ones named, or a whole directory.
+
+    ``vendors.yaml`` and the rest of ``index/`` are not dataset collections, so
+    they are dropped however they arrived - the workflow passes whichever files
+    the pull request changed.
+    """
+    if paths:
+        return [path for path in paths if path.name not in NON_DATASET_FILES]
+    if index_dir is None:
+        return dataset_files()
+    return sorted(p for p in index_dir.rglob("*.y*ml") if p.name not in NON_DATASET_FILES)
+
+
+def fill_file(path: Path) -> tuple[list[str], bool]:
+    """Fill one index file in; ``(summary lines, whether it went cleanly)``.
+
+    The entries are rebuilt through
+    :func:`~emdatabase.new_dataset.build_document` before they are written, so a
+    field that was missing altogether lands in the shipped key order rather than
+    at the end of the entry.
+    """
+    document = yaml.safe_load(path.read_text(encoding="utf-8"))
+    try:
+        lines = fill_download_fields(document)
+    except (OSError, ValueError) as error:
+        return [f"- **{path}**: {error}"], False
+    if not lines:
+        return [], True
+
+    rewritten = {name: build_document(name, entry)[name] for name, entry in document.items()}
+    problems = validate_document(rewritten, origin=path)
+    if problems:
+        return [f"- **{problem}**" for problem in problems], False
+    write_document(path, rewritten)
+    return [f"- {line}" for line in lines] + [f"- wrote `{path}`"], True
+
+
+def _parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="fill_download_fields.py",
+        description="Download each index entry that is missing its checksum or size.",
+    )
+    parser.add_argument(
+        "paths", nargs="*", type=Path, help="dataset YAML to fill in; default is every file"
+    )
+    parser.add_argument(
+        "--index",
+        type=Path,
+        help=f"directory of dataset YAML to fill in (default {INDEX_DIR})",
+    )
+    parser.add_argument("--summary", type=Path, help="write a markdown report of the run here")
+    return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = _parser().parse_args(argv)
+
+    lines: list[str] = []
+    ok = True
+    for path in index_files(args.index, args.paths):
+        one, one_ok = fill_file(path)
+        lines += one
+        ok &= one_ok
+    if not lines:
+        lines = ["Every entry already has its checksum and size."]
+
+    summary = "\n".join(lines)
+    print(summary)
+    if args.summary:
+        args.summary.write_text(f"{summary}\n", encoding="utf-8")
+    return 0 if ok else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

.github/scripts/issue_to_yaml.py

@@ -6,6 +6,10 @@
 ``emdatabase.metadata.validate_document`` before it is written - the same check
 the test suite and ``emdatabase.new_dataset`` run - so a malformed issue fails
 here rather than in the pull request the workflow opens.
+
+The form's checksum is optional, and the size is only what a HEAD request said,
+so ``emdatabase.new_dataset.fill_download_fields`` downloads the file for
+whichever of the two the issue left blank before any of that.
 """
 
 import re
@@ -17,6 +21,8 @@
     as_weights_family,
     build_document,
     content_length,
+    fill_download_fields,
+    normalize_url,
     split_url,
     version_date,
     write_document,
@@ -95,8 +101,10 @@ def build_yaml(data):
         sys.exit("the issue has no dataset name")
     # The form asks for the download link; the YAML wants the directory and the
     # file name separately, and keeps the whole link as `url` only when the file
-    # is not served at `source/file`.
-    url = data["URL"].rstrip("/")
+    # is not served at `source/file`. A Drive share link is rewritten to the
+    # download link before either, so the HEAD request below asks about the file
+    # rather than the viewer page.
+    url = normalize_url(data["URL"].rstrip("/"))
     source, filename, link = split_url(url)
     if not source:
         sys.exit(f"{data['URL']!r} is not a link to a file")
@@ -139,15 +147,24 @@ def build_yaml(data):
     return build_document(name, entry), name
 
 
-if __name__ == "__main__":
-    issue_file, out_dir = sys.argv[1], Path(sys.argv[2])
+def write_yaml(issue_file, out_dir):
+    """Parse one issue body and write the entry it describes into ``out_dir``."""
     document, dataset_name = build_yaml(parse_issue_body(Path(issue_file).read_text()))
-    out_path = out_dir / f"{dataset_name}.yaml"
+    out_path = Path(out_dir) / f"{dataset_name}.yaml"
+    for line in fill_download_fields(document):
+        print(line)
+    # A field the issue left blank is missing from the entry rather than at the
+    # end of it, so the document is rebuilt into the shipped key order.
+    document = {name: build_document(name, entry)[name] for name, entry in document.items()}
     problems = validate_document(document, origin=out_path)
     for problem in problems:
         print(problem)
     if problems:
         sys.exit("fix the issue and reopen it")
 
-    write_document(Path(out_path), document)
+    write_document(out_path, document)
     print(f"wrote {out_path}")
+
+
+if __name__ == "__main__":
+    write_yaml(sys.argv[1], Path(sys.argv[2]))

.github/workflows/fill_download_fields.yml

@@ -0,0 +1,92 @@
+name: fill download fields
+
+# The docs form and the issue form both let the checksum and the size be blank,
+# because a contributor cannot be asked to md5 a 100 GB file by hand, while the
+# test suite requires both on every entry. This job downloads whatever a pull
+# request left blank, fills it in and pushes the result back to the branch. A
+# fork's branch cannot be pushed to, so a pull request from one is failed with
+# the values it would have written instead.
+on:
+  pull_request:
+    paths:
+      - emdatabase/index/*.yaml
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  fill:
+    name: checksum and size are filled in
+    runs-on: ubuntu-latest
+    # The files are whole datasets, so a slow host can take hours.
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.ref }}
+          repository: ${{ github.event.pull_request.head.repo.full_name }}
+          token: ${{ secrets.EMDATABASE_PAT || secrets.GITHUB_TOKEN }}
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies and package
+        run: pip install -U -e .'[dev]'
+
+      # The checkout is the head repository, which for a fork carries its own
+      # idea of the base branch, so the base is fetched from this repository.
+      - name: Fetch the base branch
+        if: github.event_name == 'pull_request'
+        run: |
+          git fetch --no-tags https://github.com/${{ github.repository }} \
+            +refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}
+
+      - name: Fill in the checksum and size
+        run: |
+          if [ "${{ github.event_name }}" = "pull_request" ]; then
+            files=$(git diff --name-only origin/${{ github.base_ref }}... -- 'emdatabase/index/*.yaml')
+            if [ -z "$files" ]; then
+              echo "no index file changed" | tee filled.md
+              exit 0
+            fi
+            python .github/scripts/fill_download_fields.py --summary filled.md $files
+          else
+            python .github/scripts/fill_download_fields.py --summary filled.md \
+              --index emdatabase/index
+          fi
+
+      - name: Push the filled-in entries
+        if: >-
+          github.event_name == 'pull_request'
+          && github.event.pull_request.head.repo.full_name == github.repository
+        run: |
+          if git diff --quiet -- emdatabase/index; then
+            echo "nothing to fill in"
+            exit 0
+          fi
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          git commit -m "Fill in checksum and size" -- emdatabase/index
+          git push origin HEAD:${{ github.event.pull_request.head.ref }}
+
+      - name: Report what a fork has to fill in itself
+        if: >-
+          github.event_name == 'pull_request'
+          && github.event.pull_request.head.repo.full_name != github.repository
+        run: |
+          if git diff --quiet -- emdatabase/index; then
+            echo "nothing to fill in"
+            exit 0
+          fi
+          cat filled.md
+          git diff -- emdatabase/index
+          echo "This pull request comes from a fork, so these fields cannot be pushed to its"
+          echo "branch. Copy the values above into the entry, or fill them in yourself with"
+          echo "the Add Dataset form's file picker, and check the file with:"
+          echo "  python -m emdatabase.new_dataset --validate <file>"
+          exit 1

CONTRIBUTING.md

@@ -2,7 +2,10 @@
 
 A dataset is one YAML file in `emdatabase/index/`. To add one, either fill in the
 [new-dataset issue form](https://github.com/electronmicroscopy/emdatabase/issues/new?template=new_dataset.yaml),
-which opens the pull request for you, or run
+which opens the pull request for you, or the
+[Add Dataset form](https://electronmicroscopy.github.io/emdatabase/add_dataset.html),
+which takes one download link - a Google Drive share link included - and fills in the
+file name, size and md5 from your local copy of the file. Or run
 
 ```bash
 python -m emdatabase.new_dataset https://zenodo.org/records/<record>/files/<file>
@@ -11,4 +14,8 @@ python -m emdatabase.new_dataset https://zenodo.org/records/<record>/files/<file
 which fetches the checksum and size, prompts for the rest and writes the file.
 Model weights live in the same index: add `--kind weights`.
 
+Neither form needs the checksum or the size: a pull request carrying an entry that is
+missing either one has the file downloaded on GitHub and the fields filled in for it,
+unless it comes from a fork, whose branch cannot be pushed to.
+
 Full instructions, including what to do by hand and what CI checks: <https://electronmicroscopy.github.io/emdatabase/contributing.html>.

README.md

@@ -192,6 +192,19 @@ to one already on the list fails CI as a misspelling. A technique close to one i
 `techniques.yaml` fails the same way.
 
 Open an issue with the [new dataset template](https://github.com/electronmicroscopy/emdatabase/issues/new?template=new_dataset.yaml),
+fill in the [Add Dataset form](https://electronmicroscopy.github.io/emdatabase/add_dataset.html),
 or run `python -m emdatabase.new_dataset <url>`, which fetches the checksum and size,
 prompts for the rest and writes the file for you to open a pull request with.  See
 [CONTRIBUTING.md](CONTRIBUTING.md).
+
+All three take one download link and split it into `source`, `file` and, when the file
+is not served at `source/file`, `url`.  A Google Drive share link - what the share
+button copies - is rewritten to the `uc?export=download&id=<id>` link that serves the
+file.  The form also has a local file picker: point it at the copy on your machine and
+it fills in the file name, size and md5, hashing the file in the browser without
+uploading it.
+
+Neither web route needs the checksum or the size.  A pull request carrying an entry
+that is missing either one has the file downloaded on GitHub and the fields filled in
+and pushed back to the branch; a pull request from a fork, whose branch cannot be
+pushed to, fails with the values to paste in instead.