Skip to content

docs: examples, copy-and-fill templates, notebook tests; remove experiments/ - #27

Merged
hinderling merged 15 commits into
pertzlab:mainfrom
hinderling:feat/examples
Sep 14, 2026
Merged

hinderling merged 15 commits into
pertzlab:mainfrom
hinderling:feat/examples

Conversation

@hinderling

@hinderling hinderling commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

What

Replaces the experiments/ folder (migrated with full history to pertzlab/faro-experiments) with one example, two copy-and-fill templates, and notebook tests. They run on virtual-microscope-teaching (vmteach), a small pure-Python simulator of light-responsive cells behind the pymmcore-plus API, which replaces virtual-microscope as the base dependency. One code change under faro/: the simulated microscope now uses the real DMD class instead of an identity stub, so the calibration path is exercised on the virtual scope.

  • examples/live_experiment.ipynb: a complete feedback experiment on the virtual microscope. The four objects, custom segmentator / feature extractor / stimulator, napari with stock napari-micromanager, DMD calibration, three phases, validate and load, staged start, continue, and a result figure showing every cell moving toward the light. Frames image phase contrast plus DAPI; nuclei are segmented on DAPI and grown to cell bodies; stimulation goes through the CyanStim channel, the only light path the teaching sim delivers the SLM pattern on. Two napari screenshots are taken by the notebook itself with nbscreenshot; committed with outputs.
  • templates/live_experiment/ and templates/reanalysis/: each a uv project (pyproject.toml with the faro pin options) plus a notebook, plus a custom.py for the segmentator, feature extractor or stimulator that only one experiment needs, so the re-analysis notebook can import the same classes (a convention, not enforced). The live template runs end to end on the virtual microscope as shipped, including the DMD calibration cell; its TODO cells hold working defaults with the real-scope alternatives as comments. The re-analysis template only needs its paths cell filled in.
  • templates/README.md: copy, uv sync, pin faro to a commit, update the pin, work against a local checkout, where experiment-specific code goes, all written for people new to uv.
  • faro/microscope/simulation.py: UniMMCoreSimulation wraps the SLM in faro.core.dmd.DMD, uncalibrated until calibrate_dmd runs, like the real scopes. Microscope.calibrate_dmd accepts a Channel, a channel dict or a config name (the template already passed a name). The identity SimDMD stub moves to tests/fake_microscope.py, its only remaining user.
  • tests/test_examples.py: executes the example and both templates in a real kernel on the virtual microscope (gui cells dropped, a parameters cell overridden with small frame counts, the re-analysis test generates its own source run), and checks that every README anchor the notebooks link to exists. New pytest marker examples; nbclient added to the test extras.
  • tests/test_simulation_dmd.py: runs the real calibration routine against the teaching sim, whose projector can be misaligned via sim.slm_affine. A projector with 0.92 scale, 4° rotation and a 30 px shift is recovered to under 1 px over the field, and a mask pushed through the result lands on target.
  • README: examples table, new "Writing your own components" and "Validation" sections, the three links into experiments/ retargeted, a fix for the Quickstart's non-existent MoveUp stimulator (and a calibrate_dmd line), extras table spelling, Installation now shows how to open the example, and the remaining em dashes are gone.
  • experiments/ removed in its own commit (bad8434) so it can be dropped if wanted.
  • pyproject.toml: virtual-microscope-teaching is a base dependency (git source, it is not on PyPI) so the example and the templates run right after uv sync. It is pure Python and brings numba, opencv-headless and matplotlib. The virtual-microscope extra is gone, as is imageio[ffmpeg], which nothing in faro imports.

Depends on

#26 (fix/omezarr-continue): the example's continue cell needs the OME-Zarr time-axis fix. Without it the continued frames are dropped and show up as storage errors in that cell (the committed outputs were generated with the fix applied). Merge #26 first.

Follow-ups (not in this PR)

  • faro-experiments README still says to copy examples/experiment_template; it should point at templates/live_experiment. Superseded folders 01, 02, 03, 90 (and 99_demo_data unless kept as a try-out dataset) can go there.
  • RunStatus from wait() reports 0 background errors even when the storage worker logged some, because analyzer errors are only folded in at finish_experiment. The example prints the status before that, so the count is misleading; worth fixing separately.
  • DMD.calibrate leaves the core exposure at its 25 ms default; runs are unaffected since events carry their own exposure, but hand snaps after a calibration are dim.
  • Two pre-existing test failures on this branch in a fresh environment (MDASequence compatibility, wait shift) are dependency drift, not this branch.
  • The two window screenshots make the notebook about 2.3 MB.

https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy

Replace the demo notebooks (migrated to pertzlab/faro-experiments) with
two examples that run on the virtual microscope and two template folders
that users copy into their experiments repository:

- examples/01_getting_started.ipynb: ten-frame timelapse, the four
  objects, where results land.
- examples/02_live_experiment.ipynb: custom segmentator, feature
  extractor and stimulator, three phases, napari GUI, continue, plots.
- examples/templates/live_experiment/ and templates/reanalysis/: notebook
  with TODO cells that raise until filled in, pyproject.toml with the
  faro pin options, README with a short uv guide (sync, pin, repin,
  local checkout).

tests/test_examples.py executes both examples and both templates in a
kernel on the virtual microscope (TODO cells swapped for fillers, gui
cells dropped, small frame counts injected), and checks that every
README anchor the notebooks link to exists. New pytest marker
"examples"; nbclient added to the test extras.

README: examples table, new "Writing your own components" and
"Validation" sections, retargeted the three links into experiments/,
extras table spelling (virtual-microscope), and the remaining em dashes.

Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
The folders and their full git history now live in
https://github.com/pertzlab/faro-experiments, one uv project per
experiment pinned to a faro commit. The demo notebooks are replaced by
examples/ and the templates; 99_demo_data is no longer needed because the
notebook tests generate their source data on the virtual microscope.

Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
Templates sit next to examples/ instead of inside it, and the two template
READMEs merge into templates/README.md (start steps, uv guide, pinning).
Path references in the README, the example notebooks, the template
pyprojects and the tests follow.

Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
…continued run

Template: the DMD calibration cell no longer skips when an affine is
already set; calibrate_dmd always runs when called and no-ops without a
DMD. Demo: the continue cell prints the continued run's frame count and
background errors, and the notebooks carry their figures (the inline
backend was being overridden when outputs were generated).

The continued phase needs the OME-Zarr time-axis fix on branch
fix/omezarr-continue; without it the extra frames are dropped and show
up as background errors in that cell.

Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
…, load before start

Demo 02 takes two nbscreenshot() captures itself (live view of the
virtual microscope, status widget after load_experiment), so no image
assets are checked in; the cells are tagged gui and skipped headless.
load_experiment now follows the validation assert in the demo and the
template, and the run cell only starts. The closing figure gains a
panel with every cell's vertical displacement behind the median.

Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
Both ran on the virtual microscope, so 01 was a subset of 02. The four
object table, the Micro-Manager demo-devices note, the tracks-file
explanation and the next-steps list move into the single example, now
examples/live_experiment.ipynb. README, template links and tests follow.

Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
"Connect to the microscope" explains that the virtual scope stands in
for a real one and swapping the import is the only change. The result
figure drops the speed panel; the displacement panel is negated so
upward movement plots upward.

Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
Quickstart used a MoveUp stimulator that does not exist; Simulated
Controller still spoke of demo data; the examples table promised a uv
guide inside each template folder; Installation never said how to open
the example, and Jupyter is not in the environment so the commands use
uv run --with jupyterlab. Template pyprojects list jupyterlab as a
commented dependency.

Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
The live experiment template now runs end to end on the virtual
microscope as shipped: every TODO cell holds a working default and lists
the real-scope alternatives as comments, and segmentation, features,
stimulation, tracking, pipeline and controller sit in one "set up the
pipeline" cell. StimPercentageOfCell is the default stimulator, so the
stimulation phase carries stim_cell_percentage in its rtm_metadata. The
re-analysis template keeps a raise only on its paths cell.

The notebook test executes templates verbatim (settings cell tagged
parameters gets small frame counts); fillers remain only for raise
cells, and a structure test keeps the two in sync.

Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
hinderling and others added 5 commits September 7, 2026 18:07
It runs on the virtual microscope as shipped, so a copied folder needs
the extra for uv sync to work out of the box. Verified that faro's git
sources and index pins carry over to a template project.

Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
The example notebook and the live template run on the virtual
microscope as shipped, and they are the first thing a new user tries.
Requiring an extra for them was the one step where that first run could
fail. virtual-microscope is pure Python; it brings numba and opencv
(about 250 MB on a 1.4 GB base install). The extra is removed together
with imageio[ffmpeg], which nothing in faro imports. Docs, the template
pyproject and the tests no longer mention the extra.

Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
Replace the virtual-microscope git dependency with virtual-microscope-teaching
(vmteach): a ~3k-line pure-Python simulator of light-responsive cells behind
the pymmcore-plus API, 5 ms per frame and sub-second load. The notebook tests
and the reanalysis source run use load_microscope(mode="realtime") and image
the DAPI channel, which the teaching sim renders for segmentation.
UniMMCoreSimulation used an identity SimDMD stub, so the calibration routine
and the affine path never ran on the virtual scope. It now attaches
faro.core.dmd.DMD like the real scopes, uncalibrated until calibrate_dmd
runs. calibrate_dmd accepts a Channel, a channel dict or a config name, which
is what the template already passed. The SimDMD stub moves to the test fakes.

tests/test_simulation_dmd.py calibrates against the teaching sim's
slm_affine: a projector with 0.92 scale, 4 deg rotation and a 30 px shift is
recovered to under 1 px over the field.
Example and live template connect with vmteach.load_microscope, image
phase-contrast plus DAPI, segment nuclei on DAPI (grown to cell bodies in the
example) and stimulate through the CyanStim channel, which is the only light
path the teaching sim delivers the SLM pattern on. Both calibrate the DMD
before running; the example gets a section explaining why.

Each template folder gains a custom.py for the segmentator, feature extractor
or stimulator that only one experiment needs, so the re-analysis notebook can
import the same classes. Outputs regenerated on the teaching sim.
@hinderling
hinderling merged commit 608f492 into pertzlab:main Sep 14, 2026
@hinderling
hinderling deleted the feat/examples branch September 14, 2026 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant