docs: examples, copy-and-fill templates, notebook tests; remove experiments/ - #27
Merged
Merged
Conversation
Replace the demo notebooks (migrated to pertzlab/faro-experiments) with two examples that run on the virtual microscope and two template folders that users copy into their experiments repository: - examples/01_getting_started.ipynb: ten-frame timelapse, the four objects, where results land. - examples/02_live_experiment.ipynb: custom segmentator, feature extractor and stimulator, three phases, napari GUI, continue, plots. - examples/templates/live_experiment/ and templates/reanalysis/: notebook with TODO cells that raise until filled in, pyproject.toml with the faro pin options, README with a short uv guide (sync, pin, repin, local checkout). tests/test_examples.py executes both examples and both templates in a kernel on the virtual microscope (TODO cells swapped for fillers, gui cells dropped, small frame counts injected), and checks that every README anchor the notebooks link to exists. New pytest marker "examples"; nbclient added to the test extras. README: examples table, new "Writing your own components" and "Validation" sections, retargeted the three links into experiments/, extras table spelling (virtual-microscope), and the remaining em dashes. Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
The folders and their full git history now live in https://github.com/pertzlab/faro-experiments, one uv project per experiment pinned to a faro commit. The demo notebooks are replaced by examples/ and the templates; 99_demo_data is no longer needed because the notebook tests generate their source data on the virtual microscope. Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
Templates sit next to examples/ instead of inside it, and the two template READMEs merge into templates/README.md (start steps, uv guide, pinning). Path references in the README, the example notebooks, the template pyprojects and the tests follow. Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
…continued run Template: the DMD calibration cell no longer skips when an affine is already set; calibrate_dmd always runs when called and no-ops without a DMD. Demo: the continue cell prints the continued run's frame count and background errors, and the notebooks carry their figures (the inline backend was being overridden when outputs were generated). The continued phase needs the OME-Zarr time-axis fix on branch fix/omezarr-continue; without it the extra frames are dropped and show up as background errors in that cell. Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
…, load before start Demo 02 takes two nbscreenshot() captures itself (live view of the virtual microscope, status widget after load_experiment), so no image assets are checked in; the cells are tagged gui and skipped headless. load_experiment now follows the validation assert in the demo and the template, and the run cell only starts. The closing figure gains a panel with every cell's vertical displacement behind the median. Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
Both ran on the virtual microscope, so 01 was a subset of 02. The four object table, the Micro-Manager demo-devices note, the tracks-file explanation and the next-steps list move into the single example, now examples/live_experiment.ipynb. README, template links and tests follow. Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
"Connect to the microscope" explains that the virtual scope stands in for a real one and swapping the import is the only change. The result figure drops the speed panel; the displacement panel is negated so upward movement plots upward. Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
Quickstart used a MoveUp stimulator that does not exist; Simulated Controller still spoke of demo data; the examples table promised a uv guide inside each template folder; Installation never said how to open the example, and Jupyter is not in the environment so the commands use uv run --with jupyterlab. Template pyprojects list jupyterlab as a commented dependency. Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
The live experiment template now runs end to end on the virtual microscope as shipped: every TODO cell holds a working default and lists the real-scope alternatives as comments, and segmentation, features, stimulation, tracking, pipeline and controller sit in one "set up the pipeline" cell. StimPercentageOfCell is the default stimulator, so the stimulation phase carries stim_cell_percentage in its rtm_metadata. The re-analysis template keeps a raise only on its paths cell. The notebook test executes templates verbatim (settings cell tagged parameters gets small frame counts); fillers remain only for raise cells, and a structure test keeps the two in sync. Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
It runs on the virtual microscope as shipped, so a copied folder needs the extra for uv sync to work out of the box. Verified that faro's git sources and index pins carry over to a template project. Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
The example notebook and the live template run on the virtual microscope as shipped, and they are the first thing a new user tries. Requiring an extra for them was the one step where that first run could fail. virtual-microscope is pure Python; it brings numba and opencv (about 250 MB on a 1.4 GB base install). The extra is removed together with imageio[ffmpeg], which nothing in faro imports. Docs, the template pyproject and the tests no longer mention the extra. Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
Replace the virtual-microscope git dependency with virtual-microscope-teaching (vmteach): a ~3k-line pure-Python simulator of light-responsive cells behind the pymmcore-plus API, 5 ms per frame and sub-second load. The notebook tests and the reanalysis source run use load_microscope(mode="realtime") and image the DAPI channel, which the teaching sim renders for segmentation.
UniMMCoreSimulation used an identity SimDMD stub, so the calibration routine and the affine path never ran on the virtual scope. It now attaches faro.core.dmd.DMD like the real scopes, uncalibrated until calibrate_dmd runs. calibrate_dmd accepts a Channel, a channel dict or a config name, which is what the template already passed. The SimDMD stub moves to the test fakes. tests/test_simulation_dmd.py calibrates against the teaching sim's slm_affine: a projector with 0.92 scale, 4 deg rotation and a 30 px shift is recovered to under 1 px over the field.
Example and live template connect with vmteach.load_microscope, image phase-contrast plus DAPI, segment nuclei on DAPI (grown to cell bodies in the example) and stimulate through the CyanStim channel, which is the only light path the teaching sim delivers the SLM pattern on. Both calibrate the DMD before running; the example gets a section explaining why. Each template folder gains a custom.py for the segmentator, feature extractor or stimulator that only one experiment needs, so the re-analysis notebook can import the same classes. Outputs regenerated on the teaching sim.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the
experiments/folder (migrated with full history to pertzlab/faro-experiments) with one example, two copy-and-fill templates, and notebook tests. They run on virtual-microscope-teaching (vmteach), a small pure-Python simulator of light-responsive cells behind the pymmcore-plus API, which replacesvirtual-microscopeas the base dependency. One code change underfaro/: the simulated microscope now uses the realDMDclass instead of an identity stub, so the calibration path is exercised on the virtual scope.examples/live_experiment.ipynb: a complete feedback experiment on the virtual microscope. The four objects, custom segmentator / feature extractor / stimulator, napari with stock napari-micromanager, DMD calibration, three phases, validate and load, staged start, continue, and a result figure showing every cell moving toward the light. Frames image phase contrast plus DAPI; nuclei are segmented on DAPI and grown to cell bodies; stimulation goes through theCyanStimchannel, the only light path the teaching sim delivers the SLM pattern on. Two napari screenshots are taken by the notebook itself withnbscreenshot; committed with outputs.templates/live_experiment/andtemplates/reanalysis/: each a uv project (pyproject.tomlwith the faro pin options) plus a notebook, plus acustom.pyfor the segmentator, feature extractor or stimulator that only one experiment needs, so the re-analysis notebook can import the same classes (a convention, not enforced). The live template runs end to end on the virtual microscope as shipped, including the DMD calibration cell; its TODO cells hold working defaults with the real-scope alternatives as comments. The re-analysis template only needs its paths cell filled in.templates/README.md: copy,uv sync, pin faro to a commit, update the pin, work against a local checkout, where experiment-specific code goes, all written for people new to uv.faro/microscope/simulation.py:UniMMCoreSimulationwraps the SLM infaro.core.dmd.DMD, uncalibrated untilcalibrate_dmdruns, like the real scopes.Microscope.calibrate_dmdaccepts aChannel, a channel dict or a config name (the template already passed a name). The identitySimDMDstub moves totests/fake_microscope.py, its only remaining user.tests/test_examples.py: executes the example and both templates in a real kernel on the virtual microscope (guicells dropped, aparameterscell overridden with small frame counts, the re-analysis test generates its own source run), and checks that every README anchor the notebooks link to exists. New pytest markerexamples;nbclientadded to the test extras.tests/test_simulation_dmd.py: runs the real calibration routine against the teaching sim, whose projector can be misaligned viasim.slm_affine. A projector with 0.92 scale, 4° rotation and a 30 px shift is recovered to under 1 px over the field, and a mask pushed through the result lands on target.experiments/retargeted, a fix for the Quickstart's non-existentMoveUpstimulator (and acalibrate_dmdline), extras table spelling, Installation now shows how to open the example, and the remaining em dashes are gone.experiments/removed in its own commit (bad8434) so it can be dropped if wanted.pyproject.toml:virtual-microscope-teachingis a base dependency (git source, it is not on PyPI) so the example and the templates run right afteruv sync. It is pure Python and brings numba, opencv-headless and matplotlib. Thevirtual-microscopeextra is gone, as isimageio[ffmpeg], which nothing in faro imports.Depends on
#26 (
fix/omezarr-continue): the example's continue cell needs the OME-Zarr time-axis fix. Without it the continued frames are dropped and show up as storage errors in that cell (the committed outputs were generated with the fix applied). Merge #26 first.Follow-ups (not in this PR)
examples/experiment_template; it should point attemplates/live_experiment. Superseded folders 01, 02, 03, 90 (and 99_demo_data unless kept as a try-out dataset) can go there.RunStatusfromwait()reports 0 background errors even when the storage worker logged some, because analyzer errors are only folded in atfinish_experiment. The example prints the status before that, so the count is misleading; worth fixing separately.DMD.calibrateleaves the core exposure at its 25 ms default; runs are unaffected since events carry their own exposure, but hand snaps after a calibration are dim.https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy