fix(moench): PFS health monitoring, MDA Qt-event drain, PFS-aware stage waits - #25
Merged
Merged
Conversation
…Status read) The mid-run PFS health check previously did a ~60 ms AF-device reload once per timepoint, keyed on the event's t index. Staggered schedules interleave t across FOVs, so the dedup never matched and the monitor degenerated to one reload per event (~1900 per run instead of ~80), eating the per-visit headroom of tightly packed schedules. Monitoring is now two-tier. Every event pays only a cheap boolean probe (Moench.read_pfs_engaged_fast, one COM call, no reload): with the patched Nikon adapter DLL, isContinuousFocusEnabled() reports the live Focusing/Locked state, the same predicate as PFS_ENGAGED_STATUSES. The authoritative Status read (reload + fault reason) runs only when the probe disagrees with the expected state or when PFS_HEALTH_PERIOD_S (default 300 s) has elapsed. The periodic read is time-based, immune to event order, and doubles as the safety net on unpatched DLLs, where the probe reads a frozen value and never flags a transition. setup_sequence's run-start read now also starts the periodic clock, so the first event no longer repeats it. The fast probe takes the PFS lock so it cannot race a device reload on another thread.
Two fixes for GUI sessions sharing the core with napari: - run_mda now runs on a thread that delivers its posted Qt events before exiting. GUI handlers on the acquisition thread (psygnal backend) leave queued Qt events there; Windows delivers them during thread shutdown under the loader lock, where blocking on the GIL deadlocks the process against a painting main thread (observed as napari freezing after the last frame of a run, py-spy native dump on the rig). - Z confirmation returns at once while the PFS is engaged: the PFS owns TIZDrive, so a commanded Z target is unreachable and the wait could only burn its full timeout (5 s GUI stall per position-list click). XY and Z confirmation also re-read the commanded target each poll, so a newer move supersedes a stale target instead of timing out. Claude-Session: https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two Moench fixes for GUI sessions sharing the core with napari, plus the two-tier PFS health monitoring.
PFS health monitoring (4799b80)
Two-tier health checks for the Ti PFS: a cheap
isContinuousFocusEnabled()probe for frequent polling, and a periodic fullStatusread (~60 ms AF-device reload) that is the only way to see the failing states (Focus lock failed,Dichroic mirror not inserted,Unsupported objective lens). Seepfs_health()/read_pfs_engaged_fast()docstrings for the patched-vs-unpatched adapter caveats.Drain MDA-thread Qt events (d938c15)
With the psygnal backend, GUI handlers run on the acquisition thread and can create QObjects there; their queued Qt events accumulate on that thread with no event loop to deliver them. Windows delivers the leftovers during thread shutdown while holding the loader lock, and blocking on the GIL there deadlocks the whole process against a painting main thread (observed on the rig as napari freezing after the last frame of a run; three-way cycle GIL -> raster-paint semaphore -> loader lock confirmed with a py-spy native dump).
run_mdanow runs the acquisition on a thread that delivers its own posted Qt events before exiting, so thread shutdown has nothing left to deliver.PFS-aware stage confirmation (d938c15)
The position-confirm loop waited 5 s for a commanded Z target whenever the PFS was engaged: the PFS owns TIZDrive, so the target is unreachable and every position-list click with include-z stalled the GUI for the full timeout.
_confirm_znow returns immediately whileisContinuousFocusEnabled()is true (normal confirmation when PFS is off). Both XY and Z confirmation also re-read the commanded target every poll, so a newer move supersedes a stale target instead of timing out against it.Validated on the demo core (PFS-on z-wait returns in ~1 ms, PFS-off confirms normally; run_mda override runs and drains). The deadlock fix addresses a race, so absence-of-freeze on the rig accumulates evidence over the coming runs rather than proving it in one.
https://claude.ai/code/session_01CSrNm4WLiNzDkiCi75vZfy