refactor(remote): OETC/SSH as standalone transports (closes #683) by FBumann · Pull Request #697 · PyPSA/linopy

FBumann · 2026-05-19T09:17:02Z

Stacked on top of #690 (rebase target after that lands on master).

Closes #683 — with a different design than the issue proposed.

Why standalone, not `Solver` subclass

The issue framed OETC as a Solver subclass so Model.solve could fold the remote= branch into the unified Solver pipeline. Trying that, the fit was wrong:

Remote handlers aren't solvers — they ship a netcdf elsewhere and let someone else solve.
Forcing them through Solver required workarounds: a non-colliding inner_solver field, property-vs-field collisions on solver_name, SolverName enum entries for things that aren't algorithms.
Solver's feature-flag plumbing is about local solver capabilities; for remotes we want to validate the inner solver's flags.

So Oetc and SSH are standalone classes in linopy/remote/, parallel to Solver, not subclasses of it.

What's new

m.solve("gurobi", remote=OetcSettings(email=..., password=..., name=..., ...), Method=2)
m.solve("highs", remote=SshSettings(hostname=..., setup_commands=["conda activate linopy-env"]))

solver_name and **solver_options are the same axes as for local solves; remote= selects where to run.
New model.remote attribute mirrors model.solver for post-solve introspection.
Oetc.upload(model) / submit() / collect(model) exposed for async-style workflows (drive the seam manually).
SshSettings.setup_commands: list[str] runs shell commands on the remote before the inner solver — replaces the old "build a RemoteHandler and call .execute(...)" pattern.
Pip extra remote → ssh (matches what it installs: paramiko). OETC has its own linopy[oetc] extra; both are now symmetric. linopy[remote] is dropped (not aliased) — see Breaking Changes in the release notes.

API surface — what the new transport classes do (and don't) cover

Oetc and SSH are intentionally narrower than the deprecated Handler classes. They cover solving a model on a remote machine. They don't try to be a general remote-shell wrapper.

	Old `RemoteHandler`	New `SSH`	Replacement
Solve a model	`solve_on_remote(m)`	`solve(m)`	direct rename
Env activation before solve	`handler.execute("conda activate …")`	—	`SshSettings.setup_commands=[…]`
Reuse connection across multiple solves	implicit (long-lived handler)	implicit (`SSH._handler` is cached)	works the same
Arbitrary remote shell commands between solves	`handler.execute(any cmd)`	—	drop to `RemoteHandler` (deprecated) or use paramiko directly
Custom file transfer	`write_model_on_remote` etc.	—	drop to `RemoteHandler` (deprecated) or paramiko/SFTP directly

Asymmetry with Oetc: Oetc exposes upload / submit / collect because OETC has a genuine async seam (submit a job, walk away, collect hours later). SSH doesn't — the shell session is held open and the solve is synchronous, so there's no useful split.

The narrowing is intentional: the old RemoteHandler blurred two concerns — be a paramiko shell wrapper and solve linopy models on a remote. The new SSH is just the second. If you want the first, paramiko itself is a perfectly good API.

Deprecations

OetcHandler(...) / RemoteHandler(...) construction emits DeprecationWarning. Their solve_on_oetc / solve_on_remote return contracts are unchanged during deprecation; solve_on_oetc now internally delegates to Oetc.solve.
Model.solve(remote=<Handler>) is deprecated — pass the settings dataclass instead.
OetcCredentials is deprecated — pass email= and password= directly to OetcSettings. The OetcSettings(credentials=OetcCredentials(...)) shape still works during the deprecation period.

Other designs considered (rejected)

Remote returns Model instead of Result. Tried in a separate commit (f049999, since reverted): Oetc.solve / SSH.solve would return the round-tripped solved Model, with a new Model._assign_from_solved_model(solved) helper for in-place writeback. The pitch: remote workers natively produce inflated Models, so the Result shape forces an xarray → flat → xarray round-trip via _scatter_solution_from_solved_model. Rejected in favor of uniformity: every solve path (local Solver or Remote) returns a Result and goes through Model.assign_result. One assign-path, one mental model. Cost: the scatter pass — ~few ms + ~32 MB transient per 1M model variables, dominated by the network round-trip in practice.

Keep OetcCredentials as a separate class long-term. Considered keeping it (cloud-SDK convention, visual flag for secrets in code review) and adding OetcCredentials.from_env() for symmetry with OetcSettings.from_env(). Folded instead because:

The auth shape is uniform (always email + password), so the wrapper adds no structural payoff.
SshSettings already takes username / password inline.
Every existing OETC user is already on a deprecation cycle for OetcHandler, so the extra one-line migration (credentials=OetcCredentials(email=..., password=...) → email=..., password=...) is essentially free.

Follow-ups (not in this PR)

Remove OetcHandler / RemoteHandler. Their private transport methods (_upload_file_to_gcp, _submit_job_to_compute_service, wait_and_get_job_data, _download_file_from_gcp, __sign_in, paramiko shell management) still live on the Handler classes; Oetc / SSH reach into self._handler._upload_file_to_gcp(...) to use them. A follow-up PR will migrate that code into Oetc / SSH and port the ~50 OETC tests that exercise the handler internals.
Remove OetcCredentials. After one release cycle of deprecation warnings.
Phase 5 tests for the new standalone classes (Oetc.solve(mocked OetcHandler), SSH.solve(mocked RemoteHandler), deprecation-warning assertions, upload/submit/collect separability).

Test plan

pytest test/remote/ test/test_sos_reformulation.py test/test_oetc_settings.py — 166 pass.
pytest --ignore=test/remote — full broader suite (3495 passed, 29 skipped).
ruff check, ruff format — clean.

🤖 Generated with Claude Code

@FabianHofmann:

if you'd prefer to keep OetcCredentials as a long-term, top-level class im fine. I went with fold because the deprecation cost happens to be near-free given the Handler is also deprecated, and .from_env() handles the credentials issue cleanly.
Im not fully satisifed with "remote". Its more a transport, or offload technically... Its now or never on the rename maybe, but its probably not worth it anyway.

FabianHofmann · 2026-05-19T09:30:41Z

@FBumann I think the concept makes sense. I would like to make sure that it is flexible enough for interfaces like gurobi instant cloud. could you take that into consideration?

FBumann · 2026-05-19T09:33:37Z

@FabianHofmann Yes, I already did think about it. But i can sketch it out a bit more, maybe even create a Remote parent class...
I also didnt add the tests yet, and some cleanupy might also be possible. And the latest context manager from #690 isnt used yet

FBumann · 2026-05-19T09:44:57Z

@FBumann I think the concept makes sense. I would like to make sure that it is flexible enough for interfaces like gurobi instant cloud. could you take that into consideration?

It fits well for async instant cloud. For the sync instan cloud it doesnt fit as cleanly. But I think it can be wired in without adjusting public api. Probably just refactor some internal helpers for collection

FabianHofmann · 2026-05-19T09:59:18Z

@FBumann I think the concept makes sense. I would like to make sure that it is flexible enough for interfaces like gurobi instant cloud. could you take that into consideration?

It fits well for async instant cloud. For the sync instan cloud it doesnt fit as cleanly. But I think it can be wired in without adjusting public api. Probably just refactor some internal helpers for collection

I guess async is the most important one

FBumann · 2026-05-19T10:14:41Z

@FBumann I think the concept makes sense. I would like to make sure that it is flexible enough for interfaces like gurobi instant cloud. could you take that into consideration?

It fits well for async instant cloud. For the sync instan cloud it doesnt fit as cleanly. But I think it can be wired in without adjusting public api. Probably just refactor some internal helpers for collection

I guess async is the most important one

I thin the regular gurobi cloud can be used by just using the regular gurobi solver anyway...? It fully dispatches to gurobipy, and gets a gurobipy solution back, right? SO no extra class needed...

FBumann · 2026-05-19T10:24:15Z

@FabianHofmann
Here is some advice from CLaude after some back and forth:

## Generality check — Gurobi & other clouds

- Gurobi Instant Cloud / Compute Server sync → already works via m.solve("gurobi", env=cloud_env). Belongs as GurobiSolver config, not a remote class. A GurobiInstantCloud remote class would be a fake — construct an Env and dispatch back into the regular Solver path.
- Gurobi batch solve / detached submit-now-collect-later → genuine fit for the remote handler shape.
- Friction for any future handler: solver_name is mandatory and means inner solver (redundant for solver-specific clouds); _scatter_solution_from_solved_model assumes a netcdf round-trip with a linopy Model on the other side.

## Naming clarity

"Remote" is defensible as user-facing vocabulary but mislabels the technical category, and "async" is the wrong axis entirely:

- The actual invariant shared by Oetc and SSH is a second linopy instance does the solve (local linopy serializes → remote linopy deserializes → .solve() → result back). Gurobi Cloud is also remote geographically, but only one linopy instance is involved — gurobipy talks to a remote engine too.
- "Async" misclassifies SSH (fully synchronous, no job seam) and overpromises on Oetc (.solve() blocks; the seam is optional). True async (blocking=False returning a handle) is orthogonal — it could apply to a local callback-driven Gurobi solve too.

SO the new architecture should cover all cases. THe only issue i see is thst "remote" isnt a precise name. "worker" or "offload" might be better. But thats not as important for now i think

Closes #683. The issue framed OETC as a `Solver` subclass to fold the `remote=` branch in `Model.solve` into the unified Solver pipeline. Trying that, the fit was wrong: remote handlers aren't solvers — they ship a netcdf elsewhere and let someone else solve. Forcing them through `Solver` required workarounds (a non-colliding `inner_solver` field name, property-vs-field collisions on `solver_name`, `SolverName` enum entries for things that aren't algorithms). Going standalone instead: - `linopy.remote.Oetc(settings, solver_name, options)` — standalone class with `upload(model)` / `submit()` / `collect(model)` / `solve(model)` lifecycle. The submit/collect split is in the right shape for future async work (a `blocking=False` solve, Gurobi-batch, etc.) without baking the seam into the Solver hierarchy. - `linopy.remote.SSH(settings, solver_name, options)` — synchronous ship-and-run handler. - Both produce a label-indexed `Result` via the shared `_scatter_solution_from_solved_model` helper in `linopy/remote/_common.py`. - Both validate the inner solver locally via `_validate_inner_solver` (unknown name raises; known-but-incapable raises before the round-trip). Settings dataclasses now pure transport. `OetcSettings.solver` and `OetcSettings.solver_options` are removed — those config axes live on the outer `Model.solve` call now, mirroring the local-solve API. New `SshSettings` follows the same shape. `Model.solve` changes: - `remote=<Settings>` → standalone-handler dispatch via the new `_solve_with_remote_settings` method. - `remote=OetcHandler/RemoteHandler` → legacy shim, emits `DeprecationWarning`, builds equivalent settings, routes to the same new pipeline. - New `model.remote` slot — set to the `Oetc`/`SSH` instance after a remote solve, lets callers introspect `model.remote._job_uuid` etc. `model.solver` is None during remote solves. The reformulation lifecycle (from #690) wraps the remote dispatch via `sos_reformulation_context` + `suppress_serialization_warning`, the same context managers the local-solve path uses. The `to_netcdf` UserWarning is suppressed for the handler's internal serialization. `OetcHandler.solve_on_oetc` emits a `DeprecationWarning` when called directly, pointing at the new API. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

FBumann · 2026-05-19T13:18:51Z

@FabianHofmann I argue that the final form of API would be removing OetcHandler.
The High level api would be OetcSettings passed to Model.solve(). The low level API is using Oetc.from_model() like a Solver.

OetcHandler delegates for backwards compat, but is deprecated.

This reflects the 2-level state of regular solvers: Pass name + options in Model.solve(), or directly invoke with the Solver class, which holds the logic/validation for the solver.

Do you agree?

…eprecate legacy handlers - `OetcHandler.__init__` / `RemoteHandler.__post_init__` emit `DeprecationWarning` pointing at `Oetc` / `SSH` and `Model.solve(remote=...)`. An `_internal=True` kwarg suppresses the warning when the new classes construct the handler themselves. - `OetcHandler.solve_on_oetc` delegates to `Oetc.solve` so the upload→submit→poll→download orchestration lives in one place. Legacy `Model` return shape preserved by reading `oetc._solved_model` after `collect`. - `Oetc.upload` / `SSH.solve` no-op handler construction when one is already attached, so the deprecated handler can be reused as the underlying transport without re-running auth. - Validation moved into `Oetc.solve` (was in `upload`) so the legacy handler path is unchanged for users. Two `TestSolveOnOetc` tests grow a few mock attrs (`_xCounter=0`, empty `.items()`, `termination_condition`) so the bare `Mock()` model flows through `Oetc.collect`'s scatter step. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Documents the new `Oetc` / `SSH` standalone classes, the `Model.solve(remote=<Settings>)` entry point, and the deprecation of `OetcHandler` / `RemoteHandler`. Migration examples show both the recommended `Model.solve(remote=...)` path and the direct `Oetc.solve(m)` + `assign_result` path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

`SshSettings.setup_commands` is a list of shell commands run on the remote interactive session before the inner solver is invoked — e.g. `setup_commands=["conda activate linopy-env"]`. Replaces the old pattern of holding a `RemoteHandler` instance and manually calling `.execute(...)`. The `examples/solve-on-remote.ipynb` notebook is rewritten to: - use `Model.solve(remote=SshSettings(...))` as the primary path, - demonstrate `setup_commands` for env activation, - show `SSH(settings, solver_name, options).solve(m)` as the advanced "drive the transport directly" path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- Drops `OetcHandler` cells (deprecated) — primary path is now `Model.solve("gurobi", remote=OetcSettings(...), **opts)`. - Removes the settings-level `solver=` / `solver_options=` cell; inner solver name and options live at the call site, matching the local-solve shape. - Replaces the retry/error-handling cell with an "Advanced" section that walks through `Oetc.upload` / `Oetc.submit` / `Oetc.collect` — the async-friendly seam that motivates the standalone class. - Trims to essentials. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The rewritten notebooks dropped the notebook-level `"nbsphinx": {"execute": "never"}` metadata, which both prior versions had. Without it, the docs build tries to execute the cells and fails on `os.environ["OETC_EMAIL"]` / a live SSH connect. Restore the original metadata so the docs build returns to rendering the notebooks as static content.

OetcCredentials was a 2-field wrapper (email, password) that added an extra construction layer with no functional payoff. Inline the two fields onto OetcSettings so the construction shape matches SshSettings (which takes username/password directly). OetcCredentials stays importable and emits a DeprecationWarning on construction; OetcSettings(credentials=...) is still accepted and copies the values through. To be removed in a future release. Note: the positional argument order on OetcSettings shifts because credentials is no longer the first required field. Existing keyword-arg callers (the typical case) are unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… guide The two notebooks duplicated their model-creation cells and "Advanced: drive the transport directly" sections, while users picking a remote transport read one or the other — not both. Merge into a single `remote-machines.ipynb` with parallel SSH / OETC sections and a shared advanced section, plus a brief "which to pick?" table. Rename keeps the file out of the "solve-on-*" namespace (the docs section is already "Solving"); `remote-machines` describes what the page is about, not what you do with it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- Switch the manual OetcSettings example from os.environ[...] to literal placeholder strings. Mixing os.environ access with the manual-construction example was confusing — environment loading is what from_env() is for. - Drop the SSH-vs-OETC comparison table at the end. The information is obvious from each section's 'What you need' bullets.

The `remote` extra installed only `paramiko` — i.e., the SSH transport deps. With OETC as a parallel transport (own `linopy[oetc]` extra), the `remote` name was misleading and asymmetric. Rename to `ssh` to match what it installs. Drop the old `remote` extra (rather than alias it) because: - It only shipped in v0.7.0 (recent, narrow adoption). - Pip extras have no runtime deprecation mechanism, so the alias would just defer an inevitable break. - Aliasing leaves a redundant extra in the API surface. Documented under "Breaking Changes" in the release notes; the merged remote-machines notebook is updated to use `linopy[ssh]`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds `test/remote/test_remotes.py` covering the new public surface that `test_oetc.py` and `test_ssh.py` don't (those still focus on the deprecated Handler classes): - `Oetc.solve` happy path with a mocked `OetcHandler`. - `Oetc.upload` / `submit` / `collect` as separable steps. - `SSH.solve` happy path; `SshSettings.setup_commands` runs on the remote shell on first handler construction. - Inner-solver validation (unknown name raises in both transports). - `Model.solve(remote=OetcSettings(...))` / `Model.solve(remote=SshSettings(...))` end-to-end with `Oetc.solve` / `SSH.solve` monkeypatched. - Deprecation warnings on `OetcHandler`, `RemoteHandler`, `OetcCredentials`, and `Model.solve(remote=<Handler>)`. - `_internal=True` suppresses the handler deprecation warnings on the construction path used internally by `Oetc` / `SSH`. Also updates `test-notebooks` skip-list for the renamed merged notebook (`remote-machines.ipynb` replaces `solve-on-{remote,oetc}.ipynb`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ok headers

The API page only documented the deprecated `RemoteHandler`. Add the new public classes (`SSH`, `Oetc`, `SshSettings`, `OetcSettings`) and the remaining deprecated entries (`OetcHandler`, `OetcCredentials`) so autosummary generates a stub for each. The new entries link to the merged `remote-machines` user guide. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

FBumann · 2026-05-19T18:20:02Z

@FabianHofmann Ready for a review

Resolved conflict in examples/solve-on-oetc.ipynb: the PR intentionally deletes this notebook (replaced by examples/remote-machines.ipynb), while master only applied cosmetic nbstripout/unicode normalization. Kept the deletion.

…ng text A user passing m.solve("gurobi", remote=...) only ever supplies one solver, and remotes are transports rather than solvers, so "inner" has no "outer" to contrast with. Drop it from docstrings, validation error messages, and release notes. Internal symbols (inner_solver param, _validate_inner_solver) keep the name — there it disambiguates the shipped solver string from the transport object. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

FBumann · 2026-05-20T13:59:59Z

@FabianHofmann ready. Do you wait for feedback of oetc users?

The :class: target pointed at linopy.solvers.SSH, but SSH is exported from linopy.remote and is a transport, not a solver. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Base automatically changed from refactor/sos-reformulation-methods to master May 19, 2026 10:00

FBumann force-pushed the refactor/oetc-as-solver branch from b221cc3 to 80e8be1 Compare May 19, 2026 10:21

FBumann force-pushed the refactor/oetc-as-solver branch from 80e8be1 to daadcae Compare May 19, 2026 12:28

FBumann and others added 16 commits May 19, 2026 16:06

ci: retrigger docs build

01b407d

docs(release): note OetcCredentials deprecation

4a9bf2b

docs(release): note narrower SSH surface vs RemoteHandler

2562acb

docs(release): trim SSH-surface note

860f0d3

fix(test+docs): mypy on test_remotes.py, drop 'Option N:' from notebo…

0fa2769

…ok headers

FBumann marked this pull request as ready for review May 19, 2026 18:17

FBumann requested a review from FabianHofmann May 19, 2026 18:18

FBumann added this to the v0.8.0 milestone May 20, 2026

FBumann mentioned this pull request May 20, 2026

docs: restructure and refresh the documentation guides #704

Draft

FBumann and others added 2 commits May 20, 2026 15:32

Merge branch 'master' into refactor/oetc-as-solver

c93d792

Resolved conflict in examples/solve-on-oetc.ipynb: the PR intentionally deletes this notebook (replaced by examples/remote-machines.ipynb), while master only applied cosmetic nbstripout/unicode normalization. Kept the deletion.

docs(remote): fix stale SSH cross-reference in SshSettings docstring

bfc38eb

The :class: target pointed at linopy.solvers.SSH, but SSH is exported from linopy.remote and is a transport, not a solver. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(remote): OETC/SSH as standalone transports (closes #683)#697

refactor(remote): OETC/SSH as standalone transports (closes #683)#697
FBumann wants to merge 20 commits into
masterfrom
refactor/oetc-as-solver

FBumann commented May 19, 2026 •

edited

Loading

Uh oh!

FabianHofmann commented May 19, 2026

Uh oh!

FBumann commented May 19, 2026

Uh oh!

FBumann commented May 19, 2026

Uh oh!

FabianHofmann commented May 19, 2026

Uh oh!

FBumann commented May 19, 2026

Uh oh!

FBumann commented May 19, 2026

Uh oh!

FBumann commented May 19, 2026 •

edited

Loading

Uh oh!

FBumann commented May 19, 2026

Uh oh!

FBumann commented May 20, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

FBumann commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why standalone, not Solver subclass

What's new

API surface — what the new transport classes do (and don't) cover

Deprecations

Other designs considered (rejected)

Follow-ups (not in this PR)

Test plan

Uh oh!

FabianHofmann commented May 19, 2026

Uh oh!

FBumann commented May 19, 2026

Uh oh!

FBumann commented May 19, 2026

Uh oh!

FabianHofmann commented May 19, 2026

Uh oh!

FBumann commented May 19, 2026

Uh oh!

FBumann commented May 19, 2026

Uh oh!

FBumann commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FBumann commented May 19, 2026

Uh oh!

FBumann commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

FBumann commented May 19, 2026 •

edited

Loading

Why standalone, not `Solver` subclass

FBumann commented May 19, 2026 •

edited

Loading

FBumann commented May 20, 2026 •

edited

Loading