Kernel start fails with a bare ENOENT when a venv's kernel spec names an interpreter that no longer exists

[tkislan]

Symptom

Every kernel start against an affected venv fails, and the only thing surfaced is a Jupyter connect error carrying a bare ENOENT:

Failed to connect to the remote Jupyter Server 'http://localhost:8912/'.
Verify the server is running and reachable.
([Errno 2] No such file or directory: '/tmp/deepnote-e2e-root-s2WWxz/.venv/bin/python')

Nothing mentions a kernel spec, so the message points at the server rather than at the thing that is actually broken. The venv is fine; its Python is fine; the kernel spec inside the venv names an interpreter path that no longer exists.

Root cause

installKernelSpec treats existence as validity — src/kernels/deepnote/deepnoteToolkitInstaller.node.ts:515-550:

if (await this.fs.exists(Uri.joinPath(kernelSpecPath, 'kernel.json'))) {
    logger.info(`Kernel spec already exists at ${kernelSpecPath.fsPath}`);
    return;
}

Three facts combine:

1. The spec is written into the venv (<venv>/share/jupyter/kernels/<name>/kernel.json) and hard-codes an absolute argv[0] — whatever path ipykernel install was invoked through.
2. The spec name comes from the venv directory's last path segment (getKernelSpecName, line 488), so any venv reached as .venv is named deepnote-venv regardless of which .venv it is.
3. The early return above keeps whatever is already there.

So the first workspace to start a kernel bakes its own absolute interpreter path into the venv, and every later use of that venv through a different path inherits it. tryInstallKernelSpec (line 468) also swallows install failures by design, so nothing upstream reports the mismatch.

Observed on disk, in a venv shared by more than one workspace:

{
 "argv": [
  "/tmp/deepnote-e2e-root-s2WWxz/.venv/bin/python",
  "-Xfrozen_modules=off", "-m", "ipykernel_launcher", "-f", "{connection_file}"
 ],
 "display_name": "Deepnote (.venv)",
 "language": "python"
}

That directory had been deleted days earlier.

Who hits it

Any venv reached by a path other than the one that installed the spec:

• a venv that was moved or renamed after first use;
• a venv restored from a backup or a build cache into a different location;
• one venv shared by several workspaces through links (how the E2E suite reached its pre-baked venv, which is where this was found).

Recommended fix

Check what the spec launches, not that a file is present. Reinstalling needs no cleanup of its own: jupyter_client's install_kernel_spec does shutil.rmtree(destination) then copytree.

--- a/src/kernels/deepnote/deepnoteToolkitInstaller.node.ts
+++ b/src/kernels/deepnote/deepnoteToolkitInstaller.node.ts
@@
 import { Cancellation, isCancellationError } from '../../platform/common/cancellation';
 import { STANDARD_OUTPUT_CHANNEL } from '../../platform/common/constants';
+import { arePathsSame } from '../../platform/common/platform/fileUtils';
 import { IFileSystem } from '../../platform/common/platform/types';
@@
+    /**
+     * Whether an installed kernel spec still launches the interpreter it is being installed for.
+     *
+     * The spec lives inside the venv and hard-codes an absolute `argv[0]`, so a venv reached through
+     * a different path than the one that installed it — a link, a move, a restored copy — keeps a
+     * spec Jupyter can only fail to spawn, with a bare ENOENT and no mention of the spec. Anything
+     * unreadable counts as stale: reinstalling costs one subprocess, trusting it costs every
+     * kernel start.
+     */
+    private async kernelSpecRunsInterpreter(kernelJson: Uri, interpreter: Uri): Promise<boolean> {
+        try {
+            const { argv } = JSON.parse(await this.fs.readFile(kernelJson)) as { argv?: unknown };
+            const [command] = Array.isArray(argv) ? argv : [];
+
+            return typeof command === 'string' && arePathsSame(command, interpreter.fsPath);
+        } catch (ex) {
+            logger.warn(`Could not read the kernel spec at ${kernelJson.fsPath}`, ex);
+
+            return false;
+        }
+    }
+
     private async installKernelSpec(
         venvInterpreter: PythonEnvironment,
         venvPath: Uri,
         token?: CancellationToken
     ): Promise<void> {
         Cancellation.throwIfCanceled(token);
 
         const kernelSpecName = this.getKernelSpecName(venvPath);
         const kernelSpecPath = Uri.joinPath(venvPath, 'share', 'jupyter', 'kernels', kernelSpecName);
+        const kernelJson = Uri.joinPath(kernelSpecPath, 'kernel.json');
 
         // Keyed on kernel.json, not the directory: a cancelled ipykernel install leaves the
         // directory behind, and that must not short-circuit the reinstall.
-        if (await this.fs.exists(Uri.joinPath(kernelSpecPath, 'kernel.json'))) {
-            logger.info(`Kernel spec already exists at ${kernelSpecPath.fsPath}`);
-            return;
+        if (await this.fs.exists(kernelJson)) {
+            if (await this.kernelSpecRunsInterpreter(kernelJson, venvInterpreter.uri)) {
+                logger.info(`Kernel spec already exists at ${kernelSpecPath.fsPath}`);
+                return;
+            }
+
+            // ipykernel install replaces the whole directory, so the stale spec needs no removal.
+            logger.warn(
+                `Kernel spec at ${kernelSpecPath.fsPath} does not launch ${venvInterpreter.uri.fsPath}; reinstalling`
+            );
         }

arePathsSame (src/platform/common/platform/fileUtils.ts:11) is the house helper; a raw === would compare wrongly on Windows.

Tests

New case, in the style deepnoteToolkitInstaller.unit.test.ts already uses (ts-mockito; venvPath = /fake/venv, fakePython = /fake/venv/bin/python):

test('a kernel spec naming an interpreter outside this venv is replaced, not trusted', async () => {
    seedInterpreterCache(venvPath);
    when(mockFs.exists(anything())).thenResolve(true);
    when(mockFs.readFile(anything())).thenResolve(
        JSON.stringify({ argv: ['/gone/venv/bin/python', '-m', 'ipykernel_launcher', '-f', '{connection_file}'] })
    );
    when(mockProcessService.exec(anything(), anything(), anything())).thenResolve({ stdout: '1.2.3\n', stderr: '' });

    await installer.ensureVenvAndToolkit(venvInterpreter, venvPath, false, cts.token);

    // Exec 0 is the toolkit version probe; the reinstall is what must follow it.
    const [, args] = capture(mockProcessService.exec).second();
    assert.deepStrictEqual(
        args.slice(0, 5),
        ['-m', 'ipykernel', 'install', '--prefix', venvPath.fsPath],
        'a spec that cannot launch this venv is worse than no spec: Jupyter fails with a bare ENOENT'
    );
});

Two existing tests stub exists → true and assert the fast path runs a single exec; they need to say which spec is present, and the fact that they fail without it is the check being load-bearing:

when(mockFs.readFile(anything())).thenResolve(JSON.stringify({ argv: [fakePython.fsPath] }));

Alternatives considered

• Always reinstall. Correct, but spends a Python subprocess on every kernel start; the check is a file read.
• Delete the spec directory first. Redundant (install_kernel_spec rmtrees it), and turns a failed reinstall into no spec rather than a stale one.
• Give the spec a name unique per venv, or canonicalize the venv path with realpath before naming. Attacks venv identity rather than spec validity: it stops two paths to one venv from sharing a name, but does nothing for a venv that genuinely moved, and it renames the spec for every existing user. Worth its own discussion.

Risk

One early return changes behaviour: a spec pointing elsewhere is rebuilt instead of trusted. A matching spec takes the identical fast path, one readFile heavier. Rollback is a revert — no migration, and anything written by the new path is an ordinary ipykernel spec.

Known consequence: two workspaces alternating on one shared venv through different paths rewrite the spec on each switch — one ipykernel install, roughly a second.

---

Found while sharding the E2E suite in #470, where a pre-baked venv is shared across workspaces. That PR works around it on the test side (workspaces point at the venv by its real path instead of linking it in); the product fix is out of scope there.