diff --git a/Cargo.lock b/Cargo.lock index 9aa9d942..443d4629 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -137,6 +137,7 @@ name = "bsk" version = "0.1.10" dependencies = [ "anyhow", + "base64", "bsk-protocol", "clap", "console", diff --git a/Cargo.toml b/Cargo.toml index 007dbacf..3af54d87 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -36,6 +36,7 @@ flate2 = "1.1.9" tar = "0.4.46" zip = { version = "7.2.0", default-features = false, features = ["deflate"] } sha2 = "0.11.0" +base64 = "0.22" tempfile = "3" nix = { version = "0.29", features = ["signal", "process"] } rand = "0.8" diff --git a/apps/extension/PRIVACY.md b/apps/extension/PRIVACY.md index 9517a3c4..bfe46754 100644 --- a/apps/extension/PRIVACY.md +++ b/apps/extension/PRIVACY.md @@ -1,6 +1,6 @@ # BrowserSkill — Privacy Policy -**Last updated:** May 25, 2026 +**Last updated:** August 20, 2026 This Privacy Policy describes how the **BrowserSkill** browser extension (the "Extension") handles information when you install and use it. BrowserSkill is published as part of the open-source [BrowserSkill](https://github.com/Tencent/BrowserSkill) project. The source code is publicly auditable. @@ -14,7 +14,7 @@ BrowserSkill is a local automation bridge that lets AI coding agents (such as Cu ## 2. Single Purpose -The Extension's single purpose is to expose browser automation primitives (navigation, DOM observation, screenshots, clicks, form filling, tab management) to a locally running BrowserSkill daemon over a WebSocket connection on `127.0.0.1`, so that an AI agent invoked by the user can interact with web pages on the user's behalf. +The Extension's single purpose is to expose browser automation primitives (navigation, DOM observation, screenshots, clicks, form filling, task-scoped file transfer, and tab management) to a locally running BrowserSkill daemon over a WebSocket connection on `127.0.0.1`, so that an AI agent invoked by the user can interact with web pages on the user's behalf. ## 3. Data the Extension Accesses @@ -26,6 +26,7 @@ Depending on the commands the user (via their AI agent) sends to the local daemo | **User input simulated by the agent** | Mouse clicks, keystrokes, and form values that the AI agent dispatches through the Chrome DevTools Protocol (CDP). | Required to perform automation actions the user has asked the agent to do. | | **Tab and window metadata** | Tab IDs, URLs, titles, window IDs of the Agent Window and any tabs the user explicitly authorizes. | Required to target automation commands at the correct tab/window. | | **Local extension storage** | A randomly generated 8-character instance ID and an optional user-supplied label. | Used so the local daemon can recognize this browser instance across reconnects. No personal data is stored. | +| **File transfers requested by the agent** | Local files explicitly supplied to `bsk upload`, and the file created by a single `bsk download` action. | Required to attach a task file to a web page or return a browser-generated download to the invoking local agent. | | **OS notifications** | Permission to display a system notification when the agent requests to "borrow" one of the user's existing tabs. | Required to obtain explicit, per-tab user consent before the agent touches any pre-existing tab. | ## 4. Data the Extension Does **Not** Collect @@ -34,7 +35,7 @@ BrowserSkill does **not**: - Send any data to remote servers, the Extension's authors, or any third party. - Call any LLM, AI, or cloud API. The Extension contains no API keys, model identifiers, or remote endpoints. -- Read or transmit cookies, browsing history, bookmarks, downloads, saved passwords, or autofill data. +- Read or transmit cookies, browsing history, bookmarks, saved passwords, or autofill data. It observes only the download initiated by an active `bsk download` call, not download history generally. - Use webcam, microphone, geolocation, or any device sensor. - Include analytics, telemetry, crash reporting, advertising SDKs, or fingerprinting code. - Track users across websites or across sessions. @@ -50,6 +51,7 @@ The Extension requests the following Chrome permissions. Each is used solely for - **`alarms`** — Periodically wake the service worker to keep the local WebSocket connection alive. - **`idle`** — Detect when the device returns from idle/locked so the Extension can promptly re-establish the local WebSocket connection after the machine wakes. No idle data is stored or transmitted. - **`notifications`** — Show a system notification to obtain user approval before the agent borrows a user-owned tab. +- **`downloads`** — Correlate and route the one browser download initiated by an active `bsk download` command. If that claimed transaction fails, BrowserSkill cancels an in-progress file or removes its completed temporary browser file. It is not used to enumerate download history or alter unclaimed downloads. - **`storage`** — Persist a random instance ID and optional label in `chrome.storage.local`. - **Host permission ``** — Inject a small status overlay (showing "Agent Active") on pages controlled by the agent, and enable automation across whatever sites the user directs the agent to. The Extension does **not** read or transmit page content from sites the agent is not actively driving. @@ -61,6 +63,7 @@ All Extension activity stays on the user's local device. The only network traffi - The instance ID and optional label persist in `chrome.storage.local` until the user uninstalls the Extension or clears extension storage. - Page content, screenshots, DOM snapshots, and other observed data are returned to the local daemon in response to commands and are **not retained by the Extension**. They live only as long as the agent's tool call. +- Upload and download bytes are staged by the local daemon in a private, session-scoped directory. Download staging is removed after it is copied to the requested destination. Upload staging is retained until the session ends so a later form submission can still read the attached file. Remaining staging is removed when the session ends or disconnects, or when the daemon next starts after a crash. ## 8. User Control diff --git a/apps/extension/src/tools/__tests__/dispatcher.test.ts b/apps/extension/src/tools/__tests__/dispatcher.test.ts index c05cdb2a..ee566f53 100644 --- a/apps/extension/src/tools/__tests__/dispatcher.test.ts +++ b/apps/extension/src/tools/__tests__/dispatcher.test.ts @@ -232,6 +232,107 @@ describe("ToolDispatcher", () => { }); }); + it("bypasses and restores the control overlay for an upload trigger click", async () => { + const sendMessage = vi.fn(async () => undefined); + vi.stubGlobal("chrome", { + tabs: { + get: vi.fn(async () => ({ id: 7, windowId: 4242, active: true })), + query: vi.fn(async () => [{ id: 7, windowId: 4242, active: true }]), + sendMessage, + }, + }); + const { transport, sent, deliver } = fakeTransport(); + const sessions = new SessionManager({ + agentWindow: { + create: vi.fn(async () => 4242), + remove: vi.fn(async () => {}), + ensureActiveTab: vi.fn(async () => {}), + }, + }); + const ctx = await sessions.start("aa11"); + ctx.refStore.set("e1", 123, { tabId: 7 }); + const send = vi.fn(async (_tabId: number, method: string, params?: object) => { + if (method === "Page.getLayoutMetrics") { + return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } } as T; + } + if (method === "DOM.getContentQuads") { + return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] } as T; + } + if (method === "DOM.resolveNode") { + return { object: { objectId: "trigger-object" } } as T; + } + if (method === "DOM.describeNode") return { node: { backendNodeId: 456 } } as T; + if (method === "Runtime.callFunctionOn") { + const declaration = (params as { functionDeclaration?: string }).functionDeclaration ?? ""; + if (declaration.includes("count: state.inputs.length")) { + return { result: { value: { count: 1, multiple: false } } } as T; + } + if (declaration.includes("inputs[0]")) { + return { result: { objectId: "input-object" } } as T; + } + return { result: { value: true } } as T; + } + if (method === "Runtime.evaluate") { + const expression = (params as { expression?: string }).expression ?? ""; + if (expression.includes("overlayDetails")) { + return { result: { value: { hitIndex: 0 } } } as T; + } + if (expression.includes("overlayHostPresent")) { + return { + result: { + value: { overlayHostPresent: true, overlayHostConnected: true }, + }, + } as T; + } + if (expression.includes("count:")) { + return { result: { value: { count: 1, multiple: false } } } as T; + } + if (expression.includes("?.inputs[0]")) { + return { result: { objectId: "input-object" } } as T; + } + return { result: { value: true } } as T; + } + return {} as T; + }); + const cdp = { + send, + detachSession: vi.fn(async () => {}), + ensureNetworkCapture: vi.fn(async () => {}), + networkEntriesSince: vi.fn(() => ({ + tab_id: 7, + entries: [], + next_since: 0, + truncated: false, + })), + setDeviceMetricsOverride: vi.fn(async () => {}), + clearDeviceMetricsOverride: vi.fn(async () => {}), + setUserAgentOverride: vi.fn(async () => {}), + setTouchEmulationEnabled: vi.fn(async () => {}), + }; + const dispatcher = new ToolDispatcher({ transport, sessions, cdp: cdp as TestDispatcherCdp }); + dispatcher.start(); + + deliver( + makeRequest("tool.upload", { + session_id: "aa11", + ref: "@e1", + files: [{ transfer_id: "tr_1", name: "test.png", staged_path: "/stage/test.png" }], + }), + ); + await flushMicrotasks(); + await vi.waitFor(() => expect(sent).toHaveLength(1)); + + expect(sent[0]).toMatchObject({ result: { tab_id: 7, file_names: ["test.png"] } }); + expect(sendMessage).toHaveBeenNthCalledWith(1, 7, { + type: "bh-automation-bypass", + enabled: true, + }); + expect(sendMessage).toHaveBeenNthCalledWith(2, 7, { + type: "bh-automation-bypass", + enabled: false, + }); + }); + it("detaches CDP state before stopping a session", async () => { const { transport, sent, deliver } = fakeTransport(); const sessions = new SessionManager({ diff --git a/apps/extension/src/tools/__tests__/file-transfer.test.ts b/apps/extension/src/tools/__tests__/file-transfer.test.ts new file mode 100644 index 00000000..f3f365c3 --- /dev/null +++ b/apps/extension/src/tools/__tests__/file-transfer.test.ts @@ -0,0 +1,708 @@ +import { describe, expect, it, vi } from "vitest"; +import { SessionManager } from "@/session-manager/manager"; +import { type DownloadsApi, handleDownload } from "../download"; +import { captureBrowserDownload } from "../download-capture"; +import { uploadThroughActivatedFileInput } from "../file-input-transaction"; +import type { ResolvedActionTarget } from "../interaction"; +import type { CdpRunner } from "../shared"; +import { handleUpload } from "../upload"; + +function sessions() { + return new SessionManager({ + agentWindow: { + create: vi.fn(async () => 100), + remove: vi.fn(async () => {}), + ensureActiveTab: vi.fn(async () => {}), + }, + }); +} + +function tabsApi() { + return { + get: vi.fn( + async (tabId: number) => ({ id: tabId, windowId: 100, active: true }) as chrome.tabs.Tab, + ), + query: vi.fn(async () => [{ id: 4, windowId: 100, active: true } as chrome.tabs.Tab]), + }; +} + +function fakeEvent unknown>() { + const listeners = new Set(); + return { + addListener: (listener: T) => listeners.add(listener), + removeListener: (listener: T) => listeners.delete(listener), + emit: (...args: Parameters) => { + for (const listener of listeners) listener(...args); + }, + }; +} + +function actionTarget(frameId?: string): ResolvedActionTarget { + return { + tab: { tabId: 4, windowId: 100, active: true }, + backendNodeId: 123, + cdpTarget: { tabId: 4 }, + ...(frameId ? { frameId } : {}), + usedRef: "e3", + }; +} + +function uploadCdp( + options: { + inputCount?: number; + multiple?: boolean; + chooser?: { frameId?: string; backendNodeId?: number; mode?: string }; + pendingResolve?: boolean; + } = {}, +) { + const calls: Array<{ method: string; params?: object }> = []; + let cdpEvent: Parameters>[0] | undefined; + const send = vi.fn(async (_tabId: number, method: string, params?: object) => { + calls.push({ method, params }); + if (method === "Page.setInterceptFileChooserDialog") return {}; + if (method === "Page.getLayoutMetrics") + return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } }; + if (method === "DOM.getContentQuads") return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] }; + if (method === "DOM.resolveNode") { + if (options.pendingResolve) return new Promise(() => {}); + return { object: { objectId: "trigger-object" } }; + } + if (method === "Runtime.callFunctionOn") { + const declaration = (params as { functionDeclaration?: string }).functionDeclaration ?? ""; + if (declaration.includes("Object.defineProperty")) return { result: { value: true } }; + if (declaration.includes("count: state.inputs.length")) { + return { + result: { + value: { + count: options.inputCount ?? 1, + multiple: options.multiple ?? true, + }, + }, + }; + } + if (declaration.includes("inputs[0]")) { + return { result: { objectId: "input-object" } }; + } + return { result: { value: true } }; + } + if (method === "DOM.describeNode") return { node: { backendNodeId: 456 } }; + if ( + method === "Input.dispatchMouseEvent" && + (params as { type?: string }).type === "mousePressed" && + options.chooser + ) { + cdpEvent?.({ tabId: 4 }, "Page.fileChooserOpened", options.chooser); + } + return {}; + }); + const cdp: CdpRunner = { + send: send as unknown as CdpRunner["send"], + onEvent: (handler) => { + cdpEvent = handler; + return { dispose: vi.fn() }; + }, + }; + return { + calls, + emitChooser: (event: { frameId?: string; backendNodeId?: number; mode?: string }) => + cdpEvent?.({ tabId: 4 }, "Page.fileChooserOpened", event), + cdp, + }; +} + +describe("file transfer tools", () => { + it("captures the file input activated by the requested click and injects only staged paths", async () => { + const manager = sessions(); + const ctx = await manager.start("s1"); + ctx.refStore.set("e3", 123, { tabId: 4 }); + const { cdp, calls } = uploadCdp(); + + const result = await handleUpload( + manager, + { + session_id: "s1", + ref: "@e3", + files: [ + { transfer_id: "tr_1", name: "one.png", staged_path: "/private/stage/one" }, + { transfer_id: "tr_2", name: "two.png", staged_path: "/private/stage/two" }, + ], + }, + { cdp, tabsApi: tabsApi() }, + ); + + expect(result).toMatchObject({ tab_id: 4, file_names: ["one.png", "two.png"] }); + expect(calls[0]).toEqual({ + method: "Page.setInterceptFileChooserDialog", + params: { enabled: true }, + }); + expect(calls).toContainEqual({ + method: "Page.setInterceptFileChooserDialog", + params: { enabled: false, cancel: true }, + }); + expect(calls).toContainEqual({ + method: "DOM.setFileInputFiles", + params: { files: ["/private/stage/one", "/private/stage/two"], backendNodeId: 456 }, + }); + }); + + it("fails immediately when the trigger does not activate a file input", async () => { + const { cdp } = uploadCdp({ inputCount: 0 }); + const result = await uploadThroughActivatedFileInput({ + cdp, + actionTarget: actionTarget(), + files: ["/private/stage/one"], + timeoutMs: 100, + trigger: async () => ({ tab_id: 4, x: 10, y: 10 }), + }); + + expect(result).toMatchObject({ + code: "unsupported", + data: { reason: "file_input_not_activated", phase: "resolve_input" }, + }); + }); + + it("fails before clicking when chooser interception is unavailable", async () => { + const trigger = vi.fn(async () => ({ tab_id: 4, x: 10, y: 10 })); + const cdp: CdpRunner = { + send: vi.fn(async (_tabId: number, method: string) => { + if (method === "Page.setInterceptFileChooserDialog") { + throw new Error("method unavailable"); + } + return {}; + }) as CdpRunner["send"], + }; + + const result = await uploadThroughActivatedFileInput({ + cdp, + actionTarget: actionTarget(), + files: ["/private/stage/one"], + timeoutMs: 100, + trigger, + }); + + expect(result).toMatchObject({ + code: "cdp_failed", + data: { effect_state: "none", phase: "arm_interception" }, + }); + expect(trigger).not.toHaveBeenCalled(); + }); + + it("uses an exact chooser event as an independent input-location signal", async () => { + const { cdp, calls, emitChooser } = uploadCdp({ inputCount: 0 }); + const result = await uploadThroughActivatedFileInput({ + cdp, + actionTarget: actionTarget("f1"), + files: ["/private/stage/one"], + timeoutMs: 100, + trigger: async () => { + emitChooser({ frameId: "f1", backendNodeId: 789, mode: "selectSingle" }); + return { tab_id: 4, x: 10, y: 10 }; + }, + }); + + expect(result).toMatchObject({ multiple: false }); + expect(calls).toContainEqual({ + method: "DOM.setFileInputFiles", + params: { files: ["/private/stage/one"], backendNodeId: 789 }, + }); + }); + + it("reports a File System Access picker without waiting for a timeout", async () => { + const { cdp, emitChooser } = uploadCdp({ + inputCount: 0, + }); + const result = await uploadThroughActivatedFileInput({ + cdp, + actionTarget: actionTarget("f1"), + files: ["/private/stage/one"], + timeoutMs: 100, + trigger: async () => { + emitChooser({ frameId: "f1", mode: "selectSingle" }); + return { tab_id: 4, x: 10, y: 10 }; + }, + }); + + expect(result).toMatchObject({ + code: "unsupported", + message: "upload trigger invoked a non-input file picker", + data: { reason: "file_input_not_activated", phase: "resolve_input" }, + }); + }); + + it("bounds a stuck file-input probe", async () => { + const { cdp } = uploadCdp({ pendingResolve: true }); + + const result = await uploadThroughActivatedFileInput({ + cdp, + actionTarget: actionTarget(), + files: ["/private/stage/one"], + timeoutMs: 5, + trigger: vi.fn(), + }); + + expect(result).toMatchObject({ + code: "timeout", + data: { reason: "file_input_probe_failed", phase: "arm_input_probe" }, + }); + }); + + it("marks a timed-out file assignment unknown and detaches browser state", async () => { + const fixture = uploadCdp(); + const originalSend = fixture.cdp.send; + const detach = vi.fn(async () => {}); + fixture.cdp.detach = detach; + fixture.cdp.send = vi.fn((tabId: number, method: string, params?: object) => { + if (method === "DOM.setFileInputFiles") return new Promise(() => {}); + return originalSend(tabId, method, params); + }) as CdpRunner["send"]; + + const result = await uploadThroughActivatedFileInput({ + cdp: fixture.cdp, + actionTarget: actionTarget(), + files: ["/private/stage/one"], + timeoutMs: 10, + trigger: async () => ({ tab_id: 4, x: 10, y: 10 }), + }); + + expect(result).toMatchObject({ + code: "timeout", + data: { effect_state: "unknown", phase: "set_files" }, + }); + expect(detach).toHaveBeenCalledWith(4); + }); + + it("routes one exact-target download through a browser-relative capability", async () => { + const manager = sessions(); + const ctx = await manager.start("s1"); + ctx.refStore.set("e3", 123, { tabId: 4 }); + const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>(); + const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>(); + const onDeterminingFilename = + fakeEvent< + ( + item: chrome.downloads.DownloadItem, + suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void, + ) => void | true + >(); + const initial = { + id: 9, + url: "https://example.test/result.zip", + finalUrl: "https://example.test/result.zip", + filename: "result.zip", + state: "in_progress", + fileSize: -1, + totalBytes: 12, + mime: "application/zip", + danger: "safe", + } as chrome.downloads.DownloadItem; + const completed = { + ...initial, + filename: "/profile/Downloads/BrowserSkill/tr_1/result.zip", + state: "complete", + fileSize: 12, + } as chrome.downloads.DownloadItem; + const downloads: DownloadsApi = { + onCreated, + onChanged, + onDeterminingFilename, + search: vi.fn(async () => [completed]), + cancel: vi.fn(async () => {}), + removeFile: vi.fn(async () => {}), + }; + let cdpEvent: Parameters>[0] | undefined; + let suggested: chrome.downloads.DownloadFilenameSuggestion | undefined; + const send = vi.fn(async (_tabId: number, method: string, params?: object) => { + if (method === "Page.getLayoutMetrics") + return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } }; + if (method === "DOM.getContentQuads") return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] }; + if ( + method === "Input.dispatchMouseEvent" && + (params as { type?: string }).type === "mousePressed" + ) { + cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", { + url: initial.url, + suggestedFilename: "result.zip", + }); + await new Promise((resolve) => { + onDeterminingFilename.emit(initial, (value) => { + suggested = value; + resolve(); + }); + }); + onCreated.emit(completed); + } + return {}; + }); + const cdp: CdpRunner = { + send: send as unknown as CdpRunner["send"], + onEvent: (handler) => { + cdpEvent = handler; + return { dispose: vi.fn() }; + }, + }; + + const result = await handleDownload( + manager, + { session_id: "s1", ref: "@e3", browser_relative_dir: "BrowserSkill/tr_1" }, + { cdp, tabsApi: tabsApi(), downloads }, + ); + + expect(suggested).toEqual({ + filename: "BrowserSkill/tr_1/result.zip", + conflictAction: "overwrite", + }); + expect(result).toMatchObject({ + tab_id: 4, + suggested_filename: "result.zip", + byte_size: 12, + browser_path: completed.filename, + }); + }); + + it("does not claim a download without an intent from the exact target", async () => { + const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>(); + const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>(); + const onDeterminingFilename = + fakeEvent< + ( + item: chrome.downloads.DownloadItem, + suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void, + ) => void | true + >(); + const downloads: DownloadsApi = { + onCreated, + onChanged, + onDeterminingFilename, + search: vi.fn(async () => []), + cancel: vi.fn(async () => {}), + removeFile: vi.fn(async () => {}), + }; + let cdpEvent: Parameters>[0] | undefined; + const cdp: CdpRunner = { + send: vi.fn(async () => ({})) as unknown as CdpRunner["send"], + onEvent: (handler) => { + cdpEvent = handler; + return { dispose: vi.fn() }; + }, + }; + const unrelated = { + id: 17, + url: "https://example.test/unrelated.zip", + finalUrl: "https://example.test/unrelated.zip", + filename: "unrelated.zip", + state: "in_progress", + } as chrome.downloads.DownloadItem; + let defaultSuggestionCalled = false; + + const result = await captureBrowserDownload({ + cdp, + target: { tabId: 4, sessionId: "expected-child" }, + downloads, + browserRelativeDir: "BrowserSkill/tr_1", + timeoutMs: 5, + trigger: async () => { + cdpEvent?.({ tabId: 4, sessionId: "other-child" }, "Page.downloadWillBegin", { + url: unrelated.url, + suggestedFilename: unrelated.filename, + }); + onDeterminingFilename.emit(unrelated, (suggestion) => { + defaultSuggestionCalled = suggestion === undefined; + }); + return { tab_id: 4, x: 10, y: 10 }; + }, + }); + + expect(defaultSuggestionCalled).toBe(true); + expect(downloads.cancel).not.toHaveBeenCalled(); + expect(result).toMatchObject({ + code: "cdp_failed", + data: { reason: "download_capture_failed" }, + }); + }); + + it("correlates a filename candidate that arrives before the CDP intent", async () => { + const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>(); + const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>(); + const onDeterminingFilename = + fakeEvent< + ( + item: chrome.downloads.DownloadItem, + suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void, + ) => void | true + >(); + const initial = { + id: 21, + url: "https://example.test/candidate-first.bin", + finalUrl: "https://example.test/candidate-first.bin", + filename: "candidate-first.bin", + state: "in_progress", + fileSize: -1, + totalBytes: 4, + bytesReceived: 0, + } as chrome.downloads.DownloadItem; + const complete = { + ...initial, + state: "complete", + fileSize: 4, + } as chrome.downloads.DownloadItem; + const downloads: DownloadsApi = { + onCreated, + onChanged, + onDeterminingFilename, + search: vi.fn(async () => [complete]), + cancel: vi.fn(async () => {}), + removeFile: vi.fn(async () => {}), + }; + let cdpEvent: Parameters>[0] | undefined; + const cdp: CdpRunner = { + send: vi.fn(async () => ({})) as CdpRunner["send"], + onEvent: (handler) => { + cdpEvent = handler; + return { dispose: vi.fn() }; + }, + }; + let suggestion: chrome.downloads.DownloadFilenameSuggestion | undefined; + + const result = await captureBrowserDownload({ + cdp, + target: { tabId: 4 }, + downloads, + browserRelativeDir: "BrowserSkill/tr_21", + timeoutMs: 1_000, + trigger: async () => { + const suggested = new Promise((resolve) => { + onDeterminingFilename.emit(initial, (value) => { + suggestion = value; + resolve(); + }); + }); + cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", { + url: initial.url, + suggestedFilename: initial.filename, + }); + await suggested; + onCreated.emit(complete); + return { tab_id: 4, x: 10, y: 10 }; + }, + }); + + expect(suggestion).toEqual({ + filename: "BrowserSkill/tr_21/candidate-first.bin", + conflictAction: "overwrite", + }); + expect(result).toMatchObject({ item: { id: 21, state: "complete" } }); + }); + + it("removes a completed download when the final size exceeds the limit", async () => { + const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>(); + const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>(); + const onDeterminingFilename = + fakeEvent< + ( + item: chrome.downloads.DownloadItem, + suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void, + ) => void | true + >(); + const initial = { + id: 22, + url: "https://example.test/oversized.bin", + finalUrl: "https://example.test/oversized.bin", + filename: "oversized.bin", + state: "in_progress", + fileSize: -1, + totalBytes: -1, + bytesReceived: 0, + } as chrome.downloads.DownloadItem; + const complete = { + ...initial, + filename: "/profile/Downloads/BrowserSkill/tr_22/oversized.bin", + state: "complete", + fileSize: 8, + totalBytes: 8, + bytesReceived: 8, + } as chrome.downloads.DownloadItem; + const downloads: DownloadsApi = { + onCreated, + onChanged, + onDeterminingFilename, + search: vi.fn(async () => [complete]), + cancel: vi.fn(async () => {}), + removeFile: vi.fn(async () => {}), + }; + let cdpEvent: Parameters>[0] | undefined; + const cdp: CdpRunner = { + send: vi.fn(async () => ({})) as CdpRunner["send"], + onEvent: (handler) => { + cdpEvent = handler; + return { dispose: vi.fn() }; + }, + }; + + const result = await captureBrowserDownload({ + cdp, + target: { tabId: 4 }, + downloads, + browserRelativeDir: "BrowserSkill/tr_22", + maxByteSize: 4, + timeoutMs: 1_000, + trigger: async () => { + cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", { + url: initial.url, + suggestedFilename: initial.filename, + }); + await new Promise((resolve) => { + onDeterminingFilename.emit(initial, () => resolve()); + }); + onCreated.emit(initial); + onChanged.emit({ id: initial.id, state: { current: "complete" } }); + return { tab_id: 4, x: 10, y: 10 }; + }, + }); + + expect(result).toMatchObject({ + code: "cdp_failed", + data: { reason: "download_capture_failed", effect_state: "committed" }, + }); + expect(downloads.removeFile).toHaveBeenCalledWith(initial.id); + expect(downloads.cancel).not.toHaveBeenCalled(); + }); + + it("reconciles a download that completes while cancellation is being requested", async () => { + const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>(); + const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>(); + const onDeterminingFilename = + fakeEvent< + ( + item: chrome.downloads.DownloadItem, + suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void, + ) => void | true + >(); + const initial = { + id: 23, + url: "https://example.test/racing.bin", + finalUrl: "https://example.test/racing.bin", + filename: "racing.bin", + state: "in_progress", + fileSize: -1, + totalBytes: -1, + bytesReceived: 0, + } as chrome.downloads.DownloadItem; + const complete = { + ...initial, + filename: "/profile/Downloads/BrowserSkill/tr_23/racing.bin", + state: "complete", + fileSize: 4, + totalBytes: 4, + bytesReceived: 4, + } as chrome.downloads.DownloadItem; + const downloads: DownloadsApi = { + onCreated, + onChanged, + onDeterminingFilename, + search: vi.fn().mockResolvedValueOnce([initial]).mockResolvedValueOnce([complete]), + cancel: vi.fn(async () => { + throw new Error("download already complete"); + }), + removeFile: vi.fn(async () => {}), + }; + let cdpEvent: Parameters>[0] | undefined; + const cdp: CdpRunner = { + send: vi.fn(async () => ({})) as CdpRunner["send"], + onEvent: (handler) => { + cdpEvent = handler; + return { dispose: vi.fn() }; + }, + }; + + const result = await captureBrowserDownload({ + cdp, + target: { tabId: 4 }, + downloads, + browserRelativeDir: "BrowserSkill/tr_23", + timeoutMs: 80, + trigger: async () => { + cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", { + url: initial.url, + suggestedFilename: initial.filename, + }); + await new Promise((resolve) => { + onDeterminingFilename.emit(initial, () => resolve()); + }); + onCreated.emit(initial); + return { tab_id: 4, x: 10, y: 10 }; + }, + }); + + expect(result).toMatchObject({ + code: "cdp_failed", + data: { reason: "download_capture_failed", effect_state: "committed" }, + }); + expect(downloads.cancel).toHaveBeenCalledWith(initial.id); + expect(downloads.removeFile).toHaveBeenCalledWith(initial.id); + expect(result).not.toMatchObject({ data: { cleanup_state: "failed" } }); + }); + + it("rejects ambiguous attribution without cancelling either unclaimed download", async () => { + const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>(); + const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>(); + const onDeterminingFilename = + fakeEvent< + ( + item: chrome.downloads.DownloadItem, + suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void, + ) => void | true + >(); + const downloads: DownloadsApi = { + onCreated, + onChanged, + onDeterminingFilename, + search: vi.fn(async () => []), + cancel: vi.fn(async () => {}), + removeFile: vi.fn(async () => {}), + }; + let cdpEvent: Parameters>[0] | undefined; + const cdp: CdpRunner = { + send: vi.fn(async () => ({})) as CdpRunner["send"], + onEvent: (handler) => { + cdpEvent = handler; + return { dispose: vi.fn() }; + }, + }; + const defaults: number[] = []; + const candidate = (id: number) => + ({ + id, + url: "https://example.test/same.bin", + finalUrl: "https://example.test/same.bin", + filename: "same.bin", + state: "in_progress", + }) as chrome.downloads.DownloadItem; + + const result = await captureBrowserDownload({ + cdp, + target: { tabId: 4 }, + downloads, + browserRelativeDir: "BrowserSkill/tr_ambiguous", + timeoutMs: 100, + trigger: async () => { + cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", { + url: "https://example.test/same.bin", + suggestedFilename: "same.bin", + }); + for (const id of [31, 32]) { + onDeterminingFilename.emit(candidate(id), (value) => { + if (value === undefined) defaults.push(id); + }); + } + return { tab_id: 4, x: 10, y: 10 }; + }, + }); + + expect(defaults.sort()).toEqual([31, 32]); + expect(downloads.cancel).not.toHaveBeenCalled(); + expect(result).toMatchObject({ + code: "cdp_failed", + data: { effect_state: "unknown", phase: "attribution" }, + }); + }); +}); diff --git a/apps/extension/src/tools/dispatcher.ts b/apps/extension/src/tools/dispatcher.ts index e0a5ac4d..f6366f27 100644 --- a/apps/extension/src/tools/dispatcher.ts +++ b/apps/extension/src/tools/dispatcher.ts @@ -4,6 +4,7 @@ import type { Transport } from "@/transport/transport"; import type { ClickParams, ConsoleParams, + DownloadParams, EmulateParams, EvaluateParams, FillParams, @@ -28,10 +29,12 @@ import type { ScreenshotParams, SelectParams, SnapshotParams, + UploadParams, WaitForNavigationParams, } from "@/transport/types"; import { isRequestFrame } from "@/transport/types"; import { handleConsole } from "./console"; +import { handleDownload } from "./download"; import { type EmulateCdpRunner, handleEmulate } from "./emulate"; import { handleEvaluate } from "./evaluate"; import { handleRequestHelp } from "./human-loop"; @@ -79,6 +82,7 @@ import { type TabReturnParams, type TabSelectParams, } from "./tabs"; +import { handleUpload } from "./upload"; import { handleWaitForNavigation } from "./waits"; import { handleWindowResize, type WindowResizeParams } from "./window"; @@ -513,6 +517,40 @@ export class ToolDispatcher { ), signal, ); + case "tool.upload": + return this.withHoverReleaseForRequest( + req.params as UploadParams, + () => + this.cdp + ? handleUpload(this.sessions, req.params as UploadParams, { + cdp: this.cdp, + tabsApi: chromeTabsApi, + signal, + bypassOverlay, + }) + : Promise.resolve({ + code: "unsupported", + message: "upload requires CDP", + } satisfies RpcError), + signal, + ); + case "tool.download": + return this.withHoverReleaseForRequest( + req.params as DownloadParams, + () => + this.cdp + ? handleDownload(this.sessions, req.params as DownloadParams, { + cdp: this.cdp, + tabsApi: chromeTabsApi, + signal, + bypassOverlay, + }) + : Promise.resolve({ + code: "unsupported", + message: "download requires CDP", + } satisfies RpcError), + signal, + ); case "tool.evaluate": return handleEvaluate( this.sessions, @@ -719,6 +757,8 @@ function sessionIdForBrowserControlMethod(req: RequestFrame): string | null { case "tool.fill": case "tool.press": case "tool.select": + case "tool.upload": + case "tool.download": case "tool.evaluate": case "tool.observe": case "tool.request_help": diff --git a/apps/extension/src/tools/download-capture.ts b/apps/extension/src/tools/download-capture.ts new file mode 100644 index 00000000..6e75bff4 --- /dev/null +++ b/apps/extension/src/tools/download-capture.ts @@ -0,0 +1,364 @@ +// Order-independent coordinator for one browser download. CDP supplies the +// exact target/frame intent while chrome.downloads supplies the download id +// and filename routing hook; neither event is assumed to arrive first. + +import type { CdpTarget } from "@/browser-driver/frame-graph"; +import type { ClickResult, RpcError, TransferEffectState } from "@/transport/types"; +import { transferError } from "./errors"; +import { type CdpRunner, isRpcError } from "./shared"; + +const CORRELATION_GRACE_MS = 750; +const UNIQUE_SETTLE_MS = 50; +const SIZE_POLL_MS = 250; + +type DeterminingFilenameListener = ( + item: chrome.downloads.DownloadItem, + suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void, +) => void | true; + +interface ListenerEvent { + addListener(listener: T): void; + removeListener(listener: T): void; +} + +export interface DownloadsApi { + onCreated: ListenerEvent<(item: chrome.downloads.DownloadItem) => void>; + onChanged: ListenerEvent<(delta: chrome.downloads.DownloadDelta) => void>; + onDeterminingFilename: ListenerEvent; + search(query: chrome.downloads.DownloadQuery): Promise; + cancel(downloadId: number): Promise; + removeFile(downloadId: number): Promise; +} + +export const chromeDownloadsApi: DownloadsApi = { + get onCreated() { + return chrome.downloads.onCreated; + }, + get onChanged() { + return chrome.downloads.onChanged; + }, + get onDeterminingFilename() { + return chrome.downloads.onDeterminingFilename; + }, + search: (query) => chrome.downloads.search(query), + cancel: (id) => chrome.downloads.cancel(id), + removeFile: (id) => chrome.downloads.removeFile(id), +}; + +export interface DownloadCaptureOptions { + cdp: CdpRunner; + target: CdpTarget; + expectedFrameId?: string; + downloads: DownloadsApi; + browserRelativeDir: string; + maxByteSize?: number; + timeoutMs: number; + signal?: AbortSignal; + trigger(): Promise; +} + +export interface DownloadCaptureResult { + click: ClickResult; + item: chrome.downloads.DownloadItem; +} + +interface DownloadIntent { + url: string; + suggestedFilename: string; + frameId?: string; +} + +interface DownloadCandidate { + item: chrome.downloads.DownloadItem; + suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void; + suggested: boolean; + graceTimer: ReturnType; +} + +function safeBasename(filename: string): string { + const basename = filename.split(/[\\/]/).pop()?.trim(); + return basename && basename !== "." && basename !== ".." ? basename : "download"; +} + +function sameTarget(source: { tabId?: number; sessionId?: string }, target: CdpTarget): boolean { + return source.tabId === target.tabId && source.sessionId === target.sessionId; +} + +function matchesIntent(item: chrome.downloads.DownloadItem, intent: DownloadIntent): boolean { + const urlMatches = item.url === intent.url || item.finalUrl === intent.url; + return urlMatches && safeBasename(item.filename) === safeBasename(intent.suggestedFilename); +} + +function knownSize(item: chrome.downloads.DownloadItem): number | undefined { + if (item.fileSize >= 0) return item.fileSize; + if (item.totalBytes >= 0) return item.totalBytes; + return undefined; +} + +function captureError( + message: string, + effectState: TransferEffectState, + phase: string, + cleanupFailed = false, +): RpcError { + return transferError("cdp_failed", "download_capture_failed", message, { + effectState, + phase, + ...(cleanupFailed ? { cleanupState: "failed" } : {}), + }); +} + +async function cleanupClaimedDownload(downloads: DownloadsApi, downloadId: number): Promise { + const lookup = async () => (await downloads.search({ id: downloadId }))[0]; + const item = await lookup(); + if (!item || item.state === "interrupted") return; + if (item.state === "complete") { + await downloads.removeFile(downloadId); + return; + } + + try { + await downloads.cancel(downloadId); + } catch (cancelError) { + // Completion can win the race after the lookup but before cancellation. + // Reconcile against Chrome's authoritative state before declaring cleanup + // failed so every terminal state has one explicit cleanup path. + const reconciled = await lookup(); + if (!reconciled || reconciled.state === "interrupted") return; + if (reconciled.state === "complete") { + await downloads.removeFile(downloadId); + return; + } + throw cancelError; + } +} + +export async function captureBrowserDownload( + options: DownloadCaptureOptions, +): Promise { + let click: ClickResult | undefined; + let intent: DownloadIntent | undefined; + let capturedId: number | undefined; + let settled = false; + let succeeded = false; + let failureResult: RpcError | undefined; + let uniquenessTimer: ReturnType | undefined; + let operationTimer: ReturnType | undefined; + let sizePoll: ReturnType | undefined; + const candidates = new Map(); + const createdItems = new Map(); + + let resolveCompletion!: (item: chrome.downloads.DownloadItem) => void; + let rejectCompletion!: (error: Error) => void; + const completion = new Promise((resolve, reject) => { + resolveCompletion = resolve; + rejectCompletion = reject; + }); + const fail = (error: Error) => { + if (settled) return; + settled = true; + rejectCompletion(error); + }; + const complete = (item: chrome.downloads.DownloadItem) => { + if (settled) return; + const size = knownSize(item); + if (size !== undefined && options.maxByteSize !== undefined && size > options.maxByteSize) { + fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`)); + return; + } + settled = true; + resolveCompletion(item); + }; + const suggestDefault = (candidate: DownloadCandidate) => { + if (candidate.suggested) return; + candidate.suggested = true; + candidate.suggest(); + }; + const matchingCandidates = (): DownloadCandidate[] => { + const currentIntent = intent; + return currentIntent + ? [...candidates.values()].filter( + (candidate) => !candidate.suggested && matchesIntent(candidate.item, currentIntent), + ) + : []; + }; + + const claimUnique = () => { + uniquenessTimer = undefined; + if (settled || capturedId !== undefined || !intent) return; + const matches = matchingCandidates(); + if (matches.length !== 1) { + if (matches.length > 1) { + for (const candidate of matches) suggestDefault(candidate); + fail(new Error("download attribution is ambiguous")); + } + return; + } + const candidate = matches[0]; + candidate.suggested = true; + clearTimeout(candidate.graceTimer); + capturedId = candidate.item.id; + candidate.suggest({ + filename: `${options.browserRelativeDir}/${safeBasename(intent.suggestedFilename)}`, + conflictAction: "overwrite", + }); + const size = knownSize(candidate.item); + if (size !== undefined && options.maxByteSize !== undefined && size > options.maxByteSize) { + fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`)); + return; + } + const created = createdItems.get(candidate.item.id); + if (created?.state === "interrupted") { + fail(new Error(created.error ?? "download interrupted")); + } else if (created?.state === "complete") { + complete(created); + } + }; + const reconcile = () => { + if (settled || capturedId !== undefined || !intent) return; + const matches = matchingCandidates(); + if (matches.length > 1) { + for (const candidate of matches) suggestDefault(candidate); + fail(new Error("download attribution is ambiguous")); + return; + } + if (matches.length === 1 && !uniquenessTimer) { + uniquenessTimer = setTimeout(claimUnique, UNIQUE_SETTLE_MS); + } + }; + + const determiningListener: DeterminingFilenameListener = (item, suggest) => { + const candidate: DownloadCandidate = { + item, + suggest, + suggested: false, + graceTimer: setTimeout(() => { + suggestDefault(candidate); + candidates.delete(item.id); + if (intent && matchesIntent(item, intent) && capturedId === undefined) { + fail(new Error("download correlation grace elapsed before unique attribution")); + } + }, CORRELATION_GRACE_MS), + }; + candidates.set(item.id, candidate); + reconcile(); + return true; + }; + const createdListener = (item: chrome.downloads.DownloadItem) => { + createdItems.set(item.id, item); + if (capturedId !== item.id) return; + if (item.state === "interrupted") { + fail(new Error(item.error ?? "download interrupted")); + } else if (item.state === "complete") { + complete(item); + } + }; + const changedListener = async (delta: chrome.downloads.DownloadDelta) => { + if (capturedId === undefined || delta.id !== capturedId || settled) return; + if (delta.state?.current === "interrupted" || delta.error?.current) { + fail(new Error(delta.error?.current ?? "download interrupted")); + return; + } + if (delta.state?.current === "complete") { + try { + const [item] = await options.downloads.search({ id: delta.id }); + if (item) complete(item); + else fail(new Error("completed download disappeared")); + } catch (err) { + fail(err instanceof Error ? err : new Error(String(err))); + } + } + }; + const onAbort = () => fail(new DOMException("aborted", "AbortError")); + const cdpSubscription = options.cdp.onEvent?.((source, method, raw) => { + if (method !== "Page.downloadWillBegin" || !sameTarget(source, options.target)) return; + const event = raw as { url?: unknown; suggestedFilename?: unknown; frameId?: unknown }; + if (typeof event.url !== "string" || typeof event.suggestedFilename !== "string") return; + if (options.expectedFrameId && event.frameId !== options.expectedFrameId) { + fail(new Error("download originated from a different frame")); + return; + } + if (intent) { + fail(new Error("download trigger produced more than one browser download intent")); + return; + } + intent = { + url: event.url, + suggestedFilename: event.suggestedFilename, + ...(typeof event.frameId === "string" ? { frameId: event.frameId } : {}), + }; + reconcile(); + }); + if (!cdpSubscription) { + return captureError("CDP download intent subscription unavailable", "none", "arm"); + } + + options.downloads.onDeterminingFilename.addListener(determiningListener); + options.downloads.onCreated.addListener(createdListener); + options.downloads.onChanged.addListener(changedListener); + options.signal?.addEventListener("abort", onAbort, { once: true }); + operationTimer = setTimeout( + () => fail(new Error("download did not complete before timeout")), + options.timeoutMs, + ); + sizePoll = setInterval(() => { + if (capturedId === undefined || settled || options.maxByteSize === undefined) return; + void options.downloads + .search({ id: capturedId }) + .then(([item]) => { + if (!item || settled) return; + if (item.bytesReceived > (options.maxByteSize as number)) { + fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`)); + } + }) + .catch((err) => fail(err instanceof Error ? err : new Error(String(err)))); + }, SIZE_POLL_MS); + + try { + const triggered = await options.trigger(); + if (isRpcError(triggered)) { + void completion.catch(() => undefined); + const effect: TransferEffectState = + capturedId !== undefined ? "committed" : intent ? "unknown" : "none"; + failureResult = { + ...triggered, + data: { ...triggered.data, effect_state: effect, phase: "trigger" }, + }; + return failureResult; + } + click = triggered; + const item = await completion; + succeeded = true; + return { click, item }; + } catch (err) { + const effect: TransferEffectState = + capturedId !== undefined ? "committed" : click ? "unknown" : "none"; + failureResult = captureError( + err instanceof Error ? err.message : String(err), + effect, + capturedId !== undefined ? "download" : "attribution", + ); + return failureResult; + } finally { + settled = true; + if (operationTimer) clearTimeout(operationTimer); + if (uniquenessTimer) clearTimeout(uniquenessTimer); + if (sizePoll) clearInterval(sizePoll); + options.signal?.removeEventListener("abort", onAbort); + options.downloads.onDeterminingFilename.removeListener(determiningListener); + options.downloads.onCreated.removeListener(createdListener); + options.downloads.onChanged.removeListener(changedListener); + cdpSubscription.dispose(); + for (const candidate of candidates.values()) { + clearTimeout(candidate.graceTimer); + if (candidate.item.id !== capturedId) suggestDefault(candidate); + } + if (!succeeded && capturedId !== undefined) { + try { + await cleanupClaimedDownload(options.downloads, capturedId); + } catch { + if (failureResult?.data) failureResult.data.cleanup_state = "failed"; + } + } + } +} diff --git a/apps/extension/src/tools/download.ts b/apps/extension/src/tools/download.ts new file mode 100644 index 00000000..047e263d --- /dev/null +++ b/apps/extension/src/tools/download.ts @@ -0,0 +1,64 @@ +// Download orchestration: validate session/tab ownership, then delegate one +// browser-global chrome.downloads transaction to download-capture.ts. + +import type { SessionManager } from "@/session-manager/manager"; +import type { DownloadParams, DownloadResult, RpcError } from "@/transport/types"; +import { captureBrowserDownload, chromeDownloadsApi, type DownloadsApi } from "./download-capture"; +import { clickResolvedTarget, type InteractionDeps, resolveActionTarget } from "./interaction"; +import { enforceAgentWindow, isRpcError, lookupSession, resolveTargetTab } from "./shared"; + +let downloadActive = false; + +export type { DownloadsApi } from "./download-capture"; + +export interface DownloadDeps extends InteractionDeps { + downloads?: DownloadsApi; +} + +export async function handleDownload( + manager: SessionManager, + params: DownloadParams, + deps: DownloadDeps, +): Promise { + if (downloadActive) return { code: "invalid_params", message: "another bsk download is active" }; + downloadActive = true; + try { + const ctx = lookupSession(manager, params, "download"); + if (isRpcError(ctx)) return ctx; + const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi); + if (isRpcError(target)) return target; + const denied = enforceAgentWindow(ctx, target, "download"); + if (denied) return denied; + if (!params.browser_relative_dir) { + return { code: "invalid_params", message: "download requires a daemon capability directory" }; + } + const address = await resolveActionTarget(deps.cdp, ctx, target, params, "download"); + if (isRpcError(address)) return address; + + const capture = await captureBrowserDownload({ + cdp: deps.cdp, + target: address.cdpTarget, + downloads: deps.downloads ?? chromeDownloadsApi, + browserRelativeDir: params.browser_relative_dir, + maxByteSize: params.max_byte_size, + timeoutMs: params.timeout_ms ?? 120_000, + signal: deps.signal, + expectedFrameId: address.frameId, + trigger: () => clickResolvedTarget(ctx, address, {}, deps), + }); + if (isRpcError(capture)) return capture; + const { click, item } = capture; + return { + tab_id: target.tabId, + used_ref: click.used_ref, + used_selector: click.used_selector, + suggested_filename: item.filename.split(/[\\/]/).pop() ?? "download", + byte_size: item.fileSize >= 0 ? item.fileSize : item.totalBytes, + mime: item.mime || undefined, + danger: item.danger, + browser_path: item.filename, + }; + } finally { + downloadActive = false; + } +} diff --git a/apps/extension/src/tools/errors.ts b/apps/extension/src/tools/errors.ts index 88cfd78f..800a7655 100644 --- a/apps/extension/src/tools/errors.ts +++ b/apps/extension/src/tools/errors.ts @@ -2,10 +2,23 @@ // rendering. Extension handlers attach reasons here; human-facing copy // lives in bsk-cli `render_error.rs`. -import type { ErrorCode, RpcError, RpcErrorData, RpcErrorReason } from "@/transport/types"; +import type { + ErrorCode, + RpcError, + RpcErrorData, + RpcErrorReason, + TransferCleanupState, + TransferEffectState, +} from "@/transport/types"; export type { RpcErrorData, RpcErrorReason }; +export interface TransferErrorOptions { + effectState: TransferEffectState; + phase: string; + cleanupState?: TransferCleanupState; +} + export function rpcError( code: ErrorCode, reason: RpcErrorReason, @@ -15,3 +28,16 @@ export function rpcError( const data: RpcErrorData = { reason, ...extra }; return { code, message, data }; } + +export function transferError( + code: ErrorCode, + reason: RpcErrorReason, + message: string, + options: TransferErrorOptions, +): RpcError { + return rpcError(code, reason, message, { + effect_state: options.effectState, + phase: options.phase, + ...(options.cleanupState ? { cleanup_state: options.cleanupState } : {}), + }); +} diff --git a/apps/extension/src/tools/file-input-transaction.ts b/apps/extension/src/tools/file-input-transaction.ts new file mode 100644 index 00000000..3c78a8df --- /dev/null +++ b/apps/extension/src/tools/file-input-transaction.ts @@ -0,0 +1,484 @@ +// One upload transaction with an explicit browser-side commit boundary. +// Chrome interception prevents a native chooser from escaping automation; +// chooser events and a frame-scoped DOM probe are independent input-location +// signals, so event delivery is not required for standard file inputs. + +import type { ClickResult, RpcError, TransferEffectState } from "@/transport/types"; +import { transferError } from "./errors"; +import type { ResolvedActionTarget } from "./interaction"; +import { type CdpRunner, isRpcError, sendToCdpTarget } from "./shared"; + +const CLEANUP_TIMEOUT_MS = 1_000; +const ACTIVATION_GRACE_MS = 1_000; +const PROBE_INTERVAL_MS = 20; + +type UploadPhase = + | "arm_interception" + | "arm_input_probe" + | "trigger" + | "resolve_input" + | "set_files" + | "cleanup"; + +interface RuntimeReply { + result?: { value?: unknown; objectId?: string }; + exceptionDetails?: { text?: string; exception?: { description?: string } }; +} + +interface ChooserEvent { + frameId?: string; + backendNodeId?: number; + mode?: "selectSingle" | "selectMultiple"; +} + +export interface FileInputTransactionOptions { + cdp: CdpRunner; + actionTarget: ResolvedActionTarget; + files: string[]; + timeoutMs: number; + signal?: AbortSignal; + trigger(): Promise; +} + +export interface FileInputTransactionResult { + click: ClickResult; + multiple: boolean; +} + +class BoundedWaitError extends Error { + constructor( + readonly kind: "timeout" | "aborted", + message: string, + ) { + super(message); + } +} + +function remainingMs(deadline: number): number { + return Math.max(0, deadline - Date.now()); +} + +async function waitBounded( + promise: Promise, + deadline: number, + signal: AbortSignal | undefined, + timeoutMessage: string, +): Promise { + const remaining = remainingMs(deadline); + if (signal?.aborted) throw new BoundedWaitError("aborted", "upload transaction aborted"); + if (remaining === 0) throw new BoundedWaitError("timeout", timeoutMessage); + + let timer: ReturnType | undefined; + let onAbort: (() => void) | undefined; + const boundary = new Promise((_resolve, reject) => { + timer = setTimeout(() => reject(new BoundedWaitError("timeout", timeoutMessage)), remaining); + if (signal) { + onAbort = () => reject(new BoundedWaitError("aborted", "upload transaction aborted")); + signal.addEventListener("abort", onAbort, { once: true }); + } + }); + try { + return await Promise.race([promise, boundary]); + } finally { + if (timer) clearTimeout(timer); + if (signal && onAbort) signal.removeEventListener("abort", onAbort); + } +} + +function runtimeError(reply: RuntimeReply, fallback: string): Error | null { + if (!reply.exceptionDetails) return null; + return new Error( + reply.exceptionDetails.exception?.description ?? reply.exceptionDetails.text ?? fallback, + ); +} + +function transferFailure( + code: RpcError["code"], + reason: "file_input_probe_failed" | "file_input_not_activated" | "set_file_input_failed", + message: string, + effectState: TransferEffectState, + phase: UploadPhase, +): RpcError { + return transferError(code, reason, message, { effectState, phase }); +} + +function enrichFailure( + error: RpcError, + effectState: TransferEffectState, + phase: UploadPhase, +): RpcError { + return { + ...error, + data: { ...error.data, effect_state: effectState, phase }, + }; +} + +function sameTarget( + source: { tabId?: number; sessionId?: string }, + target: { tabId: number; sessionId?: string }, +): boolean { + return source.tabId === target.tabId && source.sessionId === target.sessionId; +} + +async function delay(ms: number, signal?: AbortSignal): Promise { + await waitBounded( + new Promise((resolve) => setTimeout(resolve, ms)), + Date.now() + ms + 1, + signal, + "upload activation probe timed out", + ); +} + +async function callOnTrigger( + options: FileInputTransactionOptions, + objectId: string, + functionDeclaration: string, + args: unknown[], + returnByValue: boolean, +): Promise { + return sendToCdpTarget(options.cdp, options.actionTarget.cdpTarget, "Runtime.callFunctionOn", { + objectId, + functionDeclaration, + arguments: args.map((value) => ({ value })), + returnByValue, + awaitPromise: false, + }); +} + +export async function uploadThroughActivatedFileInput( + options: FileInputTransactionOptions, +): Promise { + const deadline = Date.now() + options.timeoutMs; + const target = options.actionTarget.cdpTarget; + const objectGroup = `bsk-upload-${crypto.randomUUID()}`; + const stateKey = `__bskUpload_${crypto.randomUUID().replaceAll("-", "")}`; + const chooserEvents: ChooserEvent[] = []; + let interceptionArmed = false; + let probeArmed = false; + let triggerObjectId: string | undefined; + let outcome: FileInputTransactionResult | RpcError = transferFailure( + "protocol_error", + "file_input_probe_failed", + "upload transaction ended without an outcome", + "none", + "cleanup", + ); + + const chooserSubscription = options.cdp.onEvent?.((source, method, raw) => { + if (method !== "Page.fileChooserOpened" || !sameTarget(source, target)) return; + const event = raw as ChooserEvent; + chooserEvents.push(event); + }); + + try { + try { + await waitBounded( + sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", { + enabled: true, + }), + deadline, + options.signal, + "arming native file chooser interception timed out", + ); + interceptionArmed = true; + } catch (err) { + outcome = transferFailure( + err instanceof BoundedWaitError ? "timeout" : "cdp_failed", + "file_input_probe_failed", + err instanceof Error ? err.message : String(err), + "none", + "arm_interception", + ); + return outcome; + } + + try { + const resolved = await waitBounded( + sendToCdpTarget<{ object?: { objectId?: string } }>( + options.cdp, + target, + "DOM.resolveNode", + { + backendNodeId: options.actionTarget.backendNodeId, + objectGroup, + }, + ), + deadline, + options.signal, + "resolving upload trigger timed out", + ); + triggerObjectId = resolved.object?.objectId; + if (!triggerObjectId) throw new Error("DOM.resolveNode returned no trigger objectId"); + + const armed = await waitBounded( + callOnTrigger( + options, + triggerObjectId, + `function(key) { + const doc = this.ownerDocument; + const owner = doc.defaultView; + if (!owner) return false; + const state = { inputs: [], listener: null }; + Object.defineProperty(owner, key, { value: state, configurable: true }); + state.listener = event => { + const path = typeof event.composedPath === "function" ? event.composedPath() : []; + const candidate = path[0] || event.target; + if (candidate && candidate.nodeType === 1 && + candidate.localName === "input" && candidate.type === "file") { + if (!state.inputs.includes(candidate)) state.inputs.push(candidate); + event.preventDefault(); + } + }; + doc.addEventListener("click", state.listener, true); + return true; + }`, + [stateKey], + true, + ), + deadline, + options.signal, + "arming frame-scoped file input probe timed out", + ); + const armError = runtimeError(armed, "failed to arm file input probe"); + if (armError || armed.result?.value !== true) { + throw armError ?? new Error("file input probe did not arm"); + } + probeArmed = true; + } catch (err) { + outcome = transferFailure( + err instanceof BoundedWaitError ? "timeout" : "cdp_failed", + "file_input_probe_failed", + err instanceof Error ? err.message : String(err), + "none", + "arm_input_probe", + ); + return outcome; + } + + let click: ClickResult | RpcError; + try { + click = await waitBounded( + options.trigger(), + deadline, + options.signal, + "upload trigger timed out", + ); + } catch (err) { + outcome = transferFailure( + err instanceof BoundedWaitError ? "timeout" : "cdp_failed", + "file_input_probe_failed", + err instanceof Error ? err.message : String(err), + "none", + "trigger", + ); + return outcome; + } + if (isRpcError(click)) { + outcome = enrichFailure(click, "none", "trigger"); + return outcome; + } + + try { + const activationDeadline = Math.min(deadline, Date.now() + ACTIVATION_GRACE_MS); + let summary: { count: number; multiple: boolean } = { count: 0, multiple: false }; + while (Date.now() < activationDeadline) { + if (chooserEvents.length > 0) break; + const reply = await callOnTrigger( + options, + triggerObjectId, + `function(key) { + const state = this.ownerDocument.defaultView?.[key]; + return state + ? { count: state.inputs.length, multiple: state.inputs[0]?.multiple === true } + : { count: 0, multiple: false }; + }`, + [stateKey], + true, + ); + const summaryError = runtimeError(reply, "failed to inspect activated file input"); + if (summaryError) throw summaryError; + const value = reply.result?.value as { count?: unknown; multiple?: unknown } | undefined; + summary = { + count: typeof value?.count === "number" ? value.count : 0, + multiple: value?.multiple === true, + }; + if (summary.count > 0) break; + await delay(Math.min(PROBE_INTERVAL_MS, remainingMs(activationDeadline)), options.signal); + } + + if (chooserEvents.length > 1) { + outcome = transferFailure( + "unsupported", + "file_input_not_activated", + "upload trigger activated more than one file chooser", + "none", + "resolve_input", + ); + return outcome; + } + + const chooser = chooserEvents[0]; + let backendNodeId: number | undefined; + let multiple = summary.multiple; + if (chooser) { + if (options.actionTarget.frameId && chooser.frameId !== options.actionTarget.frameId) { + outcome = transferFailure( + "unsupported", + "file_input_not_activated", + "upload trigger activated a file chooser in a different frame", + "none", + "resolve_input", + ); + return outcome; + } + if (typeof chooser.backendNodeId !== "number") { + outcome = transferFailure( + "unsupported", + "file_input_not_activated", + "upload trigger invoked a non-input file picker", + "none", + "resolve_input", + ); + return outcome; + } + backendNodeId = chooser.backendNodeId; + multiple = chooser.mode === "selectMultiple"; + } else { + if (summary.count !== 1) { + outcome = transferFailure( + "unsupported", + "file_input_not_activated", + summary.count === 0 + ? "upload trigger did not activate an input[type=file]" + : "upload trigger activated more than one input[type=file]", + "none", + "resolve_input", + ); + return outcome; + } + const input = await callOnTrigger( + options, + triggerObjectId, + `function(key) { return this.ownerDocument.defaultView?.[key]?.inputs[0]; }`, + [stateKey], + false, + ); + const inputError = runtimeError(input, "failed to resolve activated file input object"); + if (inputError) throw inputError; + if (!input.result?.objectId) throw new Error("activated file input returned no objectId"); + const described = await sendToCdpTarget<{ node?: { backendNodeId?: number } }>( + options.cdp, + target, + "DOM.describeNode", + { objectId: input.result.objectId }, + ); + backendNodeId = described.node?.backendNodeId; + if (typeof backendNodeId !== "number") { + throw new Error("DOM.describeNode returned no file input backendNodeId"); + } + } + + if (!multiple && options.files.length !== 1) { + outcome = enrichFailure( + { code: "invalid_params", message: "file input accepts exactly one file" }, + "none", + "resolve_input", + ); + return outcome; + } + + try { + await waitBounded( + sendToCdpTarget(options.cdp, target, "DOM.setFileInputFiles", { + files: options.files, + backendNodeId, + }), + deadline, + options.signal, + "setting file input files timed out", + ); + } catch (err) { + outcome = transferFailure( + err instanceof BoundedWaitError ? "timeout" : "cdp_failed", + "set_file_input_failed", + err instanceof Error ? err.message : String(err), + "unknown", + "set_files", + ); + return outcome; + } + outcome = { click, multiple }; + return outcome; + } catch (err) { + outcome = transferFailure( + err instanceof BoundedWaitError ? "timeout" : "cdp_failed", + "file_input_probe_failed", + err instanceof Error ? err.message : String(err), + "none", + "resolve_input", + ); + return outcome; + } + } finally { + chooserSubscription?.dispose(); + let cleanupFailed = false; + if (probeArmed && triggerObjectId) { + try { + await waitBounded( + callOnTrigger( + options, + triggerObjectId, + `function(key) { + const owner = this.ownerDocument.defaultView; + const state = owner?.[key]; + if (state?.listener) this.ownerDocument.removeEventListener("click", state.listener, true); + if (owner) delete owner[key]; + }`, + [stateKey], + true, + ), + Date.now() + CLEANUP_TIMEOUT_MS, + undefined, + "cleaning file input probe timed out", + ); + } catch { + cleanupFailed = true; + } + } + if (interceptionArmed) { + try { + await waitBounded( + sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", { + enabled: false, + cancel: true, + }), + Date.now() + CLEANUP_TIMEOUT_MS, + undefined, + "disabling file chooser interception timed out", + ); + } catch { + cleanupFailed = true; + } + } + try { + await waitBounded( + sendToCdpTarget(options.cdp, target, "Runtime.releaseObjectGroup", { objectGroup }), + Date.now() + CLEANUP_TIMEOUT_MS, + undefined, + "releasing upload object group timed out", + ); + } catch { + cleanupFailed = true; + } + + const effect = isRpcError(outcome) + ? (outcome.data?.effect_state as TransferEffectState | undefined) + : "committed"; + if (effect === "unknown" || cleanupFailed) { + await options.cdp.detach?.(target.tabId); + } + if (cleanupFailed && isRpcError(outcome)) { + outcome.data = { ...outcome.data, cleanup_state: "failed" }; + } + } +} diff --git a/apps/extension/src/tools/interaction.ts b/apps/extension/src/tools/interaction.ts index 182cabdf..54646ed7 100644 --- a/apps/extension/src/tools/interaction.ts +++ b/apps/extension/src/tools/interaction.ts @@ -40,6 +40,7 @@ import { enforceAgentWindow, isRpcError, lookupSession, + type ResolvedTargetTab, resolveTargetTab, } from "./shared"; import { resolveSnapshotRef } from "./snapshot-ref"; @@ -56,6 +57,15 @@ export interface InteractionDeps { keepOverlayBypassAfterHover?: boolean; } +export interface ResolvedActionTarget { + tab: ResolvedTargetTab; + backendNodeId: number; + cdpTarget: CdpTarget; + frameId?: string; + usedRef?: string; + usedSelector?: string; +} + const DEFAULT_TIMEOUT_MS = 30_000; const DEFAULT_HOVER_SETTLE_MS = 200; @@ -128,7 +138,7 @@ async function wait(ms: number, signal?: AbortSignal): Promise { * `RpcError` if the caller supplied neither (or both), or if neither * lookup matched. */ -async function resolveBackendNode( +export async function resolveBackendNode( cdp: CdpRunner, ctx: SessionContext, target: { tabId: number }, @@ -215,6 +225,17 @@ async function resolveBackendNode( } } +export async function resolveActionTarget( + cdp: CdpRunner, + ctx: SessionContext, + target: ResolvedTargetTab, + params: { ref?: string; selector?: string }, + toolName: string, +): Promise { + const node = await resolveBackendNode(cdp, ctx, target, params, toolName); + return isRpcError(node) ? node : { tab: target, ...node }; +} + // --------------------------------------------------------------------------- // tool.click // --------------------------------------------------------------------------- @@ -233,10 +254,19 @@ export async function handleClick( if (isRpcError(target)) return target; const denied = enforceAgentWindow(ctx, target, "click"); if (denied) return denied; - const dialogCursor = markDialogCursor(deps.cdp, target.tabId); + const resolved = await resolveActionTarget(deps.cdp, ctx, target, params, "click"); + if (isRpcError(resolved)) return resolved; + return clickResolvedTarget(ctx, resolved, params, deps); +} - const node = await resolveBackendNode(deps.cdp, ctx, target, params, "click"); - if (isRpcError(node)) return node; +export async function clickResolvedTarget( + ctx: SessionContext, + resolved: ResolvedActionTarget, + params: Pick, + deps: InteractionDeps, +): Promise { + const { tab: target } = resolved; + const dialogCursor = markDialogCursor(deps.cdp, target.tabId); if (throwIfAborted(deps.signal)) { return { code: "cancelled", message: "click aborted" }; @@ -247,9 +277,9 @@ export async function handleClick( deps.cdp, target.tabId, { - target: node.cdpTarget, - backendNodeId: node.backendNodeId, - ...(node.frameId ? { frameId: node.frameId } : {}), + target: resolved.cdpTarget, + backendNodeId: resolved.backendNodeId, + ...(resolved.frameId ? { frameId: resolved.frameId } : {}), }, { scrollIntoView: true }, ); @@ -337,8 +367,8 @@ export async function handleClick( return attachDialogs(deps.cdp, target.tabId, dialogCursor, { tab_id: target.tabId, - used_ref: node.usedRef, - used_selector: node.usedSelector, + used_ref: resolved.usedRef, + used_selector: resolved.usedSelector, x: centre.x, y: centre.y, }); diff --git a/apps/extension/src/tools/shared.ts b/apps/extension/src/tools/shared.ts index 3b2cb79d..9483abfc 100644 --- a/apps/extension/src/tools/shared.ts +++ b/apps/extension/src/tools/shared.ts @@ -53,6 +53,7 @@ export type { DialogCursor }; export interface CdpRunner { send(tabId: number, method: string, params?: object): Promise; sendToTarget?(target: CdpTarget, method: string, params?: object): Promise; + detach?(tabId: number): Promise; getFrameGraph?(tabId: number): Promise; ensureAttachedToUrl?(tabId: number, expectedUrl: string | undefined): Promise; trackSessionTab?(sessionId: string, tabId: number): void; diff --git a/apps/extension/src/tools/upload.ts b/apps/extension/src/tools/upload.ts new file mode 100644 index 00000000..05a43ea8 --- /dev/null +++ b/apps/extension/src/tools/upload.ts @@ -0,0 +1,59 @@ +// Upload orchestration: validate the session-scoped request, resolve its click +// target, then delegate the browser protocol transaction to the file-input +// transaction module. + +import type { SessionManager } from "@/session-manager/manager"; +import type { RpcError, UploadParams, UploadResult } from "@/transport/types"; +import { uploadThroughActivatedFileInput } from "./file-input-transaction"; +import { clickResolvedTarget, type InteractionDeps, resolveActionTarget } from "./interaction"; +import { + type CdpRunner, + enforceAgentWindow, + isRpcError, + lookupSession, + resolveTargetTab, +} from "./shared"; + +const DEFAULT_TIMEOUT_MS = 120_000; + +export interface UploadDeps extends InteractionDeps { + cdp: CdpRunner; +} + +export async function handleUpload( + manager: SessionManager, + params: UploadParams, + deps: UploadDeps, +): Promise { + const ctx = lookupSession(manager, params, "upload"); + if (isRpcError(ctx)) return ctx; + const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi); + if (isRpcError(target)) return target; + const denied = enforceAgentWindow(ctx, target, "upload"); + if (denied) return denied; + if ( + params.files.length === 0 || + params.files.length > 20 || + params.files.some((file) => !file.staged_path) + ) { + return { code: "invalid_params", message: "upload requires daemon-staged files" }; + } + const address = await resolveActionTarget(deps.cdp, ctx, target, params, "upload"); + if (isRpcError(address)) return address; + const timeoutMs = params.timeout_ms ?? DEFAULT_TIMEOUT_MS; + const transaction = await uploadThroughActivatedFileInput({ + cdp: deps.cdp, + actionTarget: address, + files: params.files.map((file) => file.staged_path as string), + timeoutMs, + signal: deps.signal, + trigger: () => clickResolvedTarget(ctx, address, {}, deps), + }); + if (isRpcError(transaction)) return transaction; + return { + tab_id: target.tabId, + used_ref: transaction.click.used_ref, + used_selector: transaction.click.used_selector, + file_names: params.files.map((file) => file.name), + }; +} diff --git a/apps/extension/src/transport/types.ts b/apps/extension/src/transport/types.ts index dc60fe2e..7143cb15 100644 --- a/apps/extension/src/transport/types.ts +++ b/apps/extension/src/transport/types.ts @@ -33,10 +33,22 @@ export type RpcErrorReason = | "restricted_tab_url" | "borrow_conflict" | "screenshot_capture_failed" + | "file_input_probe_failed" + | "file_input_not_activated" + | "set_file_input_failed" + | "download_capture_failed" + | "transfer_outcome_unknown" + | "transfer_timeout" | "cleanup_failed"; +export type TransferEffectState = "none" | "committed" | "unknown"; +export type TransferCleanupState = "complete" | "failed"; + export interface RpcErrorData { reason?: RpcErrorReason; + effect_state?: TransferEffectState; + phase?: string; + cleanup_state?: TransferCleanupState; [key: string]: unknown; } @@ -500,6 +512,50 @@ export interface SelectResult { dialogs?: JavaScriptDialogInfo[]; } +export interface UploadFile { + transfer_id: string; + name: string; + staged_path?: string; +} + +export interface UploadParams { + session_id: string; + ref?: string; + selector?: string; + tab_id?: number; + files: UploadFile[]; + timeout_ms?: number; +} + +export interface UploadResult { + tab_id: number; + used_ref?: string; + used_selector?: string; + file_names: string[]; +} + +export interface DownloadParams { + session_id: string; + ref?: string; + selector?: string; + tab_id?: number; + timeout_ms?: number; + browser_relative_dir?: string; + max_byte_size?: number; +} + +export interface DownloadResult { + tab_id: number; + used_ref?: string; + used_selector?: string; + suggested_filename: string; + byte_size: number; + mime?: string; + danger?: string; + browser_path?: string; + transfer_id?: string; +} + // -------------------------------------------------------------------------- // M9 tool payloads — evaluate / wait_for_navigation / wait_ms // -------------------------------------------------------------------------- diff --git a/apps/extension/wxt.config.ts b/apps/extension/wxt.config.ts index 6e272d95..ff4fc532 100644 --- a/apps/extension/wxt.config.ts +++ b/apps/extension/wxt.config.ts @@ -30,6 +30,7 @@ export default defineConfig({ permissions: [ "alarms", "debugger", + "downloads", "idle", "notifications", "tabs", diff --git a/crates/bsk-cli/Cargo.toml b/crates/bsk-cli/Cargo.toml index afc980f4..e24ef887 100644 --- a/crates/bsk-cli/Cargo.toml +++ b/crates/bsk-cli/Cargo.toml @@ -53,6 +53,7 @@ flate2 = { workspace = true } tar = { workspace = true } zip = { workspace = true } sha2 = { workspace = true } +base64 = { workspace = true } [target.'cfg(unix)'.dependencies] nix = { workspace = true } @@ -63,6 +64,7 @@ windows-sys = { version = "0.59", features = [ "Win32_Foundation", "Win32_System_Threading", "Win32_Security", + "Win32_Storage_FileSystem", ] } [dev-dependencies] diff --git a/crates/bsk-cli/skill/SKILL.md b/crates/bsk-cli/skill/SKILL.md index d6dd1e2e..8af53834 100644 --- a/crates/bsk-cli/skill/SKILL.md +++ b/crates/bsk-cli/skill/SKILL.md @@ -206,6 +206,29 @@ Both capture from the moment the tab is attached and read a bounded per-tab buff | `bsk select --value ` | Set ``; the page may use a non-input picker such as `window.showOpenFilePicker()`. Do not retry blindly. If human help is available, call `request-help` and tell the user the exact original local path to choose. The staged daemon path is internal and must not be shown to the user. +- `reason=file_input_probe_failed` means BrowserSkill could not safely establish the browser-side upload transaction. Do not repeat the same action; use `request-help` when available. +- `reason=set_file_input_failed` means BrowserSkill found the activated file input but Chrome rejected the staged path or assignment. Check the extension's file-URL access permission; otherwise use `request-help`. +- `reason=download_capture_failed` means BrowserSkill could not attribute exactly one completed download to the requested target. Do not retry blindly or accept an unrelated browser download; use `request-help` when available. +- `effect_state=none` means BrowserSkill confirmed that no file-transfer effect was committed. Follow the accompanying reason; a corrected target or explicit human fallback may be attempted. +- `effect_state=unknown` means the browser may already have attached or created the file. Do not repeat the transfer. Observe the page if that can establish the result; otherwise stop and report the uncertainty. +- `effect_state=committed` means the browser-side effect occurred even if later completion or cleanup failed. Do not repeat it; continue only after verifying the resulting page/download state. + +If `request-help` returns `outcome="disabled"`, do not retry it. Stop gracefully and report the transfer mechanism that requires human intervention. + ### Scripting & timing | Command | Summary | diff --git a/crates/bsk-cli/src/cli/atomic_output.rs b/crates/bsk-cli/src/cli/atomic_output.rs new file mode 100644 index 00000000..5efacf25 --- /dev/null +++ b/crates/bsk-cli/src/cli/atomic_output.rs @@ -0,0 +1,75 @@ +//! Atomic visibility boundary for files received into a same-directory temp. + +use std::path::Path; + +pub fn commit(temp: &Path, out: &Path, overwrite: bool) -> std::io::Result<()> { + if !overwrite { + std::fs::hard_link(temp, out)?; + std::fs::remove_file(temp)?; + return Ok(()); + } + replace(temp, out) +} + +#[cfg(unix)] +fn replace(temp: &Path, out: &Path) -> std::io::Result<()> { + // POSIX rename replaces an existing non-directory destination atomically. + std::fs::rename(temp, out) +} + +#[cfg(windows)] +fn replace(temp: &Path, out: &Path) -> std::io::Result<()> { + use std::os::windows::ffi::OsStrExt; + use windows_sys::Win32::Storage::FileSystem::{ + MOVEFILE_REPLACE_EXISTING, MOVEFILE_WRITE_THROUGH, MoveFileExW, + }; + + let from: Vec = temp.as_os_str().encode_wide().chain(Some(0)).collect(); + let to: Vec = out.as_os_str().encode_wide().chain(Some(0)).collect(); + // SAFETY: both buffers are NUL-terminated and remain alive for the call. + let ok = unsafe { + MoveFileExW( + from.as_ptr(), + to.as_ptr(), + MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH, + ) + }; + if ok == 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(()) + } +} + +#[cfg(not(any(unix, windows)))] +fn replace(temp: &Path, out: &Path) -> std::io::Result<()> { + std::fs::rename(temp, out) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn no_replace_never_overwrites_existing_output() { + let dir = tempfile::tempdir().unwrap(); + let temp = dir.path().join("new.part"); + let out = dir.path().join("out.bin"); + std::fs::write(&temp, b"new").unwrap(); + std::fs::write(&out, b"old").unwrap(); + assert!(commit(&temp, &out, false).is_err()); + assert_eq!(std::fs::read(&out).unwrap(), b"old"); + } + + #[test] + fn overwrite_replaces_existing_output_without_predelete() { + let dir = tempfile::tempdir().unwrap(); + let temp = dir.path().join("new.part"); + let out = dir.path().join("out.bin"); + std::fs::write(&temp, b"new").unwrap(); + std::fs::write(&out, b"old").unwrap(); + commit(&temp, &out, true).unwrap(); + assert_eq!(std::fs::read(&out).unwrap(), b"new"); + assert!(!temp.exists()); + } +} diff --git a/crates/bsk-cli/src/cli/download.rs b/crates/bsk-cli/src/cli/download.rs new file mode 100644 index 00000000..5a97b28a --- /dev/null +++ b/crates/bsk-cli/src/cli/download.rs @@ -0,0 +1,147 @@ +//! `bsk download` — capture one browser download and atomically commit it. + +use std::fs::OpenOptions; +use std::io::Write; +use std::path::{Path, PathBuf}; +use std::time::Duration; + +use anyhow::Context; +use base64::Engine; +use bsk_protocol::Method; +use bsk_protocol::tools::{ + DownloadParams, DownloadResult, TransferChunkParams, TransferChunkResult, TransferIdParams, + TransferReleaseResult, +}; +use clap::Args; +use uuid::Uuid; + +use crate::cli::atomic_output; +use crate::cli::ensure_daemon::ensure_daemon; +use crate::cli::error::{CliError, Format}; +use crate::cli::interaction::split_target; +use crate::cli::navigate::parse_timeout_ms; + +#[derive(Debug, Clone, Args)] +pub struct DownloadArgs { + /// Snapshot ref (`@e3`) or CSS selector for the download trigger. + pub target: Option, + #[arg(long = "ref")] + pub ref_: Option, + #[arg(long = "selector")] + pub selector: Option, + #[arg(long)] + pub out: PathBuf, + #[arg(long)] + pub session: String, + #[arg(long = "tab-id")] + pub tab_id: Option, + #[arg(long, default_value = "2m", value_parser = parse_timeout_ms)] + pub timeout: u32, + #[arg(long)] + pub overwrite: bool, +} + +pub fn dispatch(args: DownloadArgs, format: Format) -> Result<(), CliError> { + if args.out.exists() && !args.overwrite { + return Err(CliError::Local(anyhow::anyhow!( + "output already exists (pass --overwrite to replace it): {}", + args.out.display() + ))); + } + let info = ensure_daemon().context("ensure daemon is running")?; + let (ref_, selector) = split_target(args.target, args.ref_, args.selector)?; + let params = DownloadParams { + session_id: args.session, + ref_, + selector, + tab_id: args.tab_id, + timeout_ms: Some(args.timeout), + browser_relative_dir: None, + max_byte_size: None, + }; + let reply: DownloadResult = crate::cli::business_rpc::call( + info.sock_path.clone(), + "download", + Method::ToolDownload, + Some(params), + ipc_timeout(args.timeout), + )?; + let transfer_id = reply.transfer_id.clone().ok_or_else(|| { + CliError::Local(anyhow::anyhow!("daemon returned no download transfer id")) + })?; + let write_result = write_transfer(&info.sock_path, &transfer_id, &args.out, args.overwrite); + let _: Result = crate::cli::business_rpc::call( + info.sock_path, + "transfer-release", + Method::TransferRelease, + Some(TransferIdParams { transfer_id }), + Duration::from_secs(5), + ); + write_result?; + match format { + Format::Json => { + let mut value = serde_json::to_value(&reply).unwrap(); + value["path"] = serde_json::json!(args.out.to_string_lossy()); + if let Some(obj) = value.as_object_mut() { + obj.remove("transfer_id"); + obj.remove("browser_path"); + } + println!("{}", serde_json::to_string_pretty(&value).unwrap()); + } + Format::Human => println!("{}", args.out.display()), + } + Ok(()) +} + +fn write_transfer(sock: &Path, id: &str, out: &Path, overwrite: bool) -> Result<(), CliError> { + let parent = out + .parent() + .filter(|p| !p.as_os_str().is_empty()) + .unwrap_or(Path::new(".")); + let temp = parent.join(format!(".bsk-download-{}.part", Uuid::new_v4().simple())); + let result = (|| { + let mut file = OpenOptions::new() + .write(true) + .create_new(true) + .open(&temp) + .with_context(|| format!("create temporary download {}", temp.display())) + .map_err(CliError::Local)?; + let mut offset = 0u64; + loop { + let chunk: TransferChunkResult = crate::cli::business_rpc::call( + sock.to_path_buf(), + "transfer-read", + Method::TransferRead, + Some(TransferChunkParams { + transfer_id: id.to_string(), + offset, + data_base64: String::new(), + }), + Duration::from_secs(30), + )?; + let encoded = chunk.data_base64.as_deref().unwrap_or(""); + let bytes = base64::engine::general_purpose::STANDARD + .decode(encoded) + .map_err(|e| CliError::Local(anyhow::anyhow!("decode download chunk: {e}")))?; + file.write_all(&bytes) + .map_err(|e| CliError::Local(e.into()))?; + offset = chunk.next_offset; + if chunk.eof { + break; + } + } + file.sync_all().map_err(|e| CliError::Local(e.into()))?; + drop(file); + atomic_output::commit(&temp, out, overwrite) + .with_context(|| format!("atomically commit download to {}", out.display())) + .map_err(CliError::Local) + })(); + if result.is_err() { + let _ = std::fs::remove_file(&temp); + } + result +} + +fn ipc_timeout(timeout_ms: u32) -> Duration { + Duration::from_millis(u64::from(timeout_ms) + 5_000) +} diff --git a/crates/bsk-cli/src/cli/interaction.rs b/crates/bsk-cli/src/cli/interaction.rs index a6fa8569..075f9ac4 100644 --- a/crates/bsk-cli/src/cli/interaction.rs +++ b/crates/bsk-cli/src/cli/interaction.rs @@ -71,7 +71,7 @@ pub(crate) fn parse_modifiers(input: &str) -> Result, String> { Ok(out) } -fn split_target( +pub(crate) fn split_target( positional: Option, explicit_ref: Option, explicit_selector: Option, diff --git a/crates/bsk-cli/src/cli/mod.rs b/crates/bsk-cli/src/cli/mod.rs index cba2d499..4776ebe1 100644 --- a/crates/bsk-cli/src/cli/mod.rs +++ b/crates/bsk-cli/src/cli/mod.rs @@ -2,6 +2,7 @@ use std::time::Duration; +mod atomic_output; pub mod browser_wait; pub mod browsers; pub mod business_rpc; @@ -9,6 +10,7 @@ pub mod console; pub mod daemon; pub mod dialogs; pub mod doctor; +pub mod download; pub mod emulate; pub mod ensure_daemon; pub mod error; @@ -31,6 +33,7 @@ pub mod snapshot; pub mod status; pub mod tab; pub mod update; +pub mod upload; pub mod waits; pub mod window; @@ -38,6 +41,7 @@ use clap::{Args, Parser, Subcommand}; use crate::cli::console::ConsoleArgs; use crate::cli::daemon::DaemonCmd; +use crate::cli::download::DownloadArgs; use crate::cli::emulate::EmulateArgs; use crate::cli::evaluate::EvaluateArgs; use crate::cli::get_html::GetHtmlArgs; @@ -53,6 +57,7 @@ use crate::cli::session::SessionCmd; use crate::cli::snapshot::SnapshotArgs; use crate::cli::tab::TabCmd; use crate::cli::update::UpdateArgs; +use crate::cli::upload::UploadArgs; use crate::cli::waits::{WaitForNavigationArgs, WaitMsArgs}; use crate::cli::window::WindowCmd; @@ -177,6 +182,12 @@ pub enum Command { /// Set `` option(s) by `value` (repeat `--value` for multi-select) | | `bsk press ` | Key/combo (`Enter`, `Ctrl+A`, …; optional `--ref` to focus first) | +### File transfer (require `--session`) + +| Command | Summary | +|---------|---------| +| `bsk upload --file ` | Click one upload trigger and attach an agent-readable local file (`--file` is repeatable) | +| `bsk download --out ` | Click one download trigger and copy the single completed file to an exact local path (`--overwrite` is opt-in) | + +The agent/harness decides whether a file transfer is appropriate and which local path belongs to the task. Treat upload as disclosure of that file to the current website, and download as accepting website-controlled bytes onto the local filesystem. Use only paths that are necessary for the user's bounded goal. + +BrowserSkill enforces the mechanical boundary: files are staged under a session-scoped opaque transfer, only daemon-minted capabilities reach the extension, upload/download still obey Agent Window tab checks, and transfers are chunk/size bounded. Upload intercepts the native chooser for one transaction, locates the input activated in the resolved target's document, and assigns only the staged file paths. Download uniquely correlates one exact-target browser intent with one Chrome download in either event order, routes it through a daemon-minted relative directory, and lets the daemon validate and import it. Upload staging remains available for a later form submission and is removed when the session ends. Downloads cannot overwrite an existing destination unless `--overwrite` is explicit. BrowserSkill does not inspect file content or decide whether its meaning is sensitive. + +Do not use `request-help` merely because a native file chooser or browser download is involved; try these commands first. For transfer failures, use the structured error instead of retrying blindly: + +- `reason=file_input_not_activated` means the requested click did not activate exactly one ``; the page may use a non-input picker such as `window.showOpenFilePicker()`. Do not retry blindly. If human help is available, call `request-help` and tell the user the exact original local path to choose. The staged daemon path is internal and must not be shown to the user. +- `reason=file_input_probe_failed` means BrowserSkill could not safely establish the browser-side upload transaction. Do not repeat the same action; use `request-help` when available. +- `reason=set_file_input_failed` means BrowserSkill found the activated file input but Chrome rejected the staged path or assignment. Check the extension's file-URL access permission; otherwise use `request-help`. +- `reason=download_capture_failed` means BrowserSkill could not attribute exactly one completed download to the requested target. Do not retry blindly or accept an unrelated browser download; use `request-help` when available. +- `effect_state=none` means BrowserSkill confirmed that no file-transfer effect was committed. Follow the accompanying reason; a corrected target or explicit human fallback may be attempted. +- `effect_state=unknown` means the browser may already have attached or created the file. Do not repeat the transfer. Observe the page if that can establish the result; otherwise stop and report the uncertainty. +- `effect_state=committed` means the browser-side effect occurred even if later completion or cleanup failed. Do not repeat it; continue only after verifying the resulting page/download state. + +If `request-help` returns `outcome="disabled"`, do not retry it. Stop gracefully and report the transfer mechanism that requires human intervention. + ### Scripting & timing | Command | Summary |