Computer use

A computer-use agent works a desktop the way a person does: it looks at the screen, decides, then clicks or types. sandbox.desktop gives an agent that desktop inside a sandbox: screenshots, mouse, keyboard, scroll, a resizable display, and a live viewer you can open in a browser. Every call is one request to the sandbox. A click is one round trip, a drag is one request, a whole model turn can go in one request, and typing has no per-character delay. A paused sandbox resumes on first use, as with commands and files.

Create a desktop sandbox

The sandbox must come from a desktop-enabled template. superserve/desktop ships the xfce desktop, Chrome, and a browser-based viewer, and runs the session as the non-root desktop user (also the default user for commands.run). The display starts at 1280x800.

import { Sandbox } from "@superserve/sdk"

const sandbox = await Sandbox.create({
  name: "desktop",
  fromTemplate: "superserve/desktop",
})

from superserve import Sandbox

sandbox = Sandbox.create(name="desktop", from_template="superserve/desktop")

The agent loop

Your code owns the model call and the conversation. The sandbox runs the actions. Each turn is a screenshot in, a list of actions out:

await sandbox.desktop.resize(1024, 768) // match the model's screen size

let shot = await sandbox.desktop.screenshot()
while (true) {
  const actions = await nextActions(shot) // your model call
  if (actions.length === 0) break
  const step = await sandbox.desktop.step(actions, { settleMs: 300 })
  if (step.actionError) throw new Error(step.actionError) // stopped at step.executed
  if (!step.screenshot) throw new Error(step.screenshotError)
  shot = step.screenshot
}

sandbox.desktop.resize(1024, 768)  # match the model's screen size

shot = sandbox.desktop.screenshot()
while True:
    actions = next_actions(shot)  # your model call
    if not actions:
        break
    step = sandbox.desktop.step(actions, settle_ms=300)
    if step.action_error:
        raise RuntimeError(step.action_error)  # stopped at step.executed
    if step.screenshot is None:
        raise RuntimeError(step.screenshot_error)
    shot = step.screenshot

step runs the model’s whole turn and captures the frame after it in one request, so a turn is a single round trip. The sandbox waits settleMs / settle_ms (up to 2000) before capturing, because input is delivered before the application has repainted; how long it needs depends on the application. step does not throw when the batch stops at a failing action or when the capture fails, since the actions that ran have already landed: check actionError / action_error (with executed, the index it stopped at) and screenshotError / screenshot_error. Resizing to the model’s native resolution first keeps frames small and maps its coordinates 1:1 onto the screen.

Everything the agent reads from the screen is untrusted input to the model: page content, file contents, text in dialogs. Keep credentials out of the desktop, restrict where the sandbox can connect with network rules, and have the model ask for approval before an action with an outside effect.

Watch it live

const url = await sandbox.desktop.getStreamUrl()
const readOnly = await sandbox.desktop.getStreamUrl({ viewOnly: true })

url = sandbox.desktop.get_stream_url()
read_only = sandbox.desktop.get_stream_url(view_only=True)

Returns a browser URL for the viewer on port 6080, publishing that port under the sandbox’s preview policy. Under public the URL is clean. Under private it carries a 60-second signed credential, as getSignedPreviewUrl / get_signed_preview_url does; mint a fresh URL for each viewer. viewOnly / view_only is a flag on that viewer only and does not stop someone else from opening the same URL with control, so use a private policy when the viewer must not be shared.

Screenshots

import fs from "node:fs/promises"

const shot = await sandbox.desktop.screenshot()
await fs.writeFile("screen.png", shot.data) // PNG bytes; shot.width, shot.height

from pathlib import Path

shot = sandbox.desktop.screenshot()
Path("screen.png").write_bytes(shot.data)  # PNG bytes; shot.width, shot.height

The capture includes the cursor. Coordinates in every other call are pixels in this image, with the origin at the top-left.

Mouse

await sandbox.desktop.click(640, 400)
await sandbox.desktop.click(640, 400, { button: "right" }) // or rightClick / middleClick
await sandbox.desktop.doubleClick(640, 400)
await sandbox.desktop.moveMouse(100, 100)
await sandbox.desktop.drag([10, 10], [200, 200])
await sandbox.desktop.scroll({ dy: 3 }) // positive scrolls down; dx for horizontal

sandbox.desktop.click(640, 400)
sandbox.desktop.click(640, 400, button="right")  # or right_click / middle_click
sandbox.desktop.double_click(640, 400)
sandbox.desktop.move_mouse(100, 100)
sandbox.desktop.drag((10, 10), (200, 200))
sandbox.desktop.scroll(dy=3)  # positive scrolls down; dx for horizontal

A click moves the pointer and clicks in one request. A drag runs as one atomic batch (press, move, release), so no other input can land in the middle of it. Scroll moves the wheel by whole clicks and targets whatever is under the pointer, so move first to scroll a specific element.

Keyboard

await sandbox.desktop.write("hello world")
await sandbox.desktop.press("enter")
await sandbox.desktop.press("ctrl+shift+p") // or press(["ctrl", "shift", "p"])

sandbox.desktop.write("hello world")
sandbox.desktop.press("enter")
sandbox.desktop.press("ctrl+shift+p")  # or press(["ctrl", "shift", "p"])

write types literal text with no per-character pacing, so long strings land in well under a second. press sends a key or chord. Friendly names (enter, esc, tab, backspace, delete, up, down, pageup, ctrl, alt, shift, cmd) are accepted as written; any other key uses its X keysym name, so F5, Return, and KP_Enter work as written.

Batch actions

Models that emit several actions per turn can send them in one request:

await sandbox.desktop.actions([
  { type: "click", x: 640, y: 32 },
  { type: "write", text: "https://example.com" },
  { type: "press", key: "enter" },
])

sandbox.desktop.actions([
    {"type": "click", "x": 640, "y": 32},
    {"type": "write", "text": "https://example.com"},
    {"type": "press", "key": "enter"},
])

The whole batch is validated before anything runs, then runs in order and stops at the first failing action. step is the same batch followed by a capture in the same request, for the agent loop above; actions is for input you do not need to look at afterwards.

TypeScript `type`	Python `type`	Fields
`click`	`click`	`x`, `y`, `button?`
`doubleClick`	`double_click`	`x`, `y`
`move`	`move`	`x`, `y`
`mouseDown` / `mouseUp`	`mouse_down` / `mouse_up`	`x`, `y`, `button?`
`press`	`press`	`key` (string or list)
`write`	`write`	`text`
`scroll`	`scroll`	`dx?`, `dy?`

Input is not idempotent. If a pointer, key, or batch call fails with a transport error, the input may already have been delivered. Take a screenshot and check the screen before repeating it.

Resize the display

await sandbox.desktop.resize(1920, 1080)

sandbox.desktop.resize(1920, 1080)

Takes effect live, with no restart: open windows re-lay out. Width must be a multiple of 8 from 320 to 8192; height from 200 to 8192.

Pause and resume

The desktop survives a pause. Open windows, running applications, and the screen come back exactly as they were, and the next desktop call on a paused sandbox resumes it first.

MCP

The MCP server exposes the same surface as the sandbox_computer tool, one action per call (screenshot, left_click, type, key, scroll, left_click_drag, …) in the vocabulary computer-use models are trained on. Input actions return a fresh screenshot of the result by default. See MCP Sandboxes.

SDK reference: Sandbox
Preview URLs — access policy for the viewer port
Templates — the superserve/desktop template

Get Started

Sandbox

Secrets & env vars

Templates

Commands

Filesystem

Storage

Errors

Create a desktop sandbox

The agent loop

Watch it live

Screenshots

Mouse

Keyboard

Batch actions

Resize the display

Pause and resume

MCP

​Create a desktop sandbox

​The agent loop

​Watch it live

​Screenshots

​Mouse

​Keyboard

​Batch actions

​Resize the display

​Pause and resume

​MCP

​Related

Create a desktop sandbox

The agent loop

Watch it live

Screenshots

Mouse

Keyboard

Batch actions

Resize the display

Pause and resume

MCP

Related