> ## Documentation Index
> Fetch the complete documentation index at: https://docs.superserve.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Computer use

> Give an agent a GUI desktop inside a sandbox, with screenshots, mouse, keyboard, and a live browser viewer.

A computer-use agent works a desktop the way a person does: it looks at the
screen, decides, then clicks or types. `sandbox.desktop` gives an agent that
desktop inside a sandbox: screenshots, mouse, keyboard, scroll, a resizable
display, and a live viewer you can open in a browser.

Every call is one request to the sandbox. A click is one round trip, a drag
is one request, a whole model turn can go in one request, and typing has no
per-character delay. A paused sandbox resumes on first use, as
with `commands` and `files`.

## Create a desktop sandbox

The sandbox must come from a desktop-enabled template. `superserve/desktop`
ships the xfce desktop, Chrome, and a browser-based viewer, and runs the
session as the non-root `desktop` user (also the default user for
`commands.run`). The display starts at 1280x800.

<CodeGroup>
  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  import { Sandbox } from "@superserve/sdk"

  const sandbox = await Sandbox.create({
    name: "desktop",
    fromTemplate: "superserve/desktop",
  })
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  from superserve import Sandbox

  sandbox = Sandbox.create(name="desktop", from_template="superserve/desktop")
  ```
</CodeGroup>

## The agent loop

Your code owns the model call and the conversation. The sandbox runs the
actions. Each turn is a screenshot in, a list of actions out:

<CodeGroup>
  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  await sandbox.desktop.resize(1024, 768) // match the model's screen size

  let shot = await sandbox.desktop.screenshot()
  while (true) {
    const actions = await nextActions(shot) // your model call
    if (actions.length === 0) break
    const step = await sandbox.desktop.step(actions, { settleMs: 300 })
    if (step.actionError) throw new Error(step.actionError) // stopped at step.executed
    if (!step.screenshot) throw new Error(step.screenshotError)
    shot = step.screenshot
  }
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  sandbox.desktop.resize(1024, 768)  # match the model's screen size

  shot = sandbox.desktop.screenshot()
  while True:
      actions = next_actions(shot)  # your model call
      if not actions:
          break
      step = sandbox.desktop.step(actions, settle_ms=300)
      if step.action_error:
          raise RuntimeError(step.action_error)  # stopped at step.executed
      if step.screenshot is None:
          raise RuntimeError(step.screenshot_error)
      shot = step.screenshot
  ```
</CodeGroup>

`step` runs the model's whole turn and captures the frame after it in one
request, so a turn is a single round trip. The sandbox waits `settleMs` /
`settle_ms` (up to 2000) before capturing, because input is delivered before
the application has repainted; how long it needs depends on the application.
`step` does not throw when the batch stops at a failing action or when the
capture fails, since the actions that ran have already landed: check
`actionError` / `action_error` (with `executed`, the index it stopped at) and
`screenshotError` / `screenshot_error`. Resizing to the model's native
resolution first keeps frames small and maps its coordinates 1:1 onto the
screen.

<Note>
  Everything the agent reads from the screen is untrusted input to the model:
  page content, file contents, text in dialogs. Keep credentials out of the
  desktop, restrict where the sandbox can connect with [network
  rules](/sandbox/networking), and have the model ask for approval before an
  action with an outside effect.
</Note>

## Watch it live

<CodeGroup>
  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  const url = await sandbox.desktop.getStreamUrl()
  const readOnly = await sandbox.desktop.getStreamUrl({ viewOnly: true })
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  url = sandbox.desktop.get_stream_url()
  read_only = sandbox.desktop.get_stream_url(view_only=True)
  ```
</CodeGroup>

Returns a browser URL for the viewer on port `6080`, publishing that port
under the sandbox's [preview policy](/sandbox/preview-urls). Under `public` the
URL is clean. Under `private` it carries a 60-second signed credential, as
`getSignedPreviewUrl` / `get_signed_preview_url` does; mint a fresh URL for
each viewer. `viewOnly` / `view_only` is a flag on that viewer only and does
not stop someone else from opening the same URL with control, so use a
`private` policy when the viewer must not be shared.

## Screenshots

<CodeGroup>
  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  import fs from "node:fs/promises"

  const shot = await sandbox.desktop.screenshot()
  await fs.writeFile("screen.png", shot.data) // PNG bytes; shot.width, shot.height
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  from pathlib import Path

  shot = sandbox.desktop.screenshot()
  Path("screen.png").write_bytes(shot.data)  # PNG bytes; shot.width, shot.height
  ```
</CodeGroup>

The capture includes the cursor. Coordinates in every other call are pixels in
this image, with the origin at the top-left.

## Mouse

<CodeGroup>
  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  await sandbox.desktop.click(640, 400)
  await sandbox.desktop.click(640, 400, { button: "right" }) // or rightClick / middleClick
  await sandbox.desktop.doubleClick(640, 400)
  await sandbox.desktop.moveMouse(100, 100)
  await sandbox.desktop.drag([10, 10], [200, 200])
  await sandbox.desktop.scroll({ dy: 3 }) // positive scrolls down; dx for horizontal
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  sandbox.desktop.click(640, 400)
  sandbox.desktop.click(640, 400, button="right")  # or right_click / middle_click
  sandbox.desktop.double_click(640, 400)
  sandbox.desktop.move_mouse(100, 100)
  sandbox.desktop.drag((10, 10), (200, 200))
  sandbox.desktop.scroll(dy=3)  # positive scrolls down; dx for horizontal
  ```
</CodeGroup>

A click moves the pointer and clicks in one request. A drag runs as one atomic
batch (press, move, release), so no other input can land in the middle of it. Scroll moves the wheel by whole clicks and targets
whatever is under the pointer, so move first to scroll a specific element.

## Keyboard

<CodeGroup>
  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  await sandbox.desktop.write("hello world")
  await sandbox.desktop.press("enter")
  await sandbox.desktop.press("ctrl+shift+p") // or press(["ctrl", "shift", "p"])
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  sandbox.desktop.write("hello world")
  sandbox.desktop.press("enter")
  sandbox.desktop.press("ctrl+shift+p")  # or press(["ctrl", "shift", "p"])
  ```
</CodeGroup>

`write` types literal text with no per-character pacing, so long strings land
in well under a second. `press` sends a key or chord. Friendly names (`enter`,
`esc`, `tab`, `backspace`, `delete`, `up`, `down`, `pageup`, `ctrl`, `alt`,
`shift`, `cmd`) are accepted as written; any other key uses its X keysym name,
so `F5`, `Return`, and `KP_Enter` work as written.

## Batch actions

Models that emit several actions per turn can send them in one request:

<CodeGroup>
  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  await sandbox.desktop.actions([
    { type: "click", x: 640, y: 32 },
    { type: "write", text: "https://example.com" },
    { type: "press", key: "enter" },
  ])
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  sandbox.desktop.actions([
      {"type": "click", "x": 640, "y": 32},
      {"type": "write", "text": "https://example.com"},
      {"type": "press", "key": "enter"},
  ])
  ```
</CodeGroup>

The whole batch is validated before anything runs, then runs in order and
stops at the first failing action. `step` is the same batch followed by a
capture in the same request, for the agent loop above; `actions` is for input
you do not need to look at afterwards.

| TypeScript `type` | Python `type` | Fields |
| - | - | - |
| `click` | `click` | `x`, `y`, `button?` |
| `doubleClick` | `double_click` | `x`, `y` |
| `move` | `move` | `x`, `y` |
| `mouseDown` / `mouseUp` | `mouse_down` / `mouse_up` | `x`, `y`, `button?` |
| `press` | `press` | `key` (string or list) |
| `write` | `write` | `text` |
| `scroll` | `scroll` | `dx?`, `dy?` |

<Warning>
  Input is not idempotent. If a pointer, key, or batch call fails with a
  transport error, the input may already have been delivered. Take a screenshot
  and check the screen before repeating it.
</Warning>

## Resize the display

<CodeGroup>
  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  await sandbox.desktop.resize(1920, 1080)
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"vitesse-dark"}}
  sandbox.desktop.resize(1920, 1080)
  ```
</CodeGroup>

Takes effect live, with no restart: open windows re-lay out. Width must be a
multiple of 8 from 320 to 8192; height from 200 to 8192.

## Pause and resume

The desktop survives a pause. Open windows, running applications, and the
screen come back exactly as they were, and the next `desktop` call on a paused
sandbox resumes it first.

## MCP

The MCP server exposes the same surface as the `sandbox_computer` tool, one
action per call (`screenshot`, `left_click`, `type`, `key`, `scroll`,
`left_click_drag`, …) in the vocabulary computer-use models are trained on.
Input actions return a fresh screenshot of the result by default. See
[MCP Sandboxes](/integrations/mcp#computer-use).

## Related

* [SDK reference: Sandbox](/sdk-reference/sandbox#desktop-methods)
* [Preview URLs](/sandbox/preview-urls) — access policy for the viewer port
* [Templates](/templates/overview) — the `superserve/desktop` template


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.