sandbox.desktop gives an agent that
desktop inside a sandbox: screenshots, mouse, keyboard, scroll, a resizable
display, and a live viewer you can open in a browser.
Every call is one request to the sandbox. A click is one round trip, a drag
is one request, a whole model turn can go in one request, and typing has no
per-character delay. A paused sandbox resumes on first use, as
with commands and files.
Create a desktop sandbox
The sandbox must come from a desktop-enabled template.superserve/desktop
includes the xfce desktop, Chrome, and a browser-based viewer, and runs the
session as the non-root desktop user (also the default user for
commands.run). The display starts at 1280x800.
The agent loop
Your code owns the model call and the conversation. The sandbox runs the actions. Each turn is a screenshot in, a list of actions out. In the loop below,nextActions / next_actions stands for your own model call: it sends
the frame to the model and returns the model’s actions for the turn.
step runs the model’s whole turn and captures the frame after it in one
request, so a turn is a single round trip. Input is delivered before the
application has repainted, so with waitForChange / wait_for_change the
sandbox captures the first frame whose pixels differ from the frame before
the batch, waiting at most settleMs / settle_ms (default 1000, up to
2000); the result’s changed says whether a repaint arrived. changed is
meaningful only with waitForChange: without it, changed is undefined in
TypeScript and False in Python. A changed frame
proves the display repainted, not that the application finished, so check
the screenshot for what you expect. Without waitForChange the sandbox
sleeps settleMs (default 0: no sleep) and then captures. With quietMs /
quiet_ms the capture also waits until the display has been still for that long, so a batch that
paints in steps (a focus ring, then the typed text) comes back settled rather
than mid-way. quietMs requires waitForChange and can’t exceed settleMs,
so raise settleMs for a quiet window above 1000; otherwise step throws a
ValidationError. Frames are PNG unless format is
"jpeg": several times smaller and faster to encode, the right trade when a
model reads the frame. The result’s timings (queueMs, actionsMs,
settleMs, captureMs) say where the sandbox spent the turn, so a slow loop
can be attributed to input, the repaint wait or the capture rather than to the
network.
step does not throw when the batch stops at a failing action or when the
capture fails, since the actions that ran have already landed: check
actionError / action_error (with executed, the index it stopped at) and
screenshotError / screenshot_error. Resizing to the model’s native
resolution first keeps frames small and maps its coordinates 1:1 onto the
screen.
A batch that fails validation is different: step and actions throw, and
no action in the batch runs. The SDK rejects a negative or non-integer
coordinate and an empty key name before it sends the request, with Error in
TypeScript and ValueError in Python. Python also rejects an unknown action
type this way. The sandbox rejects a coordinate above 65535, and an unknown
action type sent from TypeScript, with ValidationError. Fix the batch and
send it again.
Everything the agent reads from the screen is untrusted input to the model:
page content, file contents, text in dialogs. Keep credentials out of the
desktop, restrict where the sandbox can connect with network
rules, and have the model ask for approval before an
action with an outside effect.
Watch it live
6080, publishing that port
under the sandbox’s preview policy. Under public the
URL is clean. Under private it carries a 60-second signed credential, as
getSignedPreviewUrl / get_signed_preview_url does; mint a fresh URL for
each viewer. viewOnly / view_only is a flag on that viewer only and does
not stop someone else from opening the same URL with control, so use a
private policy when the viewer must not be shared.
Screenshots
screenshot({ format: "jpeg" }) /
screenshot(format="jpeg") returns a JPEG instead; frame.format says which
encoding you got.
Mouse
drag takes the same button option as click. Scroll moves the wheel by
whole clicks and targets whatever is under the pointer, so move first to
scroll a specific element.
Coordinates are not checked against the display size. A point outside the
screen goes to the nearest edge: a click at (5000, 5000) on a 1280x800 display
lands at (1279, 799). The only upper limit is 65535: a larger coordinate
throws ValidationError.
Keyboard
write types literal text with no per-character pacing, so long strings land
in well under a second. press sends a key or chord. These friendly names work
in any case (enter, Enter, and ENTER are the same key): enter or
return, esc or escape, tab, space, backspace, delete, insert,
up, down, left, right, home, end, pageup, pagedown, ctrl or
control, alt, shift, and cmd, win, or super. Any other key uses its
X keysym name, so F5, Return, and KP_Enter work as written.
Key names are not checked. A name that is neither a friendly name nor an X
keysym, such as press("NotAKeysym"), does nothing and returns without an
error. In a batch it does not set actionError / action_error either. Check
the next screenshot when a key press must land.
commands.run sets no locale, so an application it starts can lose the
non-ASCII characters that write types. In a terminal started that way,
"A é ✓ 日本 Z" arrived as "A Z". Start desktop applications with a UTF-8
locale:
Batch actions
Models that emit several actions per turn can send them in one request:press action
is not a failing action (see Keyboard). step is the same batch
followed by a capture in the same request, for the agent loop above; actions
is for input you do not need to look at afterwards.
Resize the display
screenshot() on an
8192x8192 display threw a ServerError after about 5 seconds.
Pause and resume
The desktop survives a pause. Open windows, running applications, and the screen come back exactly as they were, and the nextdesktop call on a paused
sandbox resumes it first.
MCP
The MCP server exposes the same surface as thesandbox_computer tool, one
action per call (screenshot, left_click, type, key, scroll,
left_click_drag, …) in the vocabulary computer-use models are trained on.
Input actions return a fresh screenshot of the result by default. See
MCP server: Computer use.
Integrations
To connect a model’s computer-use tool to this desktop, see Claude computer use and OpenAI computer use.Related
- Headless browser — run a browser without a desktop
- SDK reference: Sandbox
- Preview URLs — access policy for the viewer port
- Templates — the
superserve/desktoptemplate