Skip to main content
A computer-use agent works a desktop the way a person does: it looks at the screen, decides, then clicks or types. sandbox.desktop gives an agent that desktop inside a sandbox: screenshots, mouse, keyboard, scroll, a resizable display, and a live viewer you can open in a browser. Every call is one request to the sandbox. A click is one round trip, a drag is one request, a whole model turn can go in one request, and typing has no per-character delay. A paused sandbox resumes on first use, as with commands and files.

Create a desktop sandbox

The sandbox must come from a desktop-enabled template. superserve/desktop includes the xfce desktop, Chrome, and a browser-based viewer, and runs the session as the non-root desktop user (also the default user for commands.run). The display starts at 1280x800.

The agent loop

Your code owns the model call and the conversation. The sandbox runs the actions. Each turn is a screenshot in, a list of actions out. In the loop below, nextActions / next_actions stands for your own model call: it sends the frame to the model and returns the model’s actions for the turn.
step runs the model’s whole turn and captures the frame after it in one request, so a turn is a single round trip. Input is delivered before the application has repainted, so with waitForChange / wait_for_change the sandbox captures the first frame whose pixels differ from the frame before the batch, waiting at most settleMs / settle_ms (default 1000, up to 2000); the result’s changed says whether a repaint arrived. changed is meaningful only with waitForChange: without it, changed is undefined in TypeScript and False in Python. A changed frame proves the display repainted, not that the application finished, so check the screenshot for what you expect. Without waitForChange the sandbox sleeps settleMs (default 0: no sleep) and then captures. With quietMs / quiet_ms the capture also waits until the display has been still for that long, so a batch that paints in steps (a focus ring, then the typed text) comes back settled rather than mid-way. quietMs requires waitForChange and can’t exceed settleMs, so raise settleMs for a quiet window above 1000; otherwise step throws a ValidationError. Frames are PNG unless format is "jpeg": several times smaller and faster to encode, the right trade when a model reads the frame. The result’s timings (queueMs, actionsMs, settleMs, captureMs) say where the sandbox spent the turn, so a slow loop can be attributed to input, the repaint wait or the capture rather than to the network. step does not throw when the batch stops at a failing action or when the capture fails, since the actions that ran have already landed: check actionError / action_error (with executed, the index it stopped at) and screenshotError / screenshot_error. Resizing to the model’s native resolution first keeps frames small and maps its coordinates 1:1 onto the screen. A batch that fails validation is different: step and actions throw, and no action in the batch runs. The SDK rejects a negative or non-integer coordinate and an empty key name before it sends the request, with Error in TypeScript and ValueError in Python. Python also rejects an unknown action type this way. The sandbox rejects a coordinate above 65535, and an unknown action type sent from TypeScript, with ValidationError. Fix the batch and send it again.
Everything the agent reads from the screen is untrusted input to the model: page content, file contents, text in dialogs. Keep credentials out of the desktop, restrict where the sandbox can connect with network rules, and have the model ask for approval before an action with an outside effect.

Watch it live

Returns a browser URL for the viewer on port 6080, publishing that port under the sandbox’s preview policy. Under public the URL is clean. Under private it carries a 60-second signed credential, as getSignedPreviewUrl / get_signed_preview_url does; mint a fresh URL for each viewer. viewOnly / view_only is a flag on that viewer only and does not stop someone else from opening the same URL with control, so use a private policy when the viewer must not be shared.

Screenshots

The capture includes the cursor. Coordinates in every other call are pixels in this image, with the origin at the top-left. screenshot({ format: "jpeg" }) / screenshot(format="jpeg") returns a JPEG instead; frame.format says which encoding you got.

Mouse

A click moves the pointer and clicks in one request. A drag runs as one atomic batch (press, move, release), so no other input can land in the middle of it. drag takes the same button option as click. Scroll moves the wheel by whole clicks and targets whatever is under the pointer, so move first to scroll a specific element. Coordinates are not checked against the display size. A point outside the screen goes to the nearest edge: a click at (5000, 5000) on a 1280x800 display lands at (1279, 799). The only upper limit is 65535: a larger coordinate throws ValidationError.

Keyboard

write types literal text with no per-character pacing, so long strings land in well under a second. press sends a key or chord. These friendly names work in any case (enter, Enter, and ENTER are the same key): enter or return, esc or escape, tab, space, backspace, delete, insert, up, down, left, right, home, end, pageup, pagedown, ctrl or control, alt, shift, and cmd, win, or super. Any other key uses its X keysym name, so F5, Return, and KP_Enter work as written. Key names are not checked. A name that is neither a friendly name nor an X keysym, such as press("NotAKeysym"), does nothing and returns without an error. In a batch it does not set actionError / action_error either. Check the next screenshot when a key press must land. commands.run sets no locale, so an application it starts can lose the non-ASCII characters that write types. In a terminal started that way, "A é ✓ 日本 Z" arrived as "A Z". Start desktop applications with a UTF-8 locale:

Batch actions

Models that emit several actions per turn can send them in one request:
The whole batch is validated before anything runs (a batch that fails validation throws, see The agent loop), then runs in order and stops at the first failing action. An unknown key name in a press action is not a failing action (see Keyboard). step is the same batch followed by a capture in the same request, for the agent loop above; actions is for input you do not need to look at afterwards.
Input is not idempotent. If a pointer, key, or batch call fails with a transport error, the input may already have been delivered. Take a screenshot and check the screen before repeating it.

Resize the display

Takes effect live, with no restart: open windows re-lay out. Width must be a multiple of 8 from 320 to 8192; height from 200 to 8192. At the largest sizes the capture can fail: a 7680x4320 display captures, but screenshot() on an 8192x8192 display threw a ServerError after about 5 seconds.

Pause and resume

The desktop survives a pause. Open windows, running applications, and the screen come back exactly as they were, and the next desktop call on a paused sandbox resumes it first.

MCP

The MCP server exposes the same surface as the sandbox_computer tool, one action per call (screenshot, left_click, type, key, scroll, left_click_drag, …) in the vocabulary computer-use models are trained on. Input actions return a fresh screenshot of the result by default. See MCP server: Computer use.

Integrations

To connect a model’s computer-use tool to this desktop, see Claude computer use and OpenAI computer use.