> ## Documentation Index
> Fetch the complete documentation index at: https://docs.superserve.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run agent evaluations and RL rollouts in sandboxes

> Run each evaluation trial or training rollout in its own sandbox, in parallel, and score the result as a reward.

An evaluation runs an agent or a model on many tasks and scores each result. Reinforcement learning (RL) does the same in a loop and uses the score as the reward. Both run code that a model wrote, many times and at the same time. Give each trial its own sandbox: create it, run the task, score the result, then delete it.

## Why use a sandbox

* **A clean start for each trial:** every sandbox is a new microVM from the same template, so one trial can't leave files or processes behind for the next one.
* **Isolation for model code:** code that a model wrote can't touch your training machine, its GPUs, or its credentials.
* **Parallel runs:** start one sandbox per trial and run them at the same time, up to your team's active sandbox limit.
* **Same setup every time:** bake the task's tools into a [template](https://docs.superserve.ai/templates/overview.md), or fork a prepared sandbox from a [snapshot](https://docs.superserve.ai/sandbox/snapshots.md).

## How it works

**1. Prepare the environment**

Use a public template such as `superserve/python-3.11`, or build your own template with the task's dependencies.

**2. Start one sandbox per trial**

Create the sandboxes in parallel. Block outbound traffic with a [network rule](https://docs.superserve.ai/sandbox/networking.md) if the task doesn't need it.

**3. Run the task and score it**

Send the model's output in with `files.write`, run the checks with `commands.run`, and turn the result into a score.

**4. Delete the sandbox**

Delete each sandbox when its trial ends, also when the trial fails.

## Score model outputs in parallel

This example scores two candidate programs for the same task, each in its own sandbox. The check prints the function's result, and the expected value stays in your code, outside the sandbox.

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark-default"}}
import { Sandbox } from "@superserve/sdk"

const candidates = [
  "def add(a, b):\n    return a + b\n",
  "def add(a, b):\n    return a - b\n",
] // model outputs

const check = "import json\nfrom solution import add\nprint(json.dumps(add(2, 3)))\n"
const expected = "5"

async function score(program: string): Promise<number> {
  const sandbox = await Sandbox.create({
    name: "eval-trial",
    fromTemplate: "superserve/python-3.11",
    network: { denyOut: ["0.0.0.0/0"] },
  })
  try {
    await sandbox.files.write("/work/solution.py", program)
    await sandbox.files.write("/work/check.py", check)
    const r = await sandbox.commands.run("python check.py", {
      cwd: "/work",
      timeoutMs: 30_000,
    })
    const last = r.stdout.trim().split("\n").pop()
    return r.exitCode === 0 && last === expected ? 1 : 0
  } finally {
    await sandbox.kill()
  }
}

const rewards = await Promise.all(candidates.map(score))
console.log(rewards) // [ 1, 0 ]
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark-default"}}
import asyncio

from superserve import AsyncSandbox, NetworkConfig

candidates = [
    "def add(a, b):\n    return a + b\n",
    "def add(a, b):\n    return a - b\n",
]  # model outputs

check = "import json\nfrom solution import add\nprint(json.dumps(add(2, 3)))\n"
expected = "5"


async def score(program: str) -> float:
    sandbox = await AsyncSandbox.create(
        name="eval-trial",
        from_template="superserve/python-3.11",
        network=NetworkConfig(deny_out=["0.0.0.0/0"]),
    )
    try:
        await sandbox.files.write("/work/solution.py", program)
        await sandbox.files.write("/work/check.py", check)
        r = await sandbox.commands.run("python check.py", cwd="/work", timeout_seconds=30)
        lines = r.stdout.strip().splitlines()
        return 1.0 if r.exit_code == 0 and lines and lines[-1] == expected else 0.0
    finally:
        await sandbox.kill()


async def main() -> None:
    rewards = await asyncio.gather(*(score(p) for p in candidates))
    print(rewards)  # [1.0, 0.0]


asyncio.run(main())
```

* Keep the expected values out of the sandbox. Code in the sandbox can read every file there, so a check that holds the answer lets the model's code find it.
* Past your team's active sandbox limit, `create` fails with [`too_many_sandboxes`](https://docs.superserve.ai/errors#ratelimiterror). For large batches, limit how many trials run at the same time.
* For a trial that runs an agent for several turns, keep the sandbox for the whole episode and delete it at the end. See [OpenEnv](https://docs.superserve.ai/integrations/evaluations/openenv.md).

## Integrations

To run benchmarks and training with a framework, one sandbox per trial, see [Harbor](https://docs.superserve.ai/integrations/evaluations/harbor.md), [OpenEnv](https://docs.superserve.ai/integrations/evaluations/openenv.md), [Hugging Face TRL](https://docs.superserve.ai/integrations/reinforcement-learning/trl.md), and [NVIDIA NeMo Gym](https://docs.superserve.ai/integrations/reinforcement-learning/nemo-gym.md).

## Related guides

> [card] [Code interpreter](https://docs.superserve.ai/guides/code-interpreter.md)
>
> Run model-generated code and read the output back.

> [card] [CI workers](https://docs.superserve.ai/guides/run-ci-jobs.md)
>
> Shard a test suite across sandboxes.

> [card] [Snapshots](https://docs.superserve.ai/sandbox/snapshots.md)
>
> Fork prepared sandboxes for each trial.

> [card] [Network rules](https://docs.superserve.ai/sandbox/networking.md)
>
> Block or allow outbound traffic for each trial.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.