Overview
Instead of calling a tool 50 times (e.g., web_search, grep, update_row), the LLM calls batch once with 50 inputs. Ray executes them in parallel and returns one structured result.
Usage
import superserve
from superserve import BatchTool
@superserve.tool
def lookup_topic(topic: str) -> str:
"""Look up information about a topic."""
return f"Information about {topic}"
@superserve.tool
def translate(text: str, target_lang: str) -> str:
"""Translate text to another language."""
return f"Translated '{text}' to {target_lang}"
# Create batch tool with your tools
batch_tool = BatchTool(tools=[lookup_topic, translate])
# LLM calls it like:
result = batch_tool(
tool_name="lookup_topic",
tool_inputs=[
{"topic": "Python"},
{"topic": "Rust"},
{"topic": "Go"}
]
)
# Returns:
# {
# "results": ["Information about Python", "Information about Rust", "Information about Go"],
# "errors": [None, None, None],
# "tool_name": "lookup_topic",
# "count": 3
# }
With Pydantic AI
BatchTool returns a plain callable that works with any agent framework:
import superserve
from superserve import BatchTool
from pydantic_ai import Agent
@superserve.tool
def search(query: str) -> str:
"""Search for information."""
return f"Results for {query}"
batch_search = BatchTool(tools=[search], name="batch_search")
def make_agent():
return Agent("openai:gpt-4o-mini", tools=[batch_search])
superserve.serve(make_agent, name="search-agent")
| Field | Type | Description |
|---|
tool_name | str | Name of tool to call |
tool_inputs | list[dict[str, Any]] | List of input dicts |
| Field | Type | Description |
|---|
results | list[Any] | Results (None if error) |
errors | list[str | None] | Error messages (None if success) |
tool_name | str | Name of tool that was called |
count | int | Number of inputs processed |
API Reference
| Parameter | Type | Description |
|---|
tools | list[Callable] | List of tools to register |
name | str | Tool name (default: “batch”) |
description | str | Custom description |
BatchTool handles partial failures gracefully. If some inputs fail, others still return results. Check the errors list to see which inputs failed.
