Agent tool tasks: technical evidence¶

Historical cohorts retain their pinned sources, artifacts and timing scope. See the benchmark registry for active experiments and preparation status.

Note: "local original records" and paths such as docs/src/assets/benchmarks/.data/ refer to local measurement archives that are not published with the repository or the site; they are kept for batch provenance only.

Complete repair/testing takes about 4.61 s in image-free pVisor VM and 8.51 s on Firecracker / complete Ubuntu; staging takes 0.72 s. Ubuntu boot affects short jobs, while pVisor retains substantial tool-execution costs: fast startup does not establish leading tools. Claude/VM: initialization timeout / N=0; real Codex tool-loop results follow below.

Motivation¶

To detect interference, first fix tool actions and remove inference/Internet variability, then measure real-model tasks. This supplies a repeatable tool-loop baseline.

Experiment design¶

Pin client versions, project inputs and local model responses to compare environment/tool costs across native, staged, container and VM paths. Require correct repairs, zero-exit tests, returned real tool results and client completion; staging must preserve the original faulty file. Fake credentials, no inference or paid calls. The complete Ubuntu comparison uses 10 samples per available case; the historical shared-tool environment uses 30, both with 3 warmups and randomized order. Historical arithmetic tasks use six inputs, three repetitions, no warmups and fixed ordering. Report them separately without pooling distributions.

macOS¶

These workloads were measured on Linux; macFUSE/FSKit overhead and capacity remain unmeasured. Existing macOS/HVF results are retained in VM startup and VM memory, and are not substituted for this workload.

Linux: 2026-10-04¶

Image-free pVisor and complete Ubuntu: full Agent Env¶

pVisor reuses installed host Python/Node/Rust/Git/rg/Claude/Codex without making an OS image. Firecracker/QEMU install distribution tools and the same Rust/Agent CLIs in complete official Ubuntu 26.04.1 LTS. Each trial creates a new environment, then runs the same repair plan and tests. VMs use 2 vCPU / 16 GiB; all groups share the two-core budget and warm host caches, with 10 planned samples, 3 warmups and randomized order; cells state effective N when below 10. Ubuntu Python/Node/Git versions differ from the host; full versions and kernel differences are in the method.

The QEMU rows use the same complete Ubuntu template; all 60/60 new formal task samples pass, with N=10 and 3 warmups per case. Standalone version self-check timing is unmeasured (—). QEMU and the earlier pVisor/Firecracker cohorts are separate; the chart does not pool distributions.

Repair/testing has lower launch-to-result time; tools after boot do not have the same advantage. Repair/testing is 4.61 s end to end in pVisor VM versus 8.51 s on Firecracker/Ubuntu; internal tool/grading time is 4.02 / 2.30 s respectively. Avoiding complete OS boot reduces waiting; worker time exposes tool/storage costs. If a VM executes multiple tasks, boot is amortized and worker time matters more. Resident pools and sustained throughput are unmeasured.

The largest pVisor VM phase is node-install, at about 2.21 seconds P50. This identifies a concrete optimization path. Phase medians do not sum to the total median, and ext4/virtio-fs differences do not independently establish the entire cause.

Repair covers inspection, rg search, Python repair, Python/Rust/Node tests, installing 32 local npm dependencies and a diff. CLI columns use real clients with fixed local model responses and require passing test output in the returned model request and normal completion, without simulated CLIs or real inference. Staged/VM trials also verify the unchanged source workspace, completed Bundle and requested executor. All groups use private workspace temporary files to accommodate hostroot read-only /tmp. Initial linker failures from the incorrect configuration are retained as diagnostics outside formal distributions.

Claude/VM preflight still exceeds 90 seconds during initialization, with formal N=0. This is not a completed loop or evidence of full compatibility. Firecracker/Ubuntu Claude/Codex values use a separate ten-sample follow-up after correcting serial prompt and terminal control interference, with identical parameters. They are not pooled with the original client samples; collection failures remain archived.

Codex uses uniform inner danger-full-access, with the outer runtime supplying its stated boundary; default nested sandbox compatibility is unmeasured. N=10 gives first-version budgets and repeatability, not long-term tail or real-model success guarantees. RSS sums the launcher tree every 20 ms, including separate Firecracker DNS/NAT support; QEMU uses built-in user networking. Shared pages may be counted twice and short peaks missed. Configured 16 GiB is not 16 GiB resident, and short-task RSS does not establish concurrency capacity.

Local raw record docs/src/assets/benchmarks/.data/full-ubuntu-20261004/samples.csv · Local raw record docs/src/assets/benchmarks/.data/full-ubuntu-20261004/summary.tsv · Method and reproduction · Local raw record docs/src/assets/benchmarks/.data/full-ubuntu-20261004/ubuntu-workflows-private-tmp-20261004/evidence.tar.gz · Local raw record docs/src/assets/benchmarks/.data/full-ubuntu-20261004/ubuntu-clients-console-v2-20261004/evidence.tar.gz

Complete-Ubuntu repair/testing takes 8.12 s on q35 and 10.33 s on microvm, with internal tools taking 2.75 / 2.81 s. Image-free pVisor reduces waiting for fresh jobs, while its 4.02 s worker time remains above these complete-Ubuntu paths, a priority when environments are reused. Selecting microvm does not automatically improve complete tasks in this configuration. Networking, devices and exposed CPU features differ; the entire difference cannot be assigned to block devices or FUSE.

Local raw record docs/src/assets/benchmarks/.data/full-ubuntu-qemu-20261004/manifest.tsv · Local raw record docs/src/assets/benchmarks/.data/full-ubuntu-qemu-20261004/summary.tsv · Local raw record docs/src/assets/benchmarks/.data/full-ubuntu-qemu-20261004/ubuntu-qemu-complete-20261004/evidence.tar.gz · QEMU method

Historical controlled environment: shared tools and trimmed reference VMs¶

This batch shares tool artifacts to control some version differences. Firecracker/QEMU use trimmed Linux 6.12.109 and static init, without a complete distribution boot. pVisor shares a tool directory through virtio-fs without an image. These results retain Docker and minimal-VM context; see the complete Ubuntu and image-free hostroot deployment above.

The deployed environment contains Python 3.14.7, Node 24.18.0/npm, Rust/Cargo 1.98.1, Git, rg, Claude Code 2.1.128, and Codex 0.160.0: about 3.58 GiB and 76,000 files. Native, Docker and VMs use the same tool artifacts, projects and checks. Each case has 30 samples and 3 warmups, a two-core budget, and 2 vCPU / 16 GiB for complete VMs. Full parameters and boundaries are stated separately.

The workflow inspects a project, searches for the bug, repairs Python, runs Python/Rust/Node tests, installs 32 local dependencies, and generates a diff. Fixed model responses request that plan. Real CLIs must return actual passing tool results and exit normally. This covers environment deployment and tool execution, not large repositories, real model success rates or Internet dependency installation.

Version checks establish that tools launch. The repair column includes environment startup, repair, testing and result validation. CLI columns add client initialization, local protocol requests and returned tool output. The endpoint is the host receiving a validated result, followed by checks of zero exit, Run Bundle and files. Exit/unmount time is recorded separately. Bars show P50 and lines P95; failed cases have no latency sample. Phase medians do not sum to the total median.

The staged path fits an interactive tool budget; the VM path still needs substantial work. Staged repair adds about 0.20 s over native and takes about 0.20 s less than Docker. It supplies a separate change view and Run Bundle; Docker uses a writable mount, so the workflows are not identical. The VM's 3.97 s is about 4.4 times Docker and 2.1 times QEMU microvm. Roughly 86 ms startup does not imply equally fast task execution.

VM npm installation takes about 1.74 s, its largest phase here; Rust compilation takes about 0.84 s. Together with the file-operation comparison, this identifies dense small-file operations and offline installation as priorities for investigation. Guest configuration, filesystems and initialization also differ; the table locates costs rather than establishing a single cause.

Compatibility is a measured result. Claude passes 30/30 on native/staged/Docker/Firecracker/QEMU. Its pVisor VM preflight exceeds 90 s during initialization without a completed tool loop; formal N=0. Logs and diagnostics are retained; the root cause remains unresolved. Codex passes 30/30 on all eight backends with uniform inner danger-full-access, leaving the declared boundary to the outer runtime. This does not establish default workspace-write support everywhere. Native and staged retain access outside the workspace.

Do not rank Claude against Codex by absolute time. Codex also has seconds of client waiting in this path: roughly 6.26 s in Docker and 10.93 s in pVisor VM. These numbers describe the complete client experience more closely than bare startup, for these versions and controlled tasks, not model speed.

Deployment is separate. Offline tool copying, image import and ext4 creation take about 109 s; adding the CPU affinity helper takes another 5.3 s. Downloads and kernel compilation are excluded. Per-run workspace/private-disk preparation is recorded as prepare_ms, outside the task table. These are prepared-environment task budgets, not first-install-to-completion costs. See the memory audit; 16 GiB configured RAM does not mean each Agent consumes 16 GiB resident memory.

Local raw record docs/src/assets/benchmarks/.data/reference-env-20261004/samples.csv · Local raw record docs/src/assets/benchmarks/.data/reference-env-20261004/summary.tsv · Local raw record docs/src/assets/benchmarks/.data/reference-env-20261004/evidence.tar.gz · Local raw record docs/src/assets/benchmarks/.data/reference-env-20261004/compatibility.tsv

First-edition arithmetic tasks: separate historical batch¶

The following 72/72 results use a smaller task, another GNU artifact, and fixed ordering. Their statistics remain separate from the complete environment.

Analysis¶

Checks require the corrected file, passing grading observation in the next model request, normal CLI completion, and the original faulty file retained under staging. Compare native/staged within each CLI; client startup differences are not model-speed differences. Codex staged P50 is -2.3%, but native/staged order is fixed and the desktop has background work. This does not establish acceleration or an equivalence confidence interval. Inputs and server source are archived.

A deterministic 100% pass rate does not establish unchanged SWE-bench performance or a statistical success-rate interval. Coverage is these client versions and this Bash/exec_command repair path.

Limits and next measurements¶

Real-model SWE-bench Lite, actual tokens, complex multi-tool tasks and output variation remain unmeasured; paid accounts were not authorized. Follow-up should pin task IDs, commits, models, budgets and permissions, randomize native/staged order and retain failures/native trajectories.

Reproduction and evidence¶

Run from the repository root with a new output directory. This dynamic firmware entry requires the GNU/Linux CLI; static musl builds use a different firmware entry. This host has Linux, KVM/FUSE/user namespaces, Python 3.14, Rust/GCC, Git/rg, Node 24/npm and Podman/crun. The agent suite also needs the Claude/Codex CLIs.

python3 benchmark/pvisor/product_v1.py \
  --binary /absolute/path/to/gnu-linux/pvisor \
  --firmware /absolute/path/to/libkrunfw-directory \
  --replay-binary /absolute/path/to/pvisor-replay \
  --output target/product-benchmark-new \
  --suites agent --samples 3 --warmups 0

Start with --samples 1 --warmups 0 to check prerequisites. Workloads and correctness assertions live in benchmark/pvisor/v1/. Reports pin binaries, firmware and harness source with hashes. Failed operations never enter performance distributions. Effective sample counts are stated per page; P95/P99 from small samples describe this batch rather than production tail probabilities.

Environment, artifacts and method · Local raw record docs/src/assets/benchmarks/.data/product-v1-20261004/manifest.tsv · Local raw record docs/src/assets/benchmarks/.data/product-v1-20261004/samples.csv · Local raw record docs/src/assets/benchmarks/.data/product-v1-20261004/evidence.tar.gz. Reports retain dirty source status; executable SHA256 identifies the measured artifact. The archive excludes large rootfs/binaries and reproducible workspace payloads, while retaining input hashes and each batch's harness.