Filesystem performance: technical analysis and experiment records¶

Current Linux HOST kernel caching and evidence¶

Under an explicit ownership contract, the Linux HOST API supports writable Metadata and read-only MetadataAndData; current-source correctness tests and independent request counters are valid, while formal performance acceptance remains blocked. Kernel caching and the service-side immutable-lower physical metadata cache below are separate mechanisms. Neither automatically changes product executor defaults.

Policies and admission contract¶

pvisor_overlayfs::api::OverlayMountConfig.kernel_cache explicitly selects the policy. Default behavior remains unchanged; no environment variable automatically enables extended policies.

Configured TTL defaults to 60 seconds and is ignored by Disabled/Uncached. Uncached does not enable DIRECT_IO: normal kernel page caching within an open handle remains. No policy enables writeback caching.

Extended policies require every physical lower explicitly declared Immutable in order, both OwnedViewContract assertions affirmed, explicit StableView read semantics, default_permissions and owner-only FUSE access. Contents, namespace, permissions, ownership, xattrs, hardlink aliases, physical ancestors and mount identities must remain stable. Only this adapter may mutate upper/work, which remain exclusive until actual detachment. Reads must not change backing atime: merged-mount NoAtime does not replace real backing noatime or an equivalent guarantee. A read-only mount, image digest or advisory lock does not prove these conditions. The adapter retains nonblocking exclusive flock locks on canonical upper/work directory objects; they coordinate cooperating opt-in sessions only, not out-of-band host writers.

Admission rejects journals/preimages, compact journal initialization, read metrics sinks, all custom path policies (including bound contexts) and exclusions before preparation I/O, without downgrading or silently skipping observations. StableView accepts that cache hits bypass callbacks; it provides no per-read audit, first-content-observation journal, snapshot or review compatibility. macOS and VM do not support extended policies. Virtio-fs lacks an equivalent verified notification output transport, and this DTO is not propagated into runtime/VM configuration.

Mutation effects and failure boundaries¶

Precise effects are collected before and after mutations, including requests that fail after partial copy-up or upper modification. Effects cover known object inodes, hardlink aliases, parent/ancestor attributes and exact namespace entries. Rename/exchange/removal include known subtrees, replaced objects and old/new entries; parent attribute changes do not expand into unrelated sibling-entry eviction. Covered operations include create/mknod/mkdir/symlink, unlink/rmdir, rename/exchange, link, writable open/copy-up/O_TRUNC, write, setattr, xattrs, fallocate and copy_file_range.

Copy-up and recursive directory materialization bind new physical objects back to existing FUSE identities. Late-discovered lower hardlink aliases obtain current upper identity and attributes from the Core owner-session copied_hard_link_metadata mapping. It survives adapter FORGET/reclaim; lookup/readdirplus materialize upper aliases when needed and record effects. The mapping belongs only to the current Core owner/session and is not persisted across rebuilds, reconstruction or new sessions. Adapter inode tables or canonical paths cannot replace it.

Pending is incremented before mutation, temporarily forcing zero TTL on metadata/entry replies, including mutation replies. An independent reply worker performs metadata-only inval_inode(ino, -1, 0) on affected inodes before replying; Linux cooperative mutation handling updates direct dentry/page-cache state. A separate entry worker asynchronously sends additional inval_entry(parent, name) notifications and expires attributes again. Long TTLs resume only after all batches complete. Entry notifications need not precede syscall completion. The reply worker never waits for the entry worker; FUSE callbacks do not wait for workers, notify the kernel or detach mounts. Writable inodes receive no blanket data invalidation that could wait on dirty pages/related writes; Metadata uses no KEEP_CACHE.

WRITE uses each request's current open flags, honoring dynamic clearing/restoring of O_APPEND. COPY_FILE_RANGE clears stale backing-fd O_APPEND before positional copying, without requiring an intervening WRITE. Errors after partial copying return the written byte count so the kernel learns about actual modifications.

Reply/entry queues each hold at most 256 batches, the stop queue 513 items, and global reservations at most 512 mutation batches. Each effects plan holds at most 4096 inode/entry keys plus one overflow sentinel. Overflow, worker panic or notification errors (except harmless ENOENT) stop the session; upper modifications are not rolled back. Writable admission requires the real fusectl abort endpoint and a disposable mount at the actual mountpoint verifying detach→abort. Users must not start before mount() returns.

Failure workers detach and verify mountinfo before writing connection abort; failed detach prevents the abort write. Helpers, termination, worker joins and shutdown have bounded waits, with a 15-second outer shutdown deadline; fatal termination failure ends the server process. Individual notifier calls have no independent deadline; termination limits do not guarantee that every mutation syscall completes within those limits. Abort/server death cannot revoke other processes' warm caches, and successful detachment cannot revoke held fds. EIO is not a general detachment fence. Callers must supervise and stop every user, release fds, retain backing for recovery, prohibit exported bind/namespace/FD aliases, and preserve namespace, credentials, helpers and detach permissions. Experiment failure evidence includes FUSE tasks waiting in request_wait_answer after namespace kill; simple-process termination probes do not establish immediate bounded cleanup for every FUSE wait.

Current validation and measurement scope¶

Targeted validation records on 2026-10-07 report 58 passed / 9 skipped for pvisor-overlayfs and 124 passed / 5 skipped for pvisor-overlay-core, plus 7 passed explicitly executed real-mount tests. Regressions cover late-hardlink identity after inode reclamation, dynamic APPEND with positional copy, and partial-copy failures. Independent source review found no new blocking issues in these identity and copy fixes. The validation scope does not include a comprehensive audit or performance acceptance.

B-FS-ENG plans a same newly frozen release-binary comparison of native, legacy-writable, Metadata-writable60s, Metadata-readonly60s and MetadataAndData-readonly60s: 3 warmups and 30 samples/cell, CPUs 0,1, seed 4207, 2048 byte-verified files in 32 half-deep branches. Writable comparisons hold view semantics fixed; KEEP_CACHE attribution compares only the two read-only conditions. Hot/readsearch/TTL operations have immediate priming. The TTL window waits 1.1 seconds outside timing and does not test 60-second expiry.

All native/lower/upper/work inputs reside on genuine noatime,nosuid,nodev,mode=0700,size=512m tmpfs in a private user/mount/PID namespace. Live mountinfo/device proofs and repeated physical file/directory reads with past atime cover backing including upper/work and establish unchanged atime. Merged NoAtime or future atime alone is not physical proof. RAM-backed evidence does not establish disk-backed performance/durability, VM, pvisor run or review guarantees.

The independent profile retains 90 fresh diagnostic cases, 5 persistent correctness processes, 95 logs and 152 unique final instances. Each instance's final cumulative record replaces earlier checkpoints rather than being summed. The table extracts request medians from the derived CSV for three diagnostics per condition; all ranges equal the medians. Warm deltas subtract the independently fresh-mounted prime-only case in the same round. Units are callbacks, with no timing values.

Counts support extended TTL avoiding repeated LOOKUP in this window and read-only KEEP_CACHE eliminating warm-operation READ callbacks. Writable Metadata does not eliminate GETATTR/OPEN/READ. The full lifecycle still includes initial reads; zero warm READ does not mean no physical reads or zero task cost. Diagnostic elapsed times do not enter formal distributions, rejected batches are not pooled, and earlier-source noatime performance cannot substitute for current-source acceptance. The separate valid immutable-cache results below retain their original measurement scope.

Formal acceptance status CSV · Independent request counters CSV · Engineering report · Reproduction and evidence contract. Both CSVs are directly copied derived data retaining source/binary digests and statistical scope. Raw samples, logs, frozen sources and receipts stay in benchmark/pvisor/.data/, without site publication or rewriting historical results.

Physical metadata caching for immutable lowers¶

Same-artifact real Linux FUSE A/B reduced read-heavy median operation time by roughly 24–26%, and fresh-process/mount/tool-through-unmount task time by 19.6%. This is B-FS-ENG engineering evidence, not an order-of-magnitude improvement or proof of equal gains for OCI, VM, full Agent tasks or staged execution with review journals.

Contract and experiment design¶

Each lower declares LayerMutability::{Mutable,Immutable}, defaulting to mutable. The caller guarantees lifetime stability of contents, metadata, namespace, parent and mount identity, including hardlink aliases. The cache retains successful physical-lower metadata only, bounded to 4096 entries. Upper, higher-priority mutable layers, whiteout/opaque, access policy and first observations remain checked. The kernel's one-second TTL, KEEP_CACHE and read-observation semantics are unchanged.

OCI extraction directories remain host-writable and lazy-image physical projections change on demand, so neither source is automatically promoted. Host rootfs, arbitrary directories and frozen baselines also receive no automatic promise. The experiment uses an exclusively owned, actually stable lower; a read-only mount is not a substitute for ownership.

Measured 2026-10-07 on Ryzen 7 9700X, Linux x86_64 and Btrfs, with driver/tools pinned to CPUs 0,1. Inputs have 2048 files in 32 branches, half deeply nested, totaling about 1.1 MB; host cache is warm. Native, mutable FUSE and immutable cache-off/on use one frozen release driver. Three warmups and 30 measured samples per cell are seeded-shuffled, totaling 600 measured samples with no formal failures. Both immutable arms keep identical declarations, changing only the service cache. No preimage journal is enabled, excluding review/apply cost.

Data and analysis¶

Milliseconds, P50; changes are relative to same-contract cache-off, with paired-round bootstrap 95% intervals.

Hot traversal has untimed priming. TTL conditions wait 1.1 seconds after priming outside the timer; the second pass may hit kernel cache, so this is not fully cold reading. Writes are correctness probes only, without write-performance claims. All reads verify complete bytes. Append-preserving copy-up, rename, unlink/recreate, whiteout, immediate/after-TTL visibility and exact upper inventories pass; complete lower content, namespace and metadata inventories are unchanged. Inputs do not trigger eviction. Large repositories, concurrency, macOS and guest paths remain unmeasured.

An independent profile has 24 instances, all with final records; instrumentation is excluded from timing. Persistent cache-on reduces physical parent-stat counts by 49.2% and leaf-stat counts by 84.3% versus off, with equal open/read counts. This supports less repeated server resolution, not proof of eliminating all FUSE requests or physical disk reads; inclusive spans are not summed. Cache-on remains substantially slower than native. Long kernel caching, upper consistency and mapping/fault paths need separate validation.

Formal statistics CSV · All-instance counters CSV · Engineering report and reproduction. Frozen sources, build receipts, input inventories, failed preflight, formal samples and independent profiles stay in benchmark/.data/immutable-cache-*, separate from user performance distributions.

Independent counters for pinned artifacts¶

Measured on 2026-10-06, CPUs 0,1, VM 2 vCPU/16 GiB, with verified frozen CLI, source, tool inputs and firmware. Seven-tool and fixed-repair workloads each run three times in staged/VM modes: twelve valid jobs, zero failures. Complete shared-input byte inventories are identical before and after. Executor-provided TMPDIR is preserved, with a fresh private empty tool cache per task; seven-operation Cargo temporary files and outputs remain in each independent workspace. Diagnostic stderr goes directly to a regular file, avoiding EAGAIN from nonblocking pipes during large profile writes. Profiles and startup timing are enabled; these times do not enter user performance distributions.

Every staged trial has 5/5 final instances, every VM 13/13, totaling 108 across rootfs, workspace and supervisor. Records use PID, component and instance, retaining only each instance's last cumulative snapshot. Frozen source construction order establishes rootfs/workspace scope; matching instance numbers in supervisor and VMM cannot be pooled. Inclusive spans nest and overlap across workers, so they cannot be summed into total runtime or a bottleneck percentage.

Seven-tool VM dispatch/admission records 1,260.54 ms for rootfs and 898.64 ms for workspace. It surrounds all of service_queues, including inline execution, and is not pure queueing or transport. Corresponding pool_queue_wait spans are 13.62 ms and 4.09 ms; waits from different workers still cannot be summed into task time.

Median fingerprint_bytes is 69,375,893 B for seven tools and 11,286 B for fixed repair, identical in staged/VM modes. The source hashes live-lower content on its first observation; ordinary stat/lookup does not hash regular files, and a frozen baseline skips read journaling. Checkpoint durability moves journal persistence to completion. The current seven-tool workspace journal syncs five times, with roughly 2–3 ms in the displayed sync spans; these counts do not establish per-file preimage fsync as the current dominant cost. This mutable fixture does not validate content-receipt reuse or replace read fingerprints with metadata-only observations.

All six VM diagnostics have zero rootfs CREATE, RENAME and WRITE requests with default scratch, supported by complete final records. Rootfs READ still totals roughly six thousand requests; its spans include service, locking and scheduling and cannot be read as disk or transport wait. Investigate path resolution, read observation, tool loading and cache writes separately. Repair reads hash little data but still incur rootfs resolution; the execution-mapping probe establishes only a short-process mapping-path difference. DAX, long attribute caching and individual functions' causal gains remain unverified, requiring engineering A/B that preserves access/staging semantics and independent formal timing.

The derived counter CSV retains three-run statistics and call ranges per filesystem scope, complete coverage and source digests. Original logs, frozen harness, build receipts and the summary script stay in local .data/. An earlier pipe-captured diagnostic with the same binary had five VM aborts from stderr EAGAIN printing panics; failed evidence is retained and excluded from this table. Regular-file capture is diagnostic only, and differences between capture conditions do not establish performance gains; see the counter manual.

Rootfs requests from the npm compile cache¶

The seven-tool workload's 537 rootfs renames come from Node/npm compile-cache creation. Linux, CPUs 0,1, guest 2 vCPU/16 GiB; the same frozen CLI, tool inputs and firmware alternate default and disabled caches in seeded randomized order. npm --version and the complete seven-tool workload each use six independent fresh VMs, three per condition: twelve successful VMs with 156 final filesystem instances. Both arms include the same Node status probe and subsequent cache inventory. Profiling and strace are enabled; no formal latency or performance gain is reported.

The table gives median counts across three VMs per condition. Every displayed count has the same minimum, median and maximum. Rootfs and workspace remain separate, identified by PID, component and instance; matching instance numbers in the supervisor and VMM must not be pooled.

The Node API confirms /tmp/node-compile-cache in the default arm and DISABLED in the other. Successful strace renames use the VM's root-upper/tmp/node-compile-cache paths, with counts matching rootfs RENAME, copy-up and actual cache files. Complete-tool outputs, every byte of all 256 written files, unchanged lower inventories and execution records pass validation; shared inputs are identical before and after. Disabling caching is a request-attribution control, not a product optimization or user configuration recommendation. The 537 requests cannot be converted into a verified millisecond gain.

Independent filesystem counters need to cover tool caches as well as workspace writes. Formal comparisons must declare whether caches persist between tasks, whether they start empty and which storage they use. Warm page caches and Node compile caches are separate conditions. Two npm calls within one task may reuse their cache; different backends and tasks must not accidentally share one. An empty private cache in workspace storage is a controlled policy, not a measurement of every executor's default temporary storage.

The derived cache-request CSV gives sample counts, count ranges, final coverage and binary, source, input, tracer and raw-report digests. Original syscalls, status output and audit scripts stay in local .data/, separate from user task latencies.

A separate diagnostic preserving pVisor's default temporary directory uses six fresh VMs, three per condition, with final records for all 78 filesystem instances. Guest statfs returns 0x01021994, confirming actual tmpfs. The enabled arm still creates 537 cache files every time, with zero rootfs CREATE, RENAME and WRITE requests. The disabled arm also has zero of those requests and zero cache files. Complete-tool outputs and written bytes pass validation. This establishes that cache creation can stay in default tmpfs. The older workload's tools_env() replaced the executor-provided TMPDIR with /tmp, moving cache writes into the rootfs overlay. These small instrumented diagnostics do not establish an end-to-end speedup; default temporary storage and the workspace-storage control need separate formal sampling.

The default-temporary-storage request CSV retains independent samples and provenance, without pooling them with the /tmp diagnostic.

FUSE requests and staging service costs¶

Four host conditions use the same verified pinned CLI and seven-tool inputs: native, direct host, a separately rebuilt passthrough FUSE driver and staged. Four preflights and three randomly interleaved rounds pass, totaling sixteen tasks. Every task's 256 written files are checked byte-for-byte; staged lower paths, modes and contents remain unchanged. Direct execution may rewrite only the fixture's verified toolchain path and refresh Git index stat information, with all indexed paths, modes and object IDs unchanged. Fresh caches preserve executor TMPDIR.

The driver uses the same frozen vendored fuser bytes, release profile, synchronous request loop and one-second TTL, without writeback or keep_cache. It omits OverlayCore, access policy, copy-up and preimage logging, so it is a transport-control lower bound. The following are request-count medians across three independent profiled tasks, excluding preflights:

These five measured callback classes have similar counts. Staged adds internal service work: 14,975 OverlayCore resolves with a median inclusive span of 143.52 ms, 4,853 read observations with 164.30 ms, and 2,157 content fingerprints with 45.15 ms. They nest and overlap, so they cannot be added or assigned percentages of task time. This supports testing resolution and read-observation costs separately from kernel request traffic. It does not show the passthrough driver offers equivalent semantics or establish a product speedup.

Derived FUSE request and service counters retain count ranges, component/instance scope and CLI, driver, source, input and audit digests. Instrumentation and sampled resource observations prevent these task times from entering user latency tables; unknown process lifetimes remain unknown. Raw evidence stays in .data/; reproduction is documented in the benchmark README.

Stage durability counters¶

A separate batch uses the same verified CLI and tool inputs with native, strict and checkpoint conditions. Three preflights and three randomized rounds pass, totaling twelve tasks. All staged tasks have the requested policy marker and completed seal, unchanged lower inventories, complete written bytes and five final filesystem instances. The table gives per-task medians across three independent profiled trials; it is not pooled with the FUSE-control batch.

On Linux each logged sync span invokes the journal file's sync_all; other directory/file synchronization is outside this counter. Strict has 339–340 such calls per task, checkpoint exactly five. The transaction span includes synchronization, so its time cannot be added to the sync span. This identifies persistence as a substantial strict-mode service cost. Checkpoint still records the same 2,703 appends and checks the completion seal; it permits an unsynced log before completion. These checks do not establish arbitrary crash-window equivalence, and profiled spans do not prove the same reduction in uninstrumented task latency. The engineering runner retains separate timing cohorts, distribution splits and paired intervals for at least thirty rounds.

Derived durability counters include all component instances, count ranges and source/input/audit digests. Raw reports, original stdout/stderr, output inventories and failed publication-audit attempts stay locally under .data/.

Independent executable-mapping diagnostic¶

Identical ELF, loader and all dynamic-library bytes for rg --version execute through virtio-fs files or executable memfds. Linux, inherited host affinity 0,1, guest configured with 2 vCPU/1 GiB; each condition uses three independent fresh VMs with 50 calls per VM, in randomized interleaved order. All twelve VMs and 600 calls pass output checks; every VM has final records for all 13 filesystem instances. The complete prepared input inventory passes verification before and after execution; retained workspace paths, copy bytes, commands, output and profiles pass an independent audit. Profile stderr is captured directly into a regular file. Build, input, firmware and raw-output digests stay in evidence. These profiled diagnostic runs do not enter user task-performance tables or establish complete descendant lifetime CPU placement.

First-exec statistics use the first call in each of three VMs. Repeated-exec statistics take each VM's median over its remaining 49 calls, then the median across three VMs; calls are not independent samples. Both arms copy and hash-check identical inputs and inherit the same fds; guest /dev/shm remains noexec. Original preparation prewarms original inodes; duplicate preparation reads identical bytes from separate inodes. Common Python preparation still warms its interpreter and some libraries, so this is not completely cold startup.

The repeated-exec mapping gap is about 4.4 ms, accompanied by major faults changing from 37 to zero. This supports investigating executable mapping and fault service first. Major faults do not equal physical disk reads. The substitution changes file mapping, path access and loader request paths together; it cannot attribute the whole gap to FUSE transport or missing DAX, or extrapolate gains by multiplying a complete Agent task's process count. Production changes need to preserve access policy, staging observations and evidence semantics before independent formal timing.

Derived execution statistics CSV · Provenance and measurement boundaries CSV. Raw reports, per-VM output, audit and the publication script stay in local benchmark/.data/.

Apply planning and target durability¶

CLI-only release builds share a frozen parent source; only the apply implementation differs in executed code. The compiler and features, profiles and rustflags of all 292 dependency compilation units match. CPUs 0,1, warm host caches, three warmups and 30 formal measurements per cell, randomized alternating order, independent target/stage per run; preparation is outside the apply timer. These 240 valid samples have profiling and strace disabled; changed files share one directory.

The frozen harness checks complete original, staged and final contents at runtime, then deletes generated workspace/upper files. Independent post-run checks cover retained builds, commands, exit codes, conflict output and committed ledgers; they do not independently recheck deleted final files. Conflict before apply does not test concurrent external writes during apply or crash recovery. This engineering A/B is not a comparison against Git or other products.

A separate profile/strace diagnostic has three trials per cell and 24 valid commands, retaining complete exit records and classifying calls by fd paths. Both variants have identical synchronization counts:

All these calls are zero for conflict-before-apply commands. The directory index improves planning without reducing target-durability calls; applying 10,000 files still takes about 11 seconds. Batching synchronization per unique directory at transaction boundaries merits a separate experiment, after verifying Prepared, TargetApplied and Committed durability ordering, crash recovery and concurrent external-edit protection. Counts establish call volume, not its share of all remaining time. Tracing perturbs durations; they cannot be added to nested profile spans or used in the formal benefit table.

Formal statistics CSV · Paired differences CSV · Synchronization counts CSV · Provenance and audit scope CSV. Raw records and publication scripts stay in local benchmark/.data/.

Retained independent engineering experiments¶

The new default-release local VM completes at P50 4.15 s (+3.3% versus optimized v3), with traversal at 157.57 ms (-6.5%). Warm lazy-image open/read improves 8.6% and 64 MiB reads improve 6.6%, but traversal slows and copy-up takes about 2.5× as long. This batch shows no general end-to-end acceleration; see the new-version evaluation for complete distributions and cache conditions.

Motivation¶

Agents repeatedly list, search and modify files. Tool time and full job time together show the cost of choosing staging or a VM.

Experiment design¶

The latest batch randomizes native and both versions of staged and libkrun VM. Historical matrices also include host, safe, Docker, rootless Podman/crun and pVisor OCI. Cells usually have 3 warmups and 30 measurements with warm host caches; the complete Ubuntu follow-up uses N=10, labeled separately. Image/input preparation is excluded. Worker time includes tool execution and validation; wall time includes launch and teardown. metadata/git/rg use 2,048 files in 32 directories; read validates a 64 MiB hash. product-v1 and the complete tool environment both write 256×64 KiB files, totaling 16 MiB; each cohort remains separate by configuration. cargo builds 64 dependency-free modules and verifies 2016. npm installs 32 local packages offline, without registry access.

macOS¶

These workloads were measured on Linux; macFUSE/FSKit overhead and capacity remain unmeasured. Existing macOS/HVF results are retained in VM startup and VM memory, and are not substituted for this workload.

Linux: evaluation after the shared filesystem service refactor¶

The frozen new source retains host FUSE while local, staged and lazy VM views enter the shared service through virtio-fs. Lazy images no longer create an intermediate host FUSE mount. The implementation still keeps protocol state in the two adapters; DAX, writeback and long attribute TTLs remain disabled.

Comparison and measurement scope¶

The baseline is the final optimized v3 artifact from the earlier five-item campaign, including path indexing, lazy directory allocation and queue changes; it is not a pre-P0 version. The new integrated worktree was frozen before building, with source and executable hashes recorded. Concurrent cluster/service changes are also present, so this is a version-artifact comparison, not an attribution to one function or solely to removing FUSE.

Both artifacts use identical firmware and complete tool fixtures, 2 vCPU/4 GiB and physical cores 0,1. Both staged variants must report rootless_process with read/write isolation. Each cell has one correctness preflight, three warmups and 30 measurements; preflight and warmups are excluded. Five cells shuffle each round and seven tools run in their fixed order. These workloads use local rootfs, which already had no intermediate host FUSE, making a separate lazy-image measurement necessary. Host caches are warm and CPUs are not exclusive. Release one-minute host load is 1.37 → 2.27; independent batches are not pooled.

Default release: complete local-rootfs workloads¶

P50 values below are milliseconds; negative changes mean less time. All 150 jobs and 1,050 tool measurements passed existing output, Run Bundle, untouched-lower and complete 256-file upper checks. Both release artifacts use the same size-oriented opt-level=z build.

New VM traversal P50 decreases 6.5%, but completion increases 3.3% and Git increases 22.3%. Host staged completion changes only +0.1%. This batch does not show general local-rootfs acceleration. Relative to its native control, the new VM takes approximately 34.4× for traversal, 67.5× for small-file writes, 3.6× for 64 MiB reads and 8.2× for npm. Small-file and metadata costs remain substantial.

Traversal tails decrease slightly, while read/write tails increase. Completion P95 is close and P99 decreases slightly. With 30 samples P99 is sensitive to individual jobs; median or isolated tail changes do not establish stable gains. The Local raw record docs/src/assets/benchmarks/.data/filesystem-service-20261005/local-release.tsv retains all seven worker timings and launch-to-exit time for every job.

Performance artifacts: local comparison with matched build settings¶

A separate complete batch has 150 jobs and 1,050 tool measurements, all passing correctness and isolation checks. Baseline and new versions both use opt-level=3. Host load is 0.93 → 3.87 and CPUs are not exclusive. Values are P50 milliseconds. Distributions are not pooled with release, and the new 4.15/3.81 s across those batches is not a causal build-optimization measurement.

New VM completion changes -1.3%, Git -3.9% and metadata +1.7%, with smaller median changes elsewhere; host staged completion changes -0.1%. Release completion +3.3% versus this batch's -1.3% is inconsistent with general acceleration. Current performance VM traversal is about 32.4× its same-batch native control, small-file writes 62.8×, reads 3.6× and npm 7.4×. The new performance binary is 19.38 MiB, versus release 14.99 MiB, approximately 29.3% larger.

The Local raw record docs/src/assets/benchmarks/.data/filesystem-service-20261005/local-performance.tsv retain every sample. The two figure panels are independent batches, each comparing old/new artifacts with matched build settings. They are not a same-batch release/performance A/B or individual-mechanism gains.

Lazy images: cold/warm comparison without intermediate host FUSE¶

Both artifacts use performance builds (opt-level=3) and read the same immutable rootfs through a local Unix-socket cache-v1 fixture. The Python cache server responds and verifies content hashes on separate physical cores 2,3; VM and runner use 0,1. No artificial latency is added. This is not a WAN, TCP/S3, registry-download or production Rust cache-server throughput measurement. One-minute host load is 1.00 → 1.85.

Each round shuffles version order, running cold then warm for each version. Cold uses fresh client content/metadata caches; warm reuses those disk caches but starts a new guest, backend projection and upper. Host page caches are not evicted. After tool startup the script waits a fixed 1.2 s, then measures first and immediate repeat traversal, first and immediate repeat open/read, 64 MiB SHA256 and 32 file copy-ups. “Immediate repeat” describes order, not guaranteed kernel cache hits: the default TTL can expire during long operations. The wait is excluded from workers and included in launch-to-exit, diluting whole-job percentage changes.

Four cells have three warmups and 30 measurements each, plus preflight: 120 jobs and 720 operation measurements. The 2,048 small files occupy 32 directories and contain 1,023 bytes each; 32 copy-up files are rewritten to 17 bytes and verified. All guest content/attribute, Run Bundle VM isolation, staged workspace-write and immutable-source checks pass. Mount monitoring observed host FUSE in all 60 baseline measurement jobs and no FUSE mount in the image store of all 60 new jobs.

Values below are P50 milliseconds; negative means less time.

Reads improve locally: warm-cache open/read/close decreases 8.6%, and 64 MiB reads decrease 6.6%. First traversal instead increases 10.7%/13.7% for cold/warm caches, immediate repeat traversal increases 32.6%/28.9%, and copy-up takes 2.5–2.6× as long. Whole-job P50 changes +2.3% cold and +0.6% warm, without overall acceleration. A shorter call chain does not automatically remove metadata projection, content materialization, preimage recording and synchronization costs. This batch does not separately profile those costs or attribute a regression to one specific mechanism.

Every cold job downloads the same fixture data in both versions: 2,112 content reads and 69,203,968 bytes, comprising 2,048 small files and 64 large-file blocks. Including the interpreter and other rootfs content gives 2,164 reads and 86,924,150 bytes. Both warm versions download zero content and perform no remote stat/list queries. Differences are not due to extra prefetching or omitted data validation in the new version.

Warm read tails improve in some cases; metadata/copy-up and whole-job tails increase. Baseline cold open/read P99 of 1,628.39 ms and completion P99 of 4,406.78 ms are affected by one slow job; the new values are 804.34/3,736.05 ms. Selecting only those tails would not establish stable gains. Local raw record docs/src/assets/benchmarks/.data/filesystem-service-20261005/lazy-performance.tsv are retained. The first preflight failed because the script misstated fixture byte size; corrected preflight and formal results are stored separately, with that failure excluded from timing distributions.

Linux: retained results of the earlier five-item optimization campaign¶

These independent earlier batches produced this campaign's baseline artifacts. Their percentage changes cannot be added to the new results; the historical 16 GiB VM's 5.27 s versus the new 4.15 s is not solely a refactor gain. Final v3 combines path indexing, lazy directory allocation, queue changes and related work, with 2 vCPU/4 GiB, three warmups and 30 measurements per cell. It does not attribute gains to each mechanism. The mutable fixture does not quantify immutable-image receipt reuse.

Combined release completion P95 changes +2.4% and Git P99 +25.5%; separate performance completion P95/P99 change -7.9%/-6.9%. These tradeoffs cannot be pooled into one speed curve. The Local raw record docs/src/assets/benchmarks/.data/filesystem-optimizations-20261005/code-v3-4g.tsv, Local raw record docs/src/assets/benchmarks/.data/filesystem-optimizations-20261005/profile-v3-4g.tsv and Local raw record docs/src/assets/benchmarks/.data/filesystem-optimizations-20261005/manifest.tsv retain full provenance. Kernel-mechanism and P0 batches below are also historical.

Linux: full kernel-mechanism and concurrency evaluation¶

Screening precedes the unchanged seven-tool evaluation below. These are real filesystem jobs, not extrapolated adapter timings or three-sample screens. Retained changes fuse parent statx queries, share the lock for read-only OPEN, and add opt-in lock/pool diagnostics with no clock reads when disabled. READDIRPLUS_AUTO and inline metadata dispatch remain; DAX, writeback and long-lived caches are not enabled.

Completed experiments¶

The five cells are native, baseline/candidate host FUSE staged, and baseline/candidate KVM VM. The earlier P0 modification's N=30 A/B remains separate, outside the latest two rounds' 300 measured jobs. DAX, writeback cache, FUSE passthrough, FUSE-over-io_uring, long-lived TTLs and replacement with virtiofsd have only been assessed for feasibility; no performance A/B has been implemented for them.

Screening and implementation selection¶

Each real screen uses five cells, one warmup and three measurements per cell, frozen source, firmware and fixtures, with mandatory preflight. Combined variants are labeled; separate batches are not single-change comparisons. Screens select further work, not tail or whole-task performance claims.

Initially ordinary ENOENT/ENOTDIR also fell back from statx to metadata, offsetting some savings on lower queries. The final implementation returns those namespace results directly. Unsupported/restricted queries or missing fields fall back safely, with no guessed mount ID or native parent reuse. Physical ancestor and leaf checks remain, without cross-request attribute or permission caching.

A separate release adapter microbenchmark uses two warmups, eight measurements and one separate diagnostic per case, without VM, FUSE or preimage journaling. Deep-directory P50 is 97.52 → 88.16 ms (-9.6%); shallow lookup/open/getattr 28.39 → 27.61 ms (-2.8%). The final normal fixture has no metadata fallbacks and avoids 33,664 separate mount-identity queries in the deep case. This verifies reduced duplicate work, not a predicted VM-task gain.

The new regression reproduces read-only OPEN waiting behind independent backing READ: it fails the old implementation and passes the shared-lock implementation. Writable OPEN still waits and lower content stays unchanged. APPEND/TRUNC, non-read-only and kill_priv opens, mutations, release and snapshots remain exclusive. Per-path journals still synchronize first observations; handle, descriptor RAM-lease, used-ring and freeze-drain contracts remain. All five implementation-file hashes match the actual worktree.

Same-source full seven-tool A/B¶

Both release binaries use frozen bc08f457 source, differing by the retained implementation patch. Each cell has three warmups and 30 measurements: 150 measured jobs and 1,050 tool measurements, all passing original output, Run Bundle, isolation, lower-write absence and complete 256-file upper checks. Each job has a fresh workspace/upper and runs the seven tools in their original order; five cells shuffle with a fixed seed. The retained fixture still writes 256 × 64 KiB, checks 64 MiB and traverses 2,048 files with the original cargo/npm inputs. Host cache is warm; affinity uses physical cores 0,1, VM 2 vCPU/16 GiB, and firmware matches P0. Both same-source staged variants are rootless_process, requiring read/write/non-bypassable boundaries. Published P0 staged used host_process; its comparison is separate. CPUs are not exclusive; host one-minute load is 2.02 → 2.70. Units below are ms; negative is faster.

The full run does not establish universal acceleration. Staged completion is -2.6%, VM completion -0.5%; VM Git is -12.1% in this batch, while metadata is +3.2%. The adapter's -9.6% must not be reported as VM traversal improvement, and Git cannot stand in for npm, writes or the complete task.

Tails do not improve together. Candidate VM rg P99 goes 526 → 660 ms, npm 2001 → 2094 ms, completion 5917 → 6032 ms, while write tails fall. Staged completion tails include a few slow tasks and are not the sum of seven worker times. All distributions remain available; this is one full batch on a nonexclusive host.

Against same-batch native, candidate VM traversal is 39.8×, 256-file writes 65.4×, 64 MiB reads 4.8×, npm 9.3×. Repeated directory/attribute operations, small writes and VM tool execution remain substantial costs.

Published P0 artifact rerun in the same batch¶

The published pinned P0 candidate (a1020d4b plus shared stdio readiness fix) is then compared with final P1: again three warmups and 30 measurements per cell, an additional 150 measured jobs and 1,050 tool measurements, all passing the same checks. Original fixture, firmware and two-core budget remain; load is 0.45 → 3.44. Units are ms. This is a same-batch artifact comparison with different source bases, not attribution of every difference to statx or OPEN. P0 staged is host_process, P1 rootless_process; each observed boundary is validated strictly.

Second-batch VM completion is +0.4%, again without overall acceleration; Git is -4.3%, writes +5.3%. Staged npm is +21.1%, versus -1.5% in the previous same-source rootless comparison. Different execution boundaries and other source changes enter this artifact comparison; it is not a causal claim about the filesystem patch. The same P1 artifact traverses in 194 / 228 ms and reads in 156 / 121 ms across the two batches, demonstrating batch sensitivity. Neither faster-batch selection nor percentile pooling is valid.

Original P0 published traversal 195.24 ms remains below; the identical P0 artifact measures 225.02 ms here. Published-to-current differences likewise cannot be equated with code gains. Both complete evaluations total 300 measured jobs and 2,100 tool measurements. Evidence supports eliminating duplicate parent queries and read-only OPEN serialization, not universal end-to-end acceleration.

Local raw record docs/src/assets/benchmarks/.data/filesystem-kernel-20261005/full-historical.tsv · Local raw record docs/src/assets/benchmarks/.data/filesystem-kernel-20261005/historical-summary.tsv · Local raw record docs/src/assets/benchmarks/.data/filesystem-kernel-20261005/samples.csv

Cache, concurrency and guest-local controls¶

The separate diagnostic repeats traversal of the same 2,048-file fixture, waiting 1.2 seconds before every expired-cache pass, including after the initial rglob. Candidate VM results below are in ms. Tmpfs is verified as /dev/shm inside the same guest; file preparation is excluded.

Immediate repetition shows caching helps, with substantial cumulative cost remaining. Overlapping submission reduces waits without proving concurrent Core execution. The partial-write control has one sample and is not acceptance evidence. Tmpfs excludes virtio-fs, Core, journal and storage latency together; the difference cannot all be attributed to transport.

A separate profile batch is excluded from acceptance timings. Last candidate VM checkpoints show 21,340 / 24,743 inline and 357 / 65 pool requests. Pool queue waits total 12.80 / 1.67 ms; completion-to-used-ring publication 5.16 / 1.22 ms. Combined read/write lock acquisition waits are about 1.09 / 2.55 ms, with individual maxima below 0.38 ms. This diagnostic does not show long lock waits, but excludes guest-to-host admission waiting and does not rule out inline serialization. Core checkpoints still show 121,768 / 169,649 parent observations and 63,375 / 65,488 fused statx queries, with no fallback records. Complete staged profiling records 341 journal fsync calls totaling about 119 ms. VM checkpoints are not final and may have different cutoffs. Inclusive spans are not summed; counters are not a complete syscall census.

The initial diagnostic stopped because it searched runs only, while --stage places the Run Bundle in stage. The failed batch is retained and corrected runs use new directories. The intermediate successful batch did not expire initial rglob caches; it is kept separately, excluded from final diagnostic timings. All normal staged writes remain in upper, with lower unchanged.

Priorities for further kernel mechanisms¶

1. Directory/attribute/negative caches and invalidations. Repeated traversal remains expensive. Establish immutable or single-owner tree contracts before testing longer TTL and FOPEN_CACHE_DIR. Live lowers allow external host changes, and no active invalidation transport exists; do not extend TTL globally.
2. Workload-aware READDIRPLUS and host concurrency. AUTO already exists, and forced PLUS has tradeoffs. Host fuser uses one synchronous callback loop. VM uses one request and one hiprio queue; short metadata runs inline, and a full pool pauses normal admission. More workers alone do not prove gains; measure queue/lock waits. ASYNC_READ/PARALLEL_DIROPS are already negotiated by the protocol server.
3. Writeback cache. It may coalesce small writes, but requires durable preimages before mutation, dirty-page draining before terminal/export/snapshot, and partial-write/append/truncate correctness. Kernel I/O modes
4. Data paths and local filesystems. DAX primarily removes data copies; no DAX A/B is performed here. FUSE-over-io-uring addresses host /dev/fuse, not direct replacement of VM virtqueues. FUSE passthrough needs a backing FD in the same kernel, not a host FD handed to the guest. Guest-local tmpfs or read-only block images/kernel filesystems merit separate evaluation with staging/recording/restore contracts preserved. DAX · io-uring · Passthrough

Validation passes: 97 Core tests (five skipped), 12 overlayfs tests, 294 VM tests (two skipped), 25 benchmark tests, all-targets Clippy for three relevant packages and benchmark Ruff. No semspec ledgers/snapshots were modified or approved. Evidence includes samples and failures, not only successful summaries. These numbers do not update macOS, startup, networking or complete Agent-loop measurements.

Local raw record docs/src/assets/benchmarks/.data/filesystem-kernel-20261005/full-same-source.tsv · Local raw record docs/src/assets/benchmarks/.data/filesystem-kernel-20261005/summary.tsv · Local raw record docs/src/assets/benchmarks/.data/filesystem-kernel-20261005/process.tsv · Local raw record docs/src/assets/benchmarks/.data/filesystem-kernel-20261005/micro-final.tsv · Local raw record docs/src/assets/benchmarks/.data/filesystem-kernel-20261005/kernel-probe.tsv · Local raw record docs/src/assets/benchmarks/.data/filesystem-kernel-20261005/profiles.tsv · Local raw record docs/src/assets/benchmarks/.data/filesystem-kernel-20261005/implementation.patch · Local raw record docs/src/assets/benchmarks/.data/filesystem-kernel-20261005/evidence.tar.gz · Local raw record docs/src/assets/benchmarks/.data/filesystem-kernel-20261005/manifest.tsv

Linux: 2026-10-05, P0 real VM/FUSE A/B baseline¶

This batch boots actual KVM VMs and mounts actual host FUSE, completing the end-to-end check missing from the adapter experiment. Host devices are available; the initial sandbox hid /dev/kvm and /dev/fuse. Both GNU/Linux release binaries come from the same a1020d4b source archive and differ only in OverlayCore's core.rs. Both include the same guest stdio readiness fix. Binary, firmware, source patch and harness hashes are retained.

“After” means this pinned candidate artifact, measuring the incremental benefit of skipping marker probes for an unavailable upper. Other optimizations already in the archive are shared by both binaries and are outside the A/B difference; later commits do not automatically enter these measurements.

Each round shuffles five cells: native and both versions of FUSE staged and VM. Each cell has three warmups and 30 measurements: 150 measured jobs and 1,050 tool measurements, all correct. Every job has a fresh workspace/upper and executes the seven workloads in order in one environment. Fixture copying is excluded. Host caches are warm; all launch trees use physical host cores 0,1; VMs have 2 vCPU and 16 GiB. The pinned historical fixture and product-v1 workload both write 256×64 KiB (16 MiB). Historical percentiles are not pooled. Concurrent host activity remained: one-minute load average fell from 10.38 to 3.06. Affinity is a shared execution budget, not exclusive CPUs. This is one screening batch.

Tool operation and validation P50 in ms; negative changes mean less elapsed time:

Launch-to-exit P50 for the seven-workload job is 454.50 ms native, 1288.69 → 1230.93 ms (-4.5%) FUSE staged and 4829.04 → 4777.55 ms (-1.1%) VM. Metadata savings reproduce in the actual paths. This does not establish improvements for VM git, writes or overall jobs: VM write and git became slower in this batch. Small changes and write behavior need a quieter-host rerun; a single P50 cannot establish their cause.

P95 and remaining costs¶

P95 from the same samples, in ms. Lower medians did not consistently produce lower tail latency:

Candidate staged / VM whole-job P99 is 13.36 s / 10.46 s, also retained in the raw summary. With N=30, a few slow samples influence the tail, and concurrent host load remains uncontrolled. These observations need reproduction before attributing them to an implementation change. Evidence supports metadata-path improvement; overall performance acceptance still needs repeated quieter-host batches.

Against same-batch native, candidate staged / VM metadata still takes 15.9 / 39.6 times as long, adding 73 / 190 ms per traversal. Writing 256 files still takes 199 / 257 ms, versus 3.90 ms native. VM offline npm takes 1.50 s, versus 0.17 s native. Repeated traversal, small-file writes and VM tool execution remain the next optimization targets.

An earlier warmup exited zero and completed writes without any workload stdout markers. The harness stopped and retained the failure. The ordinary VM runner now declares the named ports required by its actual non-terminal standard descriptors. The guest waits for their names before launching tools, failing after at most five seconds. Regression tests cover delayed names, later port creation and absent ports; there is no unconditional startup delay. All 68 VM runs in the formal batch, including preflight and warmups, passed output checks. The harness also verifies observed isolation, no writes in lower and all 256 upper files with correct sizes. Identical binaries, a mismatched build manifest, missing samples or interruption cannot produce a successful A/B summary.

A separate profiled batch does not enter the timing table. The candidate VM's two nonempty Core instances still record 126,034 / 168,027 parent metadata calls and 57,547 / 53,161 mount identity attempts at their last checkpoints. Dispatch checkpoints show 20,545 inline / 424 pool and 24,398 inline / 100 pool, with two workers each. The complete candidate host Core profile records 341 journal fsync calls totaling about 110 ms. Repeated Core physical path checks, first-observation/write journal costs and inline dispatch of small requests warrant separate experiments. Neither more workers nor virtiofsd can be assumed to remove these costs. VM checkpoints lack final records and are partial; nested inclusive spans must not be added, and admission time is not queue waiting time.

Validation passed: five guest tests, 292 full VM-package tests (two skipped), 24 executor tests excluding control and 24 benchmark tests, plus guest Clippy and benchmark Ruff. The wider executor subset in the worktree at measurement time had nine VM-control test failures, retained separately in the validation archive. This change does not modify control implementation or count those checks as passed.

Build both GNU/Linux binaries from the same source, applying only the proposed change between builds. Use a fresh output directory and start with --samples 1 --warmups 0 for preflight.

python3 benchmark/pvisor/filesystem_ab.py \
  --assets target/reference-env-final-20261004 \
  --baseline /absolute/path/to/pvisor-before \
  --candidate /absolute/path/to/pvisor-after \
  --firmware /absolute/path/to/libkrunfw-directory \
  --output target/filesystem-ab-new \
  --cpu-affinity 0,1 --samples 30 --warmups 3

Local raw record docs/src/assets/benchmarks/.data/filesystem-ab-20261005/samples.csv · Local raw record docs/src/assets/benchmarks/.data/filesystem-ab-20261005/summary.tsv · Local raw record docs/src/assets/benchmarks/.data/filesystem-ab-20261005/profiles.tsv · Local raw record docs/src/assets/benchmarks/.data/filesystem-ab-20261005/evidence.tar.gz

Retained historical-to-P0 comparison¶

This compares the 2026-10-04 complete-tool-environment N=30 batch with the latest candidate N=30 batch: internal tool P50 in ms. Both retain the tool fixture, but artifacts, host load and run protocols differ. Percentages describe differences across batches and cannot establish the controlled causal benefit of all optimizations. Use the same-batch A/B for the latest change's incremental benefit.

Existing measurements show lower traversal and search times; reads and writes did not improve alongside them. Docker, complete repair tasks and real Agent CLI loops were not rerun here. Historical data below retains its original dates and artifacts; filesystem percentages cannot predict their latest performance.

Local raw record docs/src/assets/benchmarks/.data/reference-env-20261004/summary.tsv · Local raw record docs/src/assets/benchmarks/.data/filesystem-ab-20261005/summary.tsv

Linux: 2026-10-05, OverlayCore resolution optimization¶

This release-mode experiment measures only the virtio-fs OverlayFs adapter, without a VM, host FUSE mount or preimage journal. Fixtures live on /tmp tmpfs: 32 directories with 64 files of 18 B each; deep paths have eight parent components. Each trial has fresh inode tables and warm host caches. Fixture and adapter construction are excluded from timing.

Preserved baseline and candidate test binaries were run through nextest binaries metadata in three alternating batches: old→new, new→old, old→new. Each case and batch has two warmups, eight unprofiled samples and one diagnostic sample. P50 uses only the 24 unprofiled samples per version. CPU affinity was not pinned; binary digests and individual batch medians are retained in the raw summary.

The shared OverlayCore now skips whiteout/opaque probes when this candidate's physical upper parent was just found missing or non-directory. Attributes and absence are not cached across requests; later components and final physical ancestors are still checked afresh. Deep-path diagnostic counts fell from 38,016 to 4,352 for each marker probe. Markers behind upper ancestor symlinks cannot hide lower entries. New tests also cover upper directories and whiteouts appearing within a walk and upper content changes between requests.

All 106 Core/host-adapter tests and 57 virtio-fs/descriptor/filesystem-snapshot tests passed, as did Clippy for all targets of the three relevant packages. The initial sandbox hid /dev/kvm and /dev/fuse, so that full VM-package attempt failed at KVM initialization. This microbenchmark did not measure actual VM/FUSE jobs, journaling, payload I/O or macOS. Subsequent host validation and end-to-end runs appear in the P0 baseline above. Unrelated VM refactors also occurred between builds; this adapter case does not execute UART/VMM/CPU initialization. Binary digests identify the measured artifacts. These reductions do not replace new measurements of the historical tool workloads below.

Reproduce one version's adapter measurement:

cargo nextest run --locked --release -p pvisor-vm \
  --run-ignored only --no-capture -E 'test(small_file_adapter_benchmark)'

Local raw record docs/src/assets/benchmarks/.data/overlay-resolution-20261005/samples.tsv · Local raw record docs/src/assets/benchmarks/.data/overlay-resolution-20261005/summary.tsv

Linux: 2026-10-04¶

Tool execution inside complete Ubuntu¶

The table excludes environment boot, measuring operations and grading. This new batch uses N=10 with 3 warmups, the same fixture and two-core budget, and 16 GiB VMs. pVisor uses host directories and staged virtio-fs; Ubuntu uses its vendor generic kernel, distribution tools and private ext4. The old Docker N=30 matrix remains separate without pooling percentiles.

These values locate waiting in traversal, search, compilation and installation. A single read does not establish that block devices always outperform FUSE: kernels, tool versions, storage and staging semantics differ together. For longer tasks, combine worker time with the complete loop rather than startup alone.

Local raw record docs/src/assets/benchmarks/.data/full-ubuntu-20261004/samples.csv · Local raw record docs/src/assets/benchmarks/.data/full-ubuntu-20261004/summary.tsv · Method and reproduction

Docker baseline in the complete tool environment¶

This adds a same-host Docker Engine measurement rather than relabeling Podman. Same two-core budget and inputs, 30 samples and 3 warmups per case; fixtures match the first edition, with seven operations executed sequentially in each fresh environment. Timings cover operations and validation, excluding environment startup. See the complete environment for the repair workflow. The two batches remain separate; percentiles are not pooled.

This establishes a concrete position: Docker metadata/read/write stay close to native. Staged small-file traversal adds about 175 ms over Docker, and a 64 MiB read about 16 ms. An individual read costs tens of additional milliseconds; repeated small-file scans accumulate. VM offline npm installation takes about 1.73 seconds, against Docker's 0.23 seconds, a clear remaining gap. The roughly 86 ms VM startup figure cannot stand in for this tool budget.

Tasks verify file counts/sizes, SHA256, clean Git state, search matches, compiled output, and installed package counts. Docker uses a writable bind mount, pVisor a staged view, and Firecracker/QEMU private ext4. Different file paths are part of actual deployment cost; this is not a causal experiment changing only the VMM over an identical filesystem. The summary also contains each operation's distribution for other runtimes.

Configuration and reproduction · Local raw record docs/src/assets/benchmarks/.data/reference-env-20261004/samples.csv · Local raw record docs/src/assets/benchmarks/.data/reference-env-20261004/summary.tsv · Local raw record docs/src/assets/benchmarks/.data/reference-env-20261004/evidence.tar.gz · Local raw record docs/src/assets/benchmarks/.data/reference-env-20261004/compatibility.tsv

First-edition Linux matrix: separate batch¶

Analysis¶

Staged sequential reads cost about +25% and offline npm +19%. Metadata is about 31× native and small-file writes about 50×, adding roughly 150–190 ms to native operations lasting only milliseconds. Host worker time is near native; full jobs include CLI/recording overhead. Safe adds namespace/policy/proxy setup. Most measured VM jobs take roughly 0.5–1 second.

This historical batch uses Podman as the OCI control; Docker daemon access was unavailable then. A later batch adds rootless Docker Engine bind-mount measurements, shown in the complete-environment comparison above. Docker overlay2 and Docker Desktop remain unmeasured. pVisor OCI prepares a private rootfs per Job; wall time includes this while worker time isolates tool execution.

Compatibility follow-up¶

Container cargo initially lacked linker startup files, a fixture-image error. After adding glibc/GCC files, Fedora linker scripts still required /lib64/libmvec.so.1. The final cargo-ready batch supplies that path and passes every sample, listed separately; both setup failures remain in reports. VM Node failed to reserve V8 address space with the 1 GiB shape; follow-up uses a 16 GiB address-space configuration, kept separate from the main 1 GiB batch.

Limits and next measurements¶

Main VM jobs use host rootfs /, 2 vCPU/1 GiB and a host read view. This is a tool compatibility profile; host-secret protection is tested separately in isolation. These small offline warm-cache tasks do not establish full-repository build performance or cold-disk throughput. macFUSE/FSKit, actual registry installs and Docker/overlay2 await comparable measurements.

Reproduction and evidence¶

Run from the repository root with a new output directory. This dynamic firmware entry requires the GNU/Linux CLI; static musl builds use a different firmware entry. This host has Linux, KVM/FUSE/user namespaces, Python 3.14, Rust/GCC, Git/rg, Node 24/npm and Podman/crun. The agent suite also needs the Claude/Codex CLIs.

python3 benchmark/pvisor/product_v1.py \
  --binary /absolute/path/to/gnu-linux/pvisor \
  --firmware /absolute/path/to/libkrunfw-directory \
  --replay-binary /absolute/path/to/pvisor-replay \
  --output target/product-benchmark-new \
  --suites filesystem --samples 30 --warmups 3

Start with --samples 1 --warmups 0 to check prerequisites. Workloads and correctness assertions live in benchmark/pvisor/v1/. Reports pin binaries, firmware and harness source with hashes. Failed operations never enter performance distributions. Effective sample counts are stated per page; P95/P99 from small samples describe this batch rather than production tail probabilities.

Environment, artifacts and method · Local raw record docs/src/assets/benchmarks/.data/product-v1-20261004/manifest.tsv · Local raw record docs/src/assets/benchmarks/.data/product-v1-20261004/samples.csv · Local raw record docs/src/assets/benchmarks/.data/product-v1-20261004/evidence.tar.gz. Reports retain dirty source status; executable SHA256 identifies the measured artifact. The archive excludes large rootfs/binaries and reproducible workspace payloads, while retaining input hashes and each batch's harness.