Create storage slab read simulator script #1696

djeebus · 2026-01-08T00:33:31Z

Mostly posting this for visibility right now.

4 MB read ahead, ReadAt = 12ms-45ms

joe_lombrozo@e2b-orch-client-djwm:~$ sudo ./simulator -read-method=ReadAt -drop-nfs-cache -nfs-stat /orchestrator/shared-store/  

==== reads ====
count: 100
min: 12.357989ms
p50: 18.470138ms
p95: 42.457205ms
p99: 43.757991ms
max: 45.82411ms
stddev: 11.766231ms

==== sizes ====
count: 100
min: 4.2 MB
p50: 4.2 MB
p95: 4.2 MB
p99: 4.2 MB
max: 4.2 MB
stddev: 0 B
nfs v4 client:  total:  1025
nfs v4 client:  read:   800
nfs v4 client:  open_noat:      100
nfs v4 client:  close:  100
nfs v4 client:  access: 8
nfs v4 client:  getattr:        1
nfs v4 client:  readdir:        16

4 MB read ahead, Read = 12-45 ms

joe_lombrozo@e2b-orch-client-djwm:~$ sudo ./simulator -read-method=ReadFile -drop-nfs-cache -nfs-stat /orchestrator/shared-store/  

==== reads ====
count: 100
min: 12.964908ms
p50: 39.625818ms
p95: 43.193261ms
p99: 44.80182ms
max: 45.016427ms
stddev: 12.697036ms

==== sizes ====
count: 100
min: 4.2 MB
p50: 4.2 MB
p95: 4.2 MB
p99: 4.2 MB
max: 4.2 MB
stddev: 0 B
nfs v4 client:  total:  1025
nfs v4 client:  read:   800
nfs v4 client:  open_noat:      100
nfs v4 client:  close:  100
nfs v4 client:  access: 8
nfs v4 client:  getattr:        1
nfs v4 client:  readdir:        16

128 KB (default) read ahead, Read = 21-100+ms

joe_lombrozo@e2b-orch-client-djwm:~$ sudo ./simulator -read-method=ReadFile -drop-nfs-cache -nfs-stat /orchestrator
/shared-store/  

==== reads ====
count: 100
min: 21.670188ms
p50: 27.763614ms
p95: 55.346971ms
p99: 114.461142ms
max: 122.071469ms
stddev: 14.161518ms

==== sizes ====
count: 100
min: 4.2 MB
p50: 4.2 MB
p95: 4.2 MB
p99: 4.2 MB
max: 4.2 MB
stddev: 0 B
nfs v4 client:  total:  1032
nfs v4 client:  read:   800
nfs v4 client:  open_noat:      100
nfs v4 client:  close:  100
nfs v4 client:  access: 8
nfs v4 client:  getattr:        1
nfs v4 client:  readdir:        16
nfs v4 client:  bind_conn_to_ses:       7

128 KB (default) read ahead, ReadAt = 21-90ms

joe_lombrozo@e2b-orch-client-djwm:~$ sudo ./simulator -read-method=ReadAt -drop-nfs-cache -nfs-stat /orchestrator/s
hared-store/  

==== reads ====
count: 100
min: 20.490773ms
p50: 26.766649ms
p95: 54.697701ms
p99: 85.661541ms
max: 90.233932ms
stddev: 11.499387ms

==== sizes ====
count: 100
min: 4.2 MB
p50: 4.2 MB
p95: 4.2 MB
p99: 4.2 MB
max: 4.2 MB
stddev: 0 B
nfs v4 client:  total:  1025
nfs v4 client:  read:   800
nfs v4 client:  open_noat:      100
nfs v4 client:  close:  100
nfs v4 client:  access: 8
nfs v4 client:  getattr:        1
nfs v4 client:  readdir:        16

Note

Adds tooling to measure storage read performance and export results for analysis.

New CLIs:
- cmd/hammer-file: sequential/parallel GCS range reads with timing output and Mermaid Gantt files
- cmd/simulate-gcs-traffic: configurable GCS read simulator (experiments for concurrency, gRPC opts, chunk size, etc.), CSV export with histograms, GCE metadata, and tests
- cmd/simulate-nfs-traffic: NFS read simulator (concurrency, read method, read-ahead/sysctl tuning), optional nfsstat capture/parse, CSV export with Filestore metadata, and tests
Storage cache tweak: in storage_cache_seekable.go, read cached chunk via Read instead of ReadAt.
Build/module: go.mod updates (promote google.golang.org/api to direct dep).

^{Written by Cursor Bugbot for commit 1b5fd6e. This will update automatically on new commits. Configure here.}

packages/orchestrator/cmd/simulate-traffic/main.go

packages/shared/pkg/storage/storage_cache_seekable.go

# Conflicts: # packages/shared/pkg/storage/storage_cache_seekable.go

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d5bb30a5d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-01-14T19:29:02Z

packages/orchestrator/cmd/hammer-file/main.go

+		for chunk := range fileSize / chunkSize {
+			offset := chunk * chunkSize


Read final partial chunk in parallel loop

The parallel read loop iterates for chunk := range fileSize / chunkSize, which truncates the chunk count. When the object size is not an exact multiple of chunkSize, the last partial chunk is never scheduled, so those tail bytes are skipped entirely and the timing stats for the parallel scenario underreport total work. This only shows up for non‑aligned object sizes; using a ceiling division (e.g., (fileSize+chunkSize-1)/chunkSize) or an explicit remainder check would ensure the final chunk is read.

Useful? React with 👍 / 👎.

This is intentional

packages/orchestrator/cmd/simulate-nfs-traffic/experiments.go

packages/orchestrator/cmd/simulate-gcs-traffic/files.go

packages/orchestrator/cmd/hammer-file/main.go

cursor · 2026-01-14T20:02:43Z

packages/orchestrator/cmd/simulate-gcs-traffic/files.go

+	const totalAttempts = 10
+	totalChunks := info.size / f.chunkSize
+	for range totalAttempts {
+		offset := f.rand.Int63n(totalChunks-1) * f.chunkSize // the last one might not be full, just skip it


Panic when file has only one chunk

Low Severity

The nextRead function computes totalChunks := info.size / f.chunkSize and then calls f.rand.Int63n(totalChunks-1). If a file has only one chunk (size between chunkSize and 2*chunkSize), totalChunks-1 becomes 0, and rand.Int63n(0) panics since it requires n > 0. While default parameters (100MB min file size, 4MB chunks) make this unlikely, changing flags or edge-case file sizes could trigger the panic.

cursor · 2026-01-14T20:02:43Z

packages/orchestrator/cmd/simulate-nfs-traffic/experiments.go

+		}
+	}
+
+	return errors.Join(errs...)


System settings not restored if setup partially fails

Low Severity

The scenario.setup method continues processing all experiments even when one fails, collecting errors instead of stopping. If an early experiment like setReadAhead or setSysFs succeeds (modifying kernel parameters), but a later experiment fails, the function returns an error and the caller returns early without calling teardown. This leaves system settings (like /sys/class/bdi/.../read_ahead_kb or sysctl values) permanently modified until manual cleanup.

Additional Locations (1)

packages/orchestrator/cmd/simulate-nfs-traffic/main.go#L209-L212

add storage simulator script

35c5d3f

djeebus changed the title ~~add storage simulator script~~ Create storage slab read simulator script Jan 8, 2026

e2b-request-same-site-reviewers bot assigned ValentaTomas Jan 8, 2026

claude bot reviewed Jan 8, 2026

View reviewed changes

packages/orchestrator/cmd/simulate-traffic/main.go Outdated Show resolved Hide resolved

claude bot reviewed Jan 8, 2026

View reviewed changes

packages/orchestrator/cmd/simulate-traffic/main.go Outdated Show resolved Hide resolved

claude bot reviewed Jan 8, 2026

View reviewed changes

packages/orchestrator/cmd/simulate-traffic/main.go Outdated Show resolved Hide resolved

claude bot reviewed Jan 8, 2026

View reviewed changes

packages/shared/pkg/storage/storage_cache_seekable.go Show resolved Hide resolved

djeebus added 23 commits January 7, 2026 16:40

tweaks, lint

97a6b25

progress towards generating experiments

ca297df

add experiments so we can test permutations

beb98e8

linting

a568334

gather files once

dea2069

perform more science

2caf169

linting

8948dc6

split it into files, add a little more metadata

fa86852

linting

0b678df

add a gcs test

e0987ea

more science tweaks

364687f

docs and more experiments

f902519

clean up the science

29f6415

tidy

672164b

linting

b6fb645

Merge branch 'main' into add-storage-simulator

d3f02d5

linting

c7602ee

fix the factory :(

7abbecb

only render experiments with choices

b182b4d

add 'repeat' parameter

a32663c

add chunk size experiment

7875cec

test increased chunk size

bfb2314

add lots more experiments

1d4ea19

djeebus added 3 commits January 12, 2026 15:47

linting

02040ab

read file in parallel

acee8f7

Merge remote-tracking branch 'origin/main' into add-storage-simulator

9d5bb30

# Conflicts: # packages/shared/pkg/storage/storage_cache_seekable.go

djeebus marked this pull request as ready for review January 14, 2026 19:26

djeebus requested review from ValentaTomas, dobrac and jakubno as code owners January 14, 2026 19:26

chatgpt-codex-connector bot reviewed Jan 14, 2026

View reviewed changes

cursor bot reviewed Jan 14, 2026

View reviewed changes

packages/orchestrator/cmd/simulate-nfs-traffic/experiments.go Show resolved Hide resolved

packages/orchestrator/cmd/simulate-gcs-traffic/files.go Show resolved Hide resolved

packages/orchestrator/cmd/hammer-file/main.go Show resolved Hide resolved

code review comments

1b5fd6e

cursor bot reviewed Jan 14, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create storage slab read simulator script #1696

Create storage slab read simulator script #1696

Uh oh!

djeebus commented Jan 8, 2026 •

edited by cursor bot

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Jan 14, 2026

Uh oh!

djeebus Jan 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot Jan 14, 2026

Uh oh!

cursor bot Jan 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		for chunk := range fileSize / chunkSize {
		offset := chunk * chunkSize

Create storage slab read simulator script #1696

Are you sure you want to change the base?

Create storage slab read simulator script #1696

Uh oh!

Conversation

djeebus commented Jan 8, 2026 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

djeebus Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot Jan 14, 2026

Choose a reason for hiding this comment

Panic when file has only one chunk

Uh oh!

cursor bot Jan 14, 2026

Choose a reason for hiding this comment

System settings not restored if setup partially fails

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

djeebus commented Jan 8, 2026 •

edited by cursor bot

Loading