Worker SIGSEGV since v0.8.7-v8

[netsgnut]

Hi, great project you have here! Not sure where to report this issue, so I am reporting it to where I see the SIGSEGV.

I am setting up OpenWorkers in a homelab and using it to run smaller tasks that I have lying around. My environment is set up using Podman Compose from openworkers-infra, with paths modified to point to the local domain names.

What I see is that since v0.8.7-v8, nginx will return 502 errors after doing exactly 3 requests, since the runners will just exit with error 139.

openworkers-proxy logs:

2026-01-24T22:21:01.079Z a575d6b28de112d6fb1d08dcbf261970 worker-engine-proxy [abc-worker.lab.example.com] abc 10.89.13.12  	 GET "/some-path" 200 0.066
2026-01-24T22:21:02.235Z a4819cf6151d17bcc1c3a089ba683e15 worker-engine-proxy [abc-worker.lab.example.com] abc 10.89.13.12  	 GET "/some-path" 200 0.063
2026-01-24T22:21:02.865Z b596fb35bb2dcfdd1f2080fc43dad968 worker-engine-proxy [abc-worker.lab.example.com] abc 10.89.13.12  	 GET "/some-path" 200 0.087
2026/01/24 22:21:03 [error] 2#2: *10 upstream prematurely closed connection while reading response header from upstream, client: 10.89.13.12, server: ~\.lab\.example\.com$, request: "GET /some-path HTTP/1.1", upstream: "http://10.89.13.7:8080/some-path", host: "abc-worker.lab.example.com"
2026/01/24 22:21:03 [warn] 2#2: *10 upstream server temporarily disabled while reading response header from upstream, client: 10.89.13.12, server: ~\.lab\.example\.com$, request: "GET /some-path HTTP/1.1", upstream: "http://10.89.13.7:8080/some-path", host: "abc-worker.lab.example.com"

(Notice that the worker URL scheme is different because I have changed them in the nginx config, but it should be irrelevant to the issue.)

A runner's log looks like this from its start to its exit:

[2026-01-24T22:38:43Z DEBUG openworkers_runner] start main (hyper)
[2026-01-24T22:38:43Z DEBUG openworkers_runner] Isolate pool config: max_size=1000, heap_initial=10MB, heap_max=50MB
[2026-01-24T22:38:43Z DEBUG openworkers_runner] connected to Postgres
[2026-01-24T22:38:43Z DEBUG openworkers_runner::nats] connecting to nats: nats://nats:4222
[2026-01-24T22:38:43Z DEBUG openworkers_runner] connected to NATS
[2026-01-24T22:38:43Z DEBUG openworkers_runner] started log publisher
[2026-01-24T22:38:43Z INFO  openworkers_runtime_v8::thread_pinned_pool] Thread-pinned pool initialized: max_per_thread=1000, max_per_owner=None, queue_size=10, queue_timeout_ms=5000
[2026-01-24T22:38:43Z DEBUG openworkers_runner] Initialized thread-pinned isolate pool with 1000 isolates per thread
Listening on http://0.0.0.0:8080 with 4 listeners
[2026-01-24T22:38:43Z DEBUG openworkers_runner::nats] connecting to nats: nats://nats:4222
[2026-01-24T22:38:43Z DEBUG openworkers_runner] Listener 0 started
[2026-01-24T22:38:43Z DEBUG openworkers_runner] Listener 1 started
[2026-01-24T22:38:43Z DEBUG openworkers_runner] Listener 3 started
[2026-01-24T22:38:43Z DEBUG openworkers_runner] Listener 2 started
[2026-01-24T22:38:43Z DEBUG openworkers_runner::event_scheduled] subscribing to scheduled JoinHandle { id: Id(17) }
[2026-01-24T22:38:43Z DEBUG openworkers_runner::nats] connecting to nats: nats://nats:4222
[2026-01-24T22:38:43Z INFO  openworkers_runner::log] Global log publisher started (batching: 10msg/100ms)
[2026-01-24T22:38:43Z DEBUG openworkers_runner::event_scheduled] listening for scheduled tasks
[2026-01-24T22:39:23Z DEBUG openworkers_runner] handle_request: GET /some-path in thread ThreadId(4)
[2026-01-24T22:39:24Z DEBUG openworkers_runner] host: Some("abc-worker.lab.example.com"), worker_id: None, worker_name: Some("abc")
[2026-01-24T22:39:24Z DEBUG openworkers_runner] request_id: b82ada08e80651965f7a60bb4236787f, worker_id: None, worker_name: Some("abc")
[2026-01-24T22:39:24Z DEBUG openworkers_runner::store] get_worker_with_bindings: Name("abc")
[2026-01-24T22:39:24Z DEBUG openworkers_runner::store] worker found: id: 21d5d010, version: 76, bindings: 0, type: typescript
[2026-01-24T22:39:24Z DEBUG openworkers_runner] worker found: true
[2026-01-24T22:39:24Z INFO  openworkers_runner::worker_pool] Initializing worker semaphore with 160 max queued workers
[2026-01-24T22:39:24Z DEBUG openworkers_runner::worker] transpile cache MISS: worker=21d5d010, version=76
[2026-01-24T22:39:24Z DEBUG openworkers_runner::worker] transpile cache HIT: worker=21d5d010, version=76
[2026-01-24T22:39:24Z INFO  openworkers_runner::worker_pool] Sequential worker pool initialized with 16 threads
[2026-01-24T22:39:24Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Thread-local pool initialized on thread ThreadId(13) (max_per_owner=None, queue_size=10)
[2026-01-24T22:39:24Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Cache MISS: creating new isolate for owner c88f9ecf-cc16-4b34-9006-f3b15346d2f8 (pool: 1/1000, owner: 1/None)
[2026-01-24T22:39:24Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Creating new TaggedIsolate for owner: c88f9ecf-cc16-4b34-9006-f3b15346d2f8
[2026-01-24T22:39:24Z INFO  openworkers_runtime_v8::thread_pinned_pool] TaggedIsolate created for owner c88f9ecf-cc16-4b34-9006-f3b15346d2f8 in 52.192604ms (snapshot: false)
[2026-01-24T22:39:24Z DEBUG openworkers_runner] handle_request done in 57ms
[2026-01-24T22:39:24Z DEBUG openworkers_runner::log] Flushing logs for worker: 21d5d010
[2026-01-24T22:39:34Z DEBUG openworkers_runner] handle_request: GET /some-path in thread ThreadId(4)
[2026-01-24T22:39:34Z DEBUG openworkers_runner] host: Some("abc-worker.lab.example.com"), worker_id: None, worker_name: Some("abc")
[2026-01-24T22:39:34Z DEBUG openworkers_runner] request_id: 09720dc84bdab9d2c928c72a45e6d65b, worker_id: None, worker_name: Some("abc")
[2026-01-24T22:39:34Z DEBUG openworkers_runner::store] get_worker_with_bindings: Name("abc")
[2026-01-24T22:39:34Z DEBUG openworkers_runner::store] worker found: id: 21d5d010, version: 76, bindings: 0, type: typescript
[2026-01-24T22:39:34Z DEBUG openworkers_runner] worker found: true
[2026-01-24T22:39:34Z DEBUG openworkers_runner::worker] transpile cache HIT: worker=21d5d010, version=76
[2026-01-24T22:39:34Z DEBUG openworkers_runner::worker] transpile cache HIT: worker=21d5d010, version=76
[2026-01-24T22:39:34Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Thread-local pool initialized on thread ThreadId(14) (max_per_owner=None, queue_size=10)
[2026-01-24T22:39:34Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Cache MISS: creating new isolate for owner c88f9ecf-cc16-4b34-9006-f3b15346d2f8 (pool: 1/1000, owner: 1/None)
[2026-01-24T22:39:34Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Creating new TaggedIsolate for owner: c88f9ecf-cc16-4b34-9006-f3b15346d2f8

I bisected versions, and I find that the last good version that works for me is openworkers-runner:v0.8.5-v8, v0.8.7-v8 and beyond exhibit the behaviour outlined above. Other package versions don't seem to matter in this case, and the schema has been updated to the latest.

Any pointers?

[max-lt]

Hi, thanks for trying out the project and reporting this! 🙏

Looking at your logs, I notice snapshot: false this means no V8 snapshot is being loaded, which could explain the instability.

Also, you're on v0.8.7-v8 which is a few versions behind. We've fixed several snapshot-related issues since then. Could you try upgrading to the latest version v0.11.3-v8 (currently in production) and see if the issue persists?

If you're not using Docker, make sure to regenerate the snapshot after upgrading:
cargo run --features v8 --bin snapshot (The Docker image already includes the snapshot)

The "3 requests then crash" pattern is interesting - it might be related to retry logic if your worker is returning errors. But regardless, it shouldn't cause a SIGSEGV, so let us know if it still happens on the latest version.

If the issue persists on the latest version, could you share more details?

• What does your worker code look like?
• Are you using any specific feature flags?

We haven't seen this behavior in production, so additional context would help us investigate.

Also, for a homelab setup running smaller tasks, you might be interested in https://github.com/openworkers/openworkers-task-executor - it's a simpler standalone executor that doesn't require the full stack. Could be a good fit for your use case!

[netsgnut]

Hi @max-lt, thanks for writing back!

I tried the latest v0.11.3-v8 and got the same error with Podman. The SIGSEGV persists, and the last error printed is exactly the same as before: Creating new TaggedIsolate for owner.

(In case it might be useful, I am running Podman v5.4.2 on Debian 13/trixie.)

I use the compose.yml found in openworkers-infra, which I believe would mean I don't need to run the snapshot command manually, so it is puzzling why it happens pretty reliably on my end seemingly for all versions beyond v0.8.5-v8.

Regarding your questions:

• I believe my worker code may not be the problem, because I can reproduce the same issue with the default TypeScript worker template.
• I don't have any feature flags.

Thanks for the pointer to openworkers-task-executor! It can definitely suit my use case, but I still prefer to manage with a GUI interface to replace my Node.js/Deno/Python scripts that are scattered all over the place :)

Edit: Looking at the code change between v0.8.5 and v0.8.7, I've a feeling that it might be related to the then-new src/task_executor.rs. But again it seems pretty weird that no actual log is found just before the SIGSEGV occurs.

[max-lt]

Hey @netsgnut, thanks for the detailed info!

I dug into the code and the crash likely happens in LockerManagedIsolate::new() when creating the V8 isolate. The key change between v0.8.5 and v0.8.7 is the introduction of v8::Locker for the isolate pool - we moved from OwnedIsolate (auto-enter) to UnenteredIsolate + explicit Locker.

My suspicion is that this may happen because Podman's default seccomp profile is more restrictive than Docker's and blocks some syscalls that V8/Locker uses internally (like specific mmap flags, futex, or memory guard pages).

Could you try running with seccomp disabled to confirm?

podman run --security-opt seccomp=unconfined ...                                                                                                                                                                                                                                                       

If that works, it's definitely a seccomp issue and we can try to identify which syscall is being blocked.

If it still crashes, let me know and I'll try to reproduce on my end with Podman.

[max-lt]

[claude-code] Summary: SIGSEGV Investigation (v0.8.7+ with Podman)

What Changed Between v0.8.5 and v0.8.7

In v0.8.6, isolate management was refactored from OwnedIsolate to UnenteredIsolate + v8::Locker for better multi-threading support.

Before v0.8.6 (OwnedIsolate):

1. v8::Isolate::New()     → Heap allocated                                                                                                                                                                                                                                                             
2. isolate.enter()        → Isolate::Enter() called                                                                                                                                                                                                                                                    
3. (No Locker at creation time)                                                                                                                                                                                                                                                                        

After v0.8.6 (UnenteredIsolate + Locker) - in locker_managed_isolate.rs:86-93:

let mut isolate = v8::Isolate::new_unentered(params);  // Step 1                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                       
let heap_limit_state = {                                                                                                                                                                                                                                                                               
    let mut locker = v8::Locker::new(&mut isolate);  // Step 2                                                                                                                                                                                                                                         
    install_heap_limit_callback(...)                                                                                                                                                                                                                                                                   
};                                                                                                                                                                                                                                                                                                     

Where Locker::new() in rusty_v8 (src/isolate.rs:2482-2493) does:

pub fn new(isolate: &'a mut UnenteredIsolate) -> Self {                                                                                                                                                                                                                                                
    unsafe { v8__Isolate__Enter(isolate_ptr.as_ptr()); }  // 2a. Enter first                                                                                                                                                                                                                           
    raw.init(isolate_ptr);  // 2b. Then Locker::Initialize()                                                                                                                                                                                                                                           
}                                                                                                                                                                                                                                                                                                      

The Key Difference

With the new architecture, Locker::Initialize() triggers a call path that didn't exist with OwnedIsolate:

V8 C++ code path (v8/src/execution/v8threads.cc):

Locker::Initialize()                                                                                                                                                                                                                                                                                   
  → thread_manager()->Lock()                                                                                                                                                                                                                                                                           
  → thread_manager()->RestoreThread()                                                                                                                                                                                                                                                                  
    → InitThread() (for new threads)                                                                                                                                                                                                                                                                   
      → isolate_->InitializeThreadLocal()                                                                                                                                                                                                                                                              
      → isolate_->stack_guard()->InitThread()  ← Key difference                                                                                                                                                                                                                                        
      → isolate_->debug()->InitThread()                                                                                                                                                                                                                                                                

The stack_guard()->InitThread() calls ThreadLocal::Initialize() (stack-guard.cc:247-260):

void StackGuard::ThreadLocal::Initialize(Isolate* isolate, ...) {                                                                                                                                                                                                                                      
  const uintptr_t kLimitSize = v8_flags.stack_size * KB;                                                                                                                                                                                                                                               
  DCHECK_GT(base::Stack::GetStackStart(), kLimitSize);                                                                                                                                                                                                                                                 
  uintptr_t limit = base::Stack::GetStackStart() - kLimitSize;                                                                                                                                                                                                                                         
  real_jslimit_ = SimulatorStack::JsLimitFromCLimit(isolate, limit);                                                                                                                                                                                                                                   
  // ...                                                                                                                                                                                                                                                                                               
}                                                                                                                                                                                                                                                                                                      

And GetStackStart() on Linux (platform-posix.cc:1456-1473) uses:

int error = pthread_getattr_np(pthread_self(), &attr);                                                                                                                                                                                                                                                 
if (error) {                                                                                                                                                                                                                                                                                           
  // Falls back to nullptr or __libc_stack_end                                                                                                                                                                                                                                                         
}                                                                                                                                                                                                                                                                                                      
error = pthread_attr_getstack(&attr, &base, &size);                                                                                                                                                                                                                                                    

Potential Root Cause

Under Podman's seccomp profile (especially in rootless mode), pthread_getattr_np() may fail or return incorrect values. If GetStackStart() returns 0 (nullptr):

1. limit = 0 - kLimitSize wraps around to a very large value (~0xFFFFFFFFFFFF...)
2. real_jslimit_ gets set to this invalid value
3. Later stack limit checks or memory accesses cause SIGSEGV

With OwnedIsolate, this code path wasn't executed during isolate creation, so the issue didn't manifest.

Crash Location

Based on the logs ("Creating new TaggedIsolate for owner"), the crash occurs in thread_pinned_pool.rs:177:

let isolate = LockerManagedIsolate::new(limits);                                                                                                                                                                                                                                                       

Which internally calls:

v8::Isolate::new_unentered(params)  ← Creates isolate                                                                                                                                                                                                                                                  
v8::Locker::new(&mut isolate)       ← Triggers InitThread() → potential crash                                                                                                                                                                                                                          

Suggested Workarounds to Try

1. Run Podman with --security-opt seccomp=unconfined to test if seccomp is the cause
2. If using rootless Podman, try rootful mode
3. Check if pthread_getattr_np() works correctly in the container

Note

This analysis is based on source code reading. The exact syscall that fails would need to be confirmed with strace or similar tooling on the affected system.

---

Known Issues: This is a Well-Documented Problem

Similar Cases in the Wild

This issue has been encountered by multiple projects using V8 or thread creation in containers:

Project	Issue	Symptoms
Deno	#23229	Check failed: Start() in DefaultWorkerThreadsTaskRunner
Supabase Edge Runtime	cli#1780	Same Check failed: Start() crash
JAX/LLVM	#24631	pthread_create failed in Podman rootless
Java/JVM	temurin#2976	pthread_create failed (EPERM)
Go apps	skopeo#1501	runtime/cgo: pthread_create failed: Operation not permitted

Root Cause: clone3 + seccomp + glibc 2.34+

Starting with glibc 2.34 (Ubuntu 21.10+, Fedora 35+), pthread_create() uses the clone3 syscall with fallback to clone:

• clone3 returns ENOSYS → glibc falls back to clone ✓
• clone3 returns EPERM → glibc treats this as fatal ✗

Older container runtimes' seccomp profiles returned EPERM for unknown syscalls instead of ENOSYS.

Podman Fix Status

Podman uses seccomp profiles from containers/common. The clone3 syscall has been added to the allowed list, so modern Podman versions should work.

However, the issue containers/podman#11862 notes that Podman historically did not have the generic ENOSYS fallback fix that Docker/runc implemented — it needed explicit updates for each new syscall.

To diagnose, ask the user for:

podman --version
podman info | grep -A10 ociRuntime
cat /usr/share/containers/seccomp.json | grep clone3

If clone3 is missing from the seccomp profile, updating Podman should fix it.

If clone3 is present, the issue might be specific to rootless mode (user namespaces, cgroups v1 vs v2, etc.) rather than seccomp.

---

Suggested Workarounds

For Users

1. Update Podman (recommended)

   podman --version  # Check current version

2. Run with seccomp=unconfined (less secure, for testing)

   podman run --security-opt seccomp=unconfined ...

3. Use rootful mode instead of rootless

   sudo podman run ...

4. Check container runtime versions

   podman info | grep -A5 ociRuntime
   runc --version  # or crun --version

Diagnostic Steps

1. Check if clone3 is in seccomp profile:

   cat /usr/share/containers/seccomp.json | grep clone3

2. Check if clone3 is the issue with strace:

   strace -f ./openworkers 2>&1 | grep clone3

   If you see clone3(...) = -1 EPERM, it's the seccomp issue.

3. Verify pthread_getattr_np() works in the container:

   // test_pthread.c
   #include <pthread.h>
   #include <stdio.h>
   int main() {
       pthread_attr_t attr;
       void *base; size_t size;
       int err = pthread_getattr_np(pthread_self(), &attr);
       printf("pthread_getattr_np: %d\n", err);
       if (!err) {
           pthread_attr_getstack(&attr, &base, &size);
           printf("stack base: %p, size: %zu\n", base, size);
       }
       return err;
   }

   gcc -o test_pthread test_pthread.c -lpthread
   ./test_pthread

---

Potential Fixes for OpenWorkers

Short-term: Documentation

Add to documentation/README:

### Container Runtime Requirements

When running OpenWorkers in containers, ensure your container runtime 
supports the `clone3` syscall properly:

- **Podman**: Recent version with updated seccomp profile
- **Docker**: Version 20.10.10 or later

If you encounter SIGSEGV crashes on startup, try:
1. Updating your container runtime
2. Running with `--security-opt seccomp=unconfined` (for testing only)
3. Using rootful mode instead of rootless

Medium-term: Defensive Code

Consider adding a runtime check before V8 initialization:

fn check_thread_creation_works() -> Result {
    // Attempt to spawn a test thread to catch seccomp issues early
    let handle = std::thread::Builder::new()
        .name("seccomp-test".into())
        .spawn(|| {})?;
    handle.join().map_err(|_| Error::ThreadCreationFailed)?;
    Ok(())
}

Long-term: Upstream Engagement

Consider filing issues or contributing to:

• rusty_v8: Better error handling when thread creation fails
• V8: More graceful degradation when GetStackStart() fails

---

References

• containers/podman#11862 - Podman: seccomp is missing clone3
• moby/moby#42680 - seccomp filter breaks glibc by blocking clone3 with EPERM
• moby/moby#42681 - seccomp: add support for "clone3" syscall
• denoland/deno#23229 - Deno crashes on startup within container
• supabase/cli#1780 - supabase start fails with Check failed: Start()
• jax-ml/jax#24631 - pthread_create failed in Podman rootless
• Containers using glibc v2.34+ don't work on older runtimes

[netsgnut]

Thanks! I tried running openworkers-runner:v0.11.3-v8 with:

    security_opt:
      - seccomp:unconfined
      - apparmor:unconfined

while keeping other services as-is. I still can reproduce the same issue, unfortunately.

I also tried to run it with root. Same segfault.

I forgot to attach this before, but here is the dmesg output for your reference when it segfaults:

[740590.642187] v8-worker-1[784145]: segfault at 7fa061610010 ip 000055c0f71b914d sp 00007fa07ecee130 error 25 in openworkers-runner[dbc14d,55c0f65c5000+202c000] likely on CPU 1 (core 1, socket 0)
[740590.642229] Code: 35 5b e5 03 02 48 8d 15 69 62 09 02 31 ff e8 ca 5a b6 ff 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 8b 07 c1 ea 04 83 e2 f0 <48> 8b 04 10 48 c1 e8 30 3d ff ff 00 00 0f 93 c0 5d c3 90 55 48 89
[740590.650159] v8-worker-1[784239]: segfault at 7efc33848010 ip 000055664beaa14d sp 00007efc50f24130 error 25 in openworkers-runner[dbc14d,55664b2b6000+202c000] likely on CPU 1 (core 1, socket 0)
[740590.650165] Code: 35 5b e5 03 02 48 8d 15 69 62 09 02 31 ff e8 ca 5a b6 ff 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 8b 07 c1 ea 04 83 e2 f0 <48> 8b 04 10 48 c1 e8 30 3d ff ff 00 00 0f 93 c0 5d c3 90 55 48 89
[740590.653445] v8-worker-1[784333]: segfault at 7faf90570010 ip 000055cfa030514d sp 00007fafadc59130 error 25 in openworkers-runner[dbc14d,55cf9f711000+202c000] likely on CPU 11 (core 3, socket 0)
[740590.653453] Code: 35 5b e5 03 02 48 8d 15 69 62 09 02 31 ff e8 ca 5a b6 ff 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 8b 07 c1 ea 04 83 e2 f0 <48> 8b 04 10 48 c1 e8 30 3d ff ff 00 00 0f 93 c0 5d c3 90 55 48 89

Since Claude asked about the ociRuntime, here it is from podman info:

  ociRuntime:
    name: crun
    package: crun_1.21-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.21
      commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +WASM:wasmedge +YAJL

[max-lt]

Hi, very interesting, can do you reproduce quickly it with task-executor ?

// filename:  /tmp/test.js

// Async task handler
addEventListener('task', async (event) => {
    const response = await fetch('https://httpbin.workers.rocks/get');
    const data = await response.json();

    event.respondWith({
        success: true,
        data: data
    });
});

podman run --rm -v /tmp/test.js:/tmp/test.js:ro ghcr.io/openworkers/openworkers-task-executor:v0.11.1 run /tmp/test.js

For information it works on my end

[max-lt]

Are you running on a VM or bare metal?

If you're on a VM, that could explain the issue — V8 can have problems with certain hypervisors (memory mapping, CPU features, etc.).

Can you check?

systemd-detect-virt

[max-lt]

Successfully reproduced on bare metal with Podman

I deployed the full stack using Podman (rootless) on bare metal and everything works fine.

Environment

• Ubuntu 24.04.3 LTS
• Podman 4.9.3 (rootless)
• podman-compose 1.0.6
• runc 1.3.4
• Bare metal server (systemd-detect-virt = none)

Step-by-step reproduction

1. Clone the infra repo

git clone https://github.com/openworkers/openworkers-infra.git
cd openworkers-infra

2. Configure environment

cp .env.example .env
nano .env
# Set POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB, JWT secrets, etc.

3. Configure Podman registries (required for short image names)

echo 'unqualified-search-registries = ["docker.io"]' | sudo tee -a /etc/containers/registries.conf

4. Start PostgreSQL

podman-compose up -d postgres

5. Clone CLI and run migrations

git clone https://github.com/openworkers/openworkers-cli.git

for f in openworkers-cli/migrations/*.sql; do
  echo "Applying $f..."
  podman exec -i openworkers-infra_postgres_1 psql -U openworkers -d openworkers < "$f"
done

6. Start Postgate and generate API token

podman-compose up -d postgate

podman exec openworkers-infra_postgate_1 postgate gen-token \
  aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa api \
  --permissions SELECT,INSERT,UPDATE,DELETE

Output:

pg_5ee54f11baaacdcee862c0aab01a1c7f32af6bc86a05c95f8f182c217fa48b12

Add to .env:

sed -i 's/^POSTGATE_TOKEN=.*/POSTGATE_TOKEN=pg_5ee54f11baaacdcee862c0aab01a1c7f32af6bc86a05c95f8f182c217fa48b12/' .env

7. Expose PostgreSQL port for CLI (added to compose.dev.yml)

services:
  postgres:
    ports:
      - "5432:5432"

8. Start the full stack

podman-compose -f compose.yml -f compose.dev.yml up -d

9. Verify all containers are running

podman ps -a --filter label=io.podman.compose.project=openworkers-infra

Output:

CONTAINER ID  IMAGE                                              STATUS                   NAMES
ddddbc84491e  docker.io/library/postgres:15.1-alpine3.17         Up 29 seconds (healthy)  openworkers-infra_postgres_1
f1d012962638  docker.io/library/nats:2.12.2-alpine3.22           Up 28 seconds (healthy)  openworkers-infra_nats_1
72ee4b5ea671  ghcr.io/openworkers/openworkers-dash:v1.4.1        Up 26 seconds (healthy)  openworkers-infra_openworkers-dash_1
337f5a6a5e4a  ghcr.io/openworkers/postgate:v0.1.3                Up 25 seconds            openworkers-infra_postgate_1
28f467eb2fd7  ghcr.io/openworkers/openworkers-api:v1.4.5         Up 24 seconds (healthy)  openworkers-infra_openworkers-api_1
fe4e5a582549  ghcr.io/openworkers/openworkers-runner:v0.11.5-v8  Up 23 seconds (healthy)  openworkers-infra_openworkers-runner_1
3f410f5ba20a  ghcr.io/openworkers/openworkers-runner:v0.11.5-v8  Up 21 seconds (healthy)  openworkers-infra_openworkers-runner_2
866943a2e4c2  ghcr.io/openworkers/openworkers-runner:v0.11.5-v8  Up 20 seconds (healthy)  openworkers-infra_openworkers-runner_3
4bfc5d33e44d  ghcr.io/openworkers/openworkers-logs:v0.1.4        Up 19 seconds            openworkers-infra_openworkers-logs_1
58ef3117c5ea  ghcr.io/openworkers/openworkers-scheduler:v0.1.3   Up 17 seconds            openworkers-infra_openworkers-scheduler_1

All 3 runners are healthy ✅

10. Check runner logs

podman logs openworkers-infra_openworkers-runner_1

Output:

[2026-01-31T11:41:38Z DEBUG openworkers_runner] start main (hyper)
[2026-01-31T11:41:38Z DEBUG openworkers_runner] Isolate pool config: max_size=1000, heap_initial=10MB, heap_max=50MB, wall_clock_timeout=64000ms, cpu_time_limit=100ms
[2026-01-31T11:41:38Z DEBUG openworkers_runner] connected to Postgres
[2026-01-31T11:41:38Z DEBUG openworkers_runner::nats] connecting to nats: nats://nats:4222
[2026-01-31T11:41:38Z DEBUG openworkers_runner] connected to NATS
[2026-01-31T11:41:38Z DEBUG openworkers_runner] started log publisher
[2026-01-31T11:41:38Z INFO  openworkers_runtime_v8::thread_pinned_pool] Thread-pinned pool initialized: max_per_thread=1000, max_per_owner=None, queue_size=10, queue_timeout_ms=5000
[2026-01-31T11:41:38Z DEBUG openworkers_runner] Initialized thread-pinned isolate pool with 1000 isolates per thread
Listening on http://0.0.0.0:8080 with 4 listeners

No crash, V8 isolates initialized successfully ✅

11. Install and configure the CLI

cd openworkers-cli
cargo install --path .

Create a local alias pointing to the database:

ow alias set local --db postgres://openworkers:your-password@localhost:5432/openworkers
ow alias set-default local

12. Create a user in the database

podman exec -it openworkers-infra_postgres_1 psql -U openworkers -d openworkers

INSERT INTO users (username) VALUES ('max');
\q

Update the alias with the user:

ow alias set local --db postgres://openworkers:your-password@localhost:5432/openworkers --user max --force

13. Create and deploy a test worker

ow workers create test-worker

Output:

Created Worker 'test-worker' created.

Name:        test-worker
ID:          1cd72c2d-2aa0-4022-9dfc-a4427ef499d2
Version:     none
Created:     2026-01-31 12:02:35
Updated:     2026-01-31 12:02:35

Create a simple worker script:

cat > /tmp/default-worker.ts << 'EOF'
addEventListener('fetch', (event) => {
  event.respondWith(new Response('<h3>Hello world!</h3>', {
    headers: { 'Content-Type': 'text/html' }
  }));
});
EOF

Deploy:

ow workers deploy test-worker /tmp/default-worker.ts

Verify:

ow workers list

Output:

Workers
────────────────────────────────────────────────────────────
  test-worker                    v1         

14. Test the worker

Execute a request from inside the runner container:

podman exec openworkers-infra_openworkers-runner_1 \
  wget -qO- --header="x-worker-name: test-worker" \
            --header="x-request-id: test-123" \
            http://127.0.0.1:8080

Output:

<h3>Hello world!</h3>

Worker executed successfully! ✅

---

Conclusion

The full OpenWorkers stack works perfectly with Podman rootless on bare metal:

• All 3 runners start and stay healthy
• V8 isolates initialize without any SIGSEGV
• Workers can be deployed and executed

The issue appears to be specific to your environment.

Can you please confirm:

1. Are you running on a VM?

   systemd-detect-virt

2. If yes, which hypervisor/cloud provider?

   cat /sys/class/dmi/id/product_name

V8 can have issues with certain VM configurations (memory mapping, CPU feature emulation, nested virtualization, etc.). This would explain why it works on my bare metal setup but not on yours.

Also, can you try the task-executor test I mentioned earlier to see if the issue is specific to openworkers-runner or affects all V8 usage?

cat > /tmp/test.js << 'EOF'
addEventListener('task', async (event) => {
    const response = await fetch('https://httpbin.org/get');
    const data = await response.json();
    event.respondWith({
        success: true,
        data: data
    });
});
EOF

podman run --rm -v /tmp/test.js:/tmp/test.js:ro ghcr.io/openworkers/openworkers-task-executor:v0.11.1 run /tmp/test.js

[netsgnut]

Thanks for the instructions and trying to reproduce on your end. I can successfully run openworkers-task-executor:v0.11.1 and use that to fetch from httpbin worker. It is as expected, I get a JSON, just as I would expect when I run curl directly.

The machine I am running on is a bare-metal instead of a VM (in case this is useful, it is an HP mini PC with Xeon W-1270 and 64GB RAM, kernel 6.12.57+deb13-amd64).

user:~$ systemd-detect-virt
none

I notice that when you reproduce the problem using Ubuntu 24.04 LTS, the default Podman version shipped is 4.9, and yet on Debian 13, the Podman version is 5.4. Of course the kernel version is different as well (6.8 vs 6.12), but the Podman version can make a bigger difference I reckon.

[netsgnut]

Also, I might want to say that from the logs, it is not that the TaggedIsolate here never works for v0.8.7-v8+. Rather, it is that for each of the runners, it only works for the first time:

[2026-01-24T22:39:24Z INFO  openworkers_runner::worker_pool] Initializing worker semaphore with 160 max queued workers
[2026-01-24T22:39:24Z DEBUG openworkers_runner::worker] transpile cache MISS: worker=21d5d010, version=76
[2026-01-24T22:39:24Z DEBUG openworkers_runner::worker] transpile cache HIT: worker=21d5d010, version=76
[2026-01-24T22:39:24Z INFO  openworkers_runner::worker_pool] Sequential worker pool initialized with 16 threads
[2026-01-24T22:39:24Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Thread-local pool initialized on thread ThreadId(13) (max_per_owner=None, queue_size=10)
[2026-01-24T22:39:24Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Cache MISS: creating new isolate for owner c88f9ecf-cc16-4b34-9006-f3b15346d2f8 (pool: 1/1000, owner: 1/None)
[2026-01-24T22:39:24Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Creating new TaggedIsolate for owner: c88f9ecf-cc16-4b34-9006-f3b15346d2f8
[2026-01-24T22:39:24Z INFO  openworkers_runtime_v8::thread_pinned_pool] TaggedIsolate created for owner c88f9ecf-cc16-4b34-9006-f3b15346d2f8 in 52.192604ms (snapshot: false)

Subsequent requests fail (segfault at the end):

[2026-01-24T22:39:34Z DEBUG openworkers_runner::store] worker found: id: 21d5d010, version: 76, bindings: 0, type: typescript
[2026-01-24T22:39:34Z DEBUG openworkers_runner] worker found: true
[2026-01-24T22:39:34Z DEBUG openworkers_runner::worker] transpile cache HIT: worker=21d5d010, version=76
[2026-01-24T22:39:34Z DEBUG openworkers_runner::worker] transpile cache HIT: worker=21d5d010, version=76
[2026-01-24T22:39:34Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Thread-local pool initialized on thread ThreadId(14) (max_per_owner=None, queue_size=10)
[2026-01-24T22:39:34Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Cache MISS: creating new isolate for owner c88f9ecf-cc16-4b34-9006-f3b15346d2f8 (pool: 1/1000, owner: 1/None)
[2026-01-24T22:39:34Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Creating new TaggedIsolate for owner: c88f9ecf-cc16-4b34-9006-f3b15346d2f8

I wonder if there is a build/build options that I can use that might give better debug messages?

[max-lt]

Hi @netsgnut,

I've added detailed TRACE logs in v0.12.3 to help identify exactly where the SIGSEGV occurs. Can you test with this version?

Test 1: Normal mode with TRACE logs

First, try running with detailed logging to see where the crash happens:

# docker-compose.yml                                                                                                                                                                                                                                                                                           
environment:
  RUST_LOG: 'info,openworkers_runner=trace,openworkers_runtime_v8=trace'
  # ... your other env vars

What to look for:
You should first see the execution mode being used:

[DEBUG openworkers_runner] V8 execution mode: Pinned

If it crashes during isolate creation, you should see:

[TRACE] Creating v8::UnenteredIsolate (before v8::Isolate::new_unentered)
[TRACE] v8::UnenteredIsolate created successfully
[TRACE] Creating v8::Locker for heap limit callback (before v8::Locker::new)
💥 SIGSEGV here

This will confirm the exact location of the crash.

Test 2: ONESHOT mode (workaround)

If the crash still occurs, try ONESHOT mode which uses a different code path (same as task-executor which works for you):

environment:
  V8_EXECUTE: 'ONESHOT'
  RUST_LOG: 'info,openworkers_runner=trace,openworkers_runtime_v8=trace'
  # ... your other env vars

You should see:

[DEBUG openworkers_runner] V8 execution mode: Oneshot                                                                                                                                                                                                                                                          

ONESHOT creates a fresh isolate per request (no pooling), which will avoid the v8::Locker issue (if this the real issue here, but it likely is). It is the same mode than before v0.8.7

Can you share the TRACE logs from both tests? This will help narrow down the root cause.

Thanks!

[netsgnut]

Hi @max-lt ,

I have done some testing with v0.12.3. The logs are prepared using this line:

environment:
  - RUST_LOG=trace,openworkers_runner=trace,openworkers_runtime=trace,openworkers_runtime_v8=trace

---

For test 1, I can confirm that the execution mode is really pinned:

[2026-02-01T09:57:35Z DEBUG openworkers_runner] V8 execution mode: Pinned

For the first request each of the runner runs, I see:

[2026-02-01T09:58:31Z DEBUG openworkers_runner::store] worker found: id: 21d5d010, version: 122, bindings: 0, type: typescript
[2026-02-01T09:58:31Z DEBUG openworkers_runner] worker found: true
[2026-02-01T09:58:31Z INFO  openworkers_runner::worker_pool] Initializing worker semaphore with 160 max queued workers
[2026-02-01T09:58:31Z DEBUG openworkers_runner::worker] transpile cache MISS: worker=21d5d010, version=122
[2026-02-01T09:58:31Z DEBUG openworkers_runner::worker] transpile cache HIT: worker=21d5d010, version=122
[2026-02-01T09:58:31Z INFO  openworkers_runner::worker_pool] Sequential worker pool initialized with 16 threads
[2026-02-01T09:58:31Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Thread-local pool initialized on thread ThreadId(12) (max_per_owner=None, queue_size=10)
[2026-02-01T09:58:31Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Cache MISS: creating new isolate for owner c88f9ecf-cc16-4b34-9006-f3b15346d2f8 (pool: 1/1000, owner: 1/None)
[2026-02-01T09:58:31Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Creating new TaggedIsolate for owner: c88f9ecf-cc16-4b34-9006-f3b15346d2f8
[2026-02-01T09:58:31Z INFO  openworkers_runtime_v8::platform] Loaded runtime snapshot (437352 bytes) from /build/snapshot.bin
[2026-02-01T09:58:31Z TRACE openworkers_runtime_v8::locker_managed_isolate] Creating v8::UnenteredIsolate (before v8::Isolate::new_unentered)
[2026-02-01T09:58:31Z TRACE openworkers_runtime_v8::locker_managed_isolate] v8::UnenteredIsolate created successfully
[2026-02-01T09:58:31Z TRACE openworkers_runtime_v8::locker_managed_isolate] Creating v8::Locker for heap limit callback (before v8::Locker::new)
[2026-02-01T09:58:31Z TRACE openworkers_runtime_v8::locker_managed_isolate] v8::Locker created successfully, installing heap limit callback
[2026-02-01T09:58:31Z TRACE openworkers_runtime_v8::locker_managed_isolate] Heap limit callback installed, v8::Locker will be dropped
[2026-02-01T09:58:31Z INFO  openworkers_runtime_v8::thread_pinned_pool] TaggedIsolate created for owner c88f9ecf-cc16-4b34-9006-f3b15346d2f8 in 21.608997ms (snapshot: true)
[2026-02-01T09:58:31Z TRACE openworkers_runtime_v8::thread_pinned_pool] Acquired thread-local isolate for owner: c88f9ecf-cc16-4b34-9006-f3b15346d2f8 (requests: 1)
[2026-02-01T09:58:31Z TRACE openworkers_runtime_v8::thread_pinned_pool] Creating v8::Locker for owner c88f9ecf-cc16-4b34-9006-f3b15346d2f8 (before v8::Locker::new)
[2026-02-01T09:58:31Z TRACE openworkers_runtime_v8::thread_pinned_pool] v8::Locker created for owner c88f9ecf-cc16-4b34-9006-f3b15346d2f8, processing deferred destructions

The one before SIGSEGV, however, looks like this:

[2026-02-01T09:58:37Z DEBUG openworkers_runner::store] worker found: id: 21d5d010, version: 122, bindings: 0, type: typescript
[2026-02-01T09:58:37Z DEBUG openworkers_runner] worker found: true
[2026-02-01T09:58:37Z DEBUG openworkers_runner::worker] transpile cache HIT: worker=21d5d010, version=122
[2026-02-01T09:58:37Z DEBUG openworkers_runner::worker] transpile cache HIT: worker=21d5d010, version=122
[2026-02-01T09:58:37Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Thread-local pool initialized on thread ThreadId(13) (max_per_owner=None, queue_size=10)
[2026-02-01T09:58:37Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Cache MISS: creating new isolate for owner c88f9ecf-cc16-4b34-9006-f3b15346d2f8 (pool: 1/1000, owner: 1/None)
[2026-02-01T09:58:37Z DEBUG openworkers_runtime_v8::thread_pinned_pool] Creating new TaggedIsolate for owner: c88f9ecf-cc16-4b34-9006-f3b15346d2f8
[2026-02-01T09:58:37Z TRACE openworkers_runtime_v8::locker_managed_isolate] Creating v8::UnenteredIsolate (before v8::Isolate::new_unentered)

---

For test 2, with V8_EXECUTE=ONESHOT,

[2026-02-01T10:06:38Z DEBUG openworkers_runner] V8 execution mode: Oneshot

I can confirm that the SIGSEGV problem no longer happens on my end.

2026-02-01T10:14:09Z DEBUG openworkers_runner::store] worker found: id: 21d5d010, version: 122, bindings: 0, type: typescript
[2026-02-01T10:14:09Z DEBUG openworkers_runner] worker found: true
[2026-02-01T10:14:09Z INFO  openworkers_runner::worker_pool] Initializing worker semaphore with 160 max queued workers
[2026-02-01T10:14:09Z DEBUG openworkers_runner::worker] transpile cache MISS: worker=21d5d010, version=122
[2026-02-01T10:14:09Z DEBUG openworkers_runner::worker] transpile cache HIT: worker=21d5d010, version=122
[2026-02-01T10:14:09Z INFO  openworkers_runner::worker_pool] Sequential worker pool initialized with 16 threads
[2026-02-01T10:14:09Z INFO  openworkers_runtime_v8::platform] Loaded runtime snapshot (437352 bytes) from /build/snapshot.bin
[2026-02-01T10:14:09Z DEBUG openworkers_runner] handle_request done in 28ms
[2026-02-01T10:14:09Z DEBUG openworkers_runner::log] Flushing logs for worker: 21d5d010
[2026-02-01T10:14:10Z DEBUG openworkers_runner] handle_request: GET / in thread ThreadId(2)
[2026-02-01T10:14:10Z DEBUG openworkers_runner] host: Some("abc-worker.lab.example.com"), worker_id: None, worker_name: Some("abc")
[2026-02-01T10:14:10Z DEBUG openworkers_runner] request_id: 2df982ba9994c731bcafe89014451d90, worker_id: None, worker_name: Some("abc")
[2026-02-01T10:14:10Z DEBUG openworkers_runner::store] get_worker_with_bindings: Name("abc")
[2026-02-01T10:14:10Z DEBUG sqlx::query] summary="SELECT W.id::text, W.name, W.user_id::text, …" db.statement="\n\n\n        SELECT\n            W.id::text,\n            W.name,\n            W.user_id::text,\n            D.code,\n            D.code_type,\n            W.current_version as version\n        FROM workers AS W\n        JOIN worker_deployments AS D ON D.worker_id = W.id AND D.version = W.current_version\n        WHERE W.name = $1\n        \n" rows_affected=1 rows_returned=1 elapsed=2.108452ms elapsed_secs=0.002108452
[2026-02-01T10:14:10Z DEBUG sqlx::query] summary="SELECT V.key, V.value, V.type …" db.statement="\n\n\n        SELECT\n            V.key,\n            V.value,\n            V.type as binding_type\n        FROM environment_values AS V\n        JOIN workers AS W ON W.environment_id = V.environment_id AND W.user_id = V.user_id\n        WHERE W.id::text = $1\n    \n" rows_affected=0 rows_returned=0 elapsed=1.270586ms elapsed_secs=0.001270586

---

So it seems my environment really dislikes V8 isolate, then.

Please let me know if there are anything else you'd like me to try, too. Thanks!

[max-lt]

Hi @netsgnut,

Glad ONESHOT mode works for you! For personal/homelab use, performance should be perfectly fine — the main difference is ~1-2ms overhead per request for isolate creation instead of reusing pooled isolates.

To help narrow down the root cause of the PINNED mode crash, could you try:

1. Docker instead of Podman
   Just to rule out Podman-specific behavior.

2. runc instead of crun
   You're using crun 1.21, while I tested with runc 1.3.4. They handle cgroups/namespaces differently:

Install runc

sudo apt install runc

Configure Podman to use runc (in /etc/containers/containers.conf or ~/.config/containers/containers.conf):

[engine]
runtime = "runc"

The crash happening on the second thread during v8::Isolate::new_unentered() suggests something in V8's multi-threaded initialization doesn't play well with crun or kernel 6.12.

Let me know what you find!

[netsgnut]

Hi @max-lt,

Yeah, I am just glad that I have a newer version I can use for now without error, which should be fine my use here indeed. I also tested POOLED, which as expected, did not work. So I am using ONESHOT mode for now.

Thanks for pointing to runc. I tried it out and it also showed the same error with crun, so I think it can be safe to assume that it is not the case.

I might try Docker later, but for now I am just glad that I have got a working environment.

Thanks again!

[max-lt]

Thanks for testing runc! Interesting that it has the same issue — rules out the container runtime.

That leaves Podman 5.4 or kernel 6.12 as potential causes. If you ever try Docker, let us know.

For now, ONESHOT works well for your use case. And don't worry about POOLED not working — it's actually the slowest of the three modes (global mutex contention).

If we find a fix for PINNED mode later, we'll update the issue.