Technical note · 2026-02-16
Two-layer synchronization for POSIX shared-memory rings
Sharing high-bandwidth sensor payloads across processes requires decoupling update notification from payload access. Here is how a two-layer synchronization scheme lets slow readers hold frames without stalling writers until buffer capacity runs out.
In a shared-memory sensor pipeline, readers consume data at different rates: a diagnostic bridge might poll at 10 Hz, while a perception model runs at 30 Hz with variable compute times.
If synchronization relies on a single shared mutex or queue, any slow consumer stalls the producer. The solution is to separate global frame discovery from per-slot payload protection.
The two-layer architecture
Inside a single mmap-ed POSIX shared-memory segment, synchronization is split
into two layers:
- Global discovery layer (
SharedHeader)- Monotonic
atomic<uint64_t> global_versionandatomic<uint32_t> latest_idx. - Process-shared condition variable (
pthread_cond_t) and robust mutex (pthread_mutex_twithPTHREAD_MUTEX_ROBUST). - Signals when a new frame is finalized and wakes sleeping readers.
- Monotonic
- Per-slot access layer (
SlotHeader[N]+PaddedSlot[N])- A ring of N fixed-capacity payload slots, each with its own
pthread_rwlock_tand atomic version. - Multiple readers can hold shared read locks (
rdlock) on a published slot via RAII guards, while the writer independently locks and fills an available slot. - Per-slot headers and data wrappers are cache-line aligned (
alignas(64)) to reduce cache bouncing (false sharing) between cores operating on different slots.
- A ring of N fixed-capacity payload slots, each with its own
Layer 1 handles frame discovery and sleep/wake signaling; Layer 2 isolates payload locking to individual ring slots so active readers on Slot 0 never block the writer on Slot 1.
Fast path vs slow path
Readers access data through two distinct paths depending on their cadence:
- Fast path (active polling / latest value): A running reader loads
latest_idxand that slot'sversionwith acquire memory ordering. If the version is newer than its last processed frame, it acquirespthread_rwlock_rdlockdirectly on that slot. It never touches the global condition variable or mutex. - Slow path (blocking wait): When a reader has caught up to the writer, it
takes the robust condition mutex and calls
pthread_cond_timedwait. On wake-up, it re-checkslatest_idxunder the mutex to prevent lost wake-ups before transitioning to a local slot read-lock.
Writer progress and backpressure
The writer rotates through the ring using non-blocking write-lock attempts
(pthread_rwlock_trywrlock). If readers are holding guards on several slots,
the writer skips them and writes into the first free slot.
Once written, the writer:
- Stores the new version on the slot and updates
latest_idxwith release semantics. - Unlocks the slot's write lock so readers can enter.
- Locks the condition mutex (
cond_mut), broadcasts onpthread_cond_tto wake sleeping readers, and releases the mutex.
If all N slots are held simultaneously by slow readers, the writer falls back to a blocking write lock on the oldest slot. This introduces explicit backpressure rather than silently dropping or corrupting memory mid-read.
Lock handoff sequence. The writer releases the slot write lock before signaling the condition variable under the condition mutex, and waking readers release the mutex before acquiring the slot read lock.
Progress and failure boundaries
| Scenario | Writer outcome | Consumer / system impact |
|---|---|---|
| Reader holds one historical slot | Writer skips to next free slot | Zero impact on producer or other readers |
| Many readers hold same slot | Still counts as one pinned slot | Shared rdlock consumes no extra ring capacity |
| Readers hold all slots ($N$) | Writer blocks on oldest slot | Bounded backpressure; rate matches slowest reader |
| Process crashes holding condition mutex | Writer/reader recovers via EOWNERDEAD + pthread_mutex_consistent | Discovery path stays operational |
Process crashes holding slot rwlock | Slot leaked; writer skips until remaining capacity exhausted | POSIX rwlock cannot recover owner death; requires supervisor restart |
Gotchas and boundaries
- Double-check versions after locking. Between reading
latest_idxand acquiring the slot's read lock, a fast producer could recycle the slot. Always re-verify the slot's atomic version afterrdlocksucceeds. - Lost wake-up prevention. Sleeping readers check the version while holding
the condition mutex before entering
pthread_cond_timedwait. The writer also acquires this mutex before broadcasting. Coupling the predicate test and signal through the mutex closes the race where a notification arrives just before a reader sleeps. - Robust mutex handling. Setting
PTHREAD_MUTEX_ROBUSTon the condition mutex ensures that if a process crashes while holding it, subsequent callers catchEOWNERDEADand callpthread_mutex_consistentinstead of hanging forever. - Slot count sizing. For the writer to stay non-blocking, at least one slot
must remain free. If consumers concurrently hold at most $D$ distinct slots,
the ring needs $N > D$ (plus one active write slot). Because slots use shared
locks (
pthread_rwlock_t), any number of readers can pin the same slot without eating extra capacity. Ring stalls only happen when readers hold multiple different historical slots simultaneously. - Process death on slot locks. While
PTHREAD_MUTEX_ROBUSTprotects the condition mutex, POSIXpthread_rwlock_thas no kernel-level owner-death recovery. Mutexes can track a single owning thread ID in the kernel lock word, but shared rwlocks only maintain anonymous reader counts in shared memory. If a process dies mid-read, its slot lock is never returned. In practice, keep read locks short (copy or serialize out immediately, then release the guard) and let process supervision (systemdcgroups) clean up and restart the shared memory ring if a required node terminates unexpectedly.
Takeaway
Decouple discovery from access: use atomics and condition variables to broadcast frame availability, but isolate data locking to independent, cache-aligned slots. This gives readers zero-copy access while preserving producer throughput.