Technical note index

Technical note · 2026-02-16

Two-layer synchronization for POSIX shared-memory rings

Sharing high-bandwidth sensor payloads across processes requires decoupling update notification from payload access. Here is how a two-layer synchronization scheme lets slow readers hold frames without stalling writers until buffer capacity runs out.

From the case study: Shared-memory communication across an embedded autonomy stack

In a shared-memory sensor pipeline, readers consume data at different rates: a diagnostic bridge might poll at 10 Hz, while a perception model runs at 30 Hz with variable compute times.

If synchronization relies on a single shared mutex or queue, any slow consumer stalls the producer. The solution is to separate global frame discovery from per-slot payload protection.

The two-layer architecture

Inside a single mmap-ed POSIX shared-memory segment, synchronization is split into two layers:

  1. Global discovery layer (SharedHeader)
    • Monotonic atomic<uint64_t> global_version and atomic<uint32_t> latest_idx.
    • Process-shared condition variable (pthread_cond_t) and robust mutex (pthread_mutex_t with PTHREAD_MUTEX_ROBUST).
    • Signals when a new frame is finalized and wakes sleeping readers.
  2. Per-slot access layer (SlotHeader[N] + PaddedSlot[N])
    • A ring of N fixed-capacity payload slots, each with its own pthread_rwlock_t and atomic version.
    • Multiple readers can hold shared read locks (rdlock) on a published slot via RAII guards, while the writer independently locks and fills an available slot.
    • Per-slot headers and data wrappers are cache-line aligned (alignas(64)) to reduce cache bouncing (false sharing) between cores operating on different slots.
Two-layer shared-memory architecture: Layer 1 tracks latest_idx and wake-up signals in SharedHeader, while Layer 2 isolates payload access into independent slots with per-slot read/write locks.

Layer 1 handles frame discovery and sleep/wake signaling; Layer 2 isolates payload locking to individual ring slots so active readers on Slot 0 never block the writer on Slot 1.

Fast path vs slow path

Readers access data through two distinct paths depending on their cadence:

  • Fast path (active polling / latest value): A running reader loads latest_idx and that slot's version with acquire memory ordering. If the version is newer than its last processed frame, it acquires pthread_rwlock_rdlock directly on that slot. It never touches the global condition variable or mutex.
  • Slow path (blocking wait): When a reader has caught up to the writer, it takes the robust condition mutex and calls pthread_cond_timedwait. On wake-up, it re-checks latest_idx under the mutex to prevent lost wake-ups before transitioning to a local slot read-lock.

Writer progress and backpressure

The writer rotates through the ring using non-blocking write-lock attempts (pthread_rwlock_trywrlock). If readers are holding guards on several slots, the writer skips them and writes into the first free slot.

Once written, the writer:

  1. Stores the new version on the slot and updates latest_idx with release semantics.
  2. Unlocks the slot's write lock so readers can enter.
  3. Locks the condition mutex (cond_mut), broadcasts on pthread_cond_t to wake sleeping readers, and releases the mutex.

If all N slots are held simultaneously by slow readers, the writer falls back to a blocking write lock on the oldest slot. This introduces explicit backpressure rather than silently dropping or corrupting memory mid-read.

Sequence diagram showing the writer releasing the slot write lock before broadcasting on the condition variable, the slow reader releasing the global mutex before locking the slot, and the fast reader bypassing the mutex entirely.

Lock handoff sequence. The writer releases the slot write lock before signaling the condition variable under the condition mutex, and waking readers release the mutex before acquiring the slot read lock.

Progress and failure boundaries

ScenarioWriter outcomeConsumer / system impact
Reader holds one historical slotWriter skips to next free slotZero impact on producer or other readers
Many readers hold same slotStill counts as one pinned slotShared rdlock consumes no extra ring capacity
Readers hold all slots ($N$)Writer blocks on oldest slotBounded backpressure; rate matches slowest reader
Process crashes holding condition mutexWriter/reader recovers via EOWNERDEAD + pthread_mutex_consistentDiscovery path stays operational
Process crashes holding slot rwlockSlot leaked; writer skips until remaining capacity exhaustedPOSIX rwlock cannot recover owner death; requires supervisor restart

Gotchas and boundaries

  • Double-check versions after locking. Between reading latest_idx and acquiring the slot's read lock, a fast producer could recycle the slot. Always re-verify the slot's atomic version after rdlock succeeds.
  • Lost wake-up prevention. Sleeping readers check the version while holding the condition mutex before entering pthread_cond_timedwait. The writer also acquires this mutex before broadcasting. Coupling the predicate test and signal through the mutex closes the race where a notification arrives just before a reader sleeps.
  • Robust mutex handling. Setting PTHREAD_MUTEX_ROBUST on the condition mutex ensures that if a process crashes while holding it, subsequent callers catch EOWNERDEAD and call pthread_mutex_consistent instead of hanging forever.
  • Slot count sizing. For the writer to stay non-blocking, at least one slot must remain free. If consumers concurrently hold at most $D$ distinct slots, the ring needs $N > D$ (plus one active write slot). Because slots use shared locks (pthread_rwlock_t), any number of readers can pin the same slot without eating extra capacity. Ring stalls only happen when readers hold multiple different historical slots simultaneously.
  • Process death on slot locks. While PTHREAD_MUTEX_ROBUST protects the condition mutex, POSIX pthread_rwlock_t has no kernel-level owner-death recovery. Mutexes can track a single owning thread ID in the kernel lock word, but shared rwlocks only maintain anonymous reader counts in shared memory. If a process dies mid-read, its slot lock is never returned. In practice, keep read locks short (copy or serialize out immediately, then release the guard) and let process supervision (systemd cgroups) clean up and restart the shared memory ring if a required node terminates unexpectedly.

Takeaway

Decouple discovery from access: use atomics and condition variables to broadcast frame availability, but isolate data locking to independent, cache-aligned slots. This gives readers zero-copy access while preserving producer throughput.