Case study · 2026-02-14
Shared-memory communication across an embedded autonomy stack
A shared-memory framework replaced internal ROS 2 communication across an embedded autonomy stack, carrying sensor data, perception results, control feedback and configuration while retaining ROS 2 diagnostics at the boundary.
Challenge
Replace internal ROS 2 communication across an embedded autonomy stack while supporting large sensor payloads, smaller state updates and existing diagnostic tooling.
Approach
Use a common single-writer/multiple-reader shared-memory abstraction with fixed-capacity slots, scoped reader access and a separate ROS 2 diagnostic adapter.
Result
- Shared-memory interfaces across sensor, perception and control processes
- Scoped payload access, including direct LiDAR decoding into SHM
- ROS 2 publication and bag recording through a separate adapter
Technologies
C++ • Linux • POSIX shared memory • ROS 2 • TI TDA4VM • NVIDIA Jetson
An embedded autonomy stack on a TI TDA4VM needed to exchange more than LiDAR clouds. Perception results fed control, control feedback informed perception, and configuration and diagnostic data had their own readers. Large payloads and small state updates needed a common way to move between local processes.
ROS 2 already used shared-memory transport in this stack, but shared memory alone does not remove the CPU, memory and configuration overhead of the ROS 2/DDS middleware. For communication between processes on the same device, I wanted a smaller interface with explicit payload ownership and latest-value access, while retaining ROS 2 for diagnostics and recording.
I implemented a single-writer/multiple-reader shared-memory framework and used it to replace the stack's internal ROS 2 interfaces. Sensor, perception, control and configuration processes use the same transport abstraction; ROS 2 remains at the boundary for diagnostic publication and bag recording.
The framework divides its shared storage into a fixed number of payload slots. A read guard holds a slot's read lock while a consumer accesses its payload. Multiple readers can hold guards on the same slot simultaneously, accessing the same completed payload without making separate copies. The writer cannot reuse that slot until every reader has released its guard, but it can write into another available slot while those readers continue.
Consumers can process the shared payload directly while holding a read guard. The LiDAR driver integration demonstrates the zero-copy path from decoder output into shared storage and on to reader access.
One interface across the stack
The framework is a typed C++ abstraction instantiated for each shared payload. A named segment has one writer and can serve multiple readers. The abstraction is used for both large point arrays and smaller state and configuration structures:
| Data | Role in the stack |
|---|---|
| LiDAR point clouds | Sensor output consumed by perception, configuration and diagnostic processes |
| Preprocessed point clouds | Perception output consumed by a downstream processing component and diagnostics |
| Processing results | Inputs to control, with diagnostic readers alongside them |
| Control feedback, status and debug data | State shared with perception, downstream processing and diagnostics |
| System and perception configuration | Parameters distributed to the processes that use them |
This replaced internal ROS message exchange, rather than adding a shared-memory shortcut for just one sensor stream. Each interface still has its own payload type and owning producer; the common framework handles shared storage, publication and reader access.
Architecture
The framework uses POSIX shared memory and process-shared synchronization primitives on Linux. It has been tested on TI TDA4VM and NVIDIA Jetson platforms; other Linux targets have not yet been validated. It does not depend on vendor-specific hardware APIs, but a new target must support the required shared-memory, pthread and atomic operations. Processes sharing a segment must also use compatible payload and synchronization-object layouts.
The transport uses POSIX shared-memory objects mapped into each participating process. Each object contains several fixed-capacity payload slots, per-slot read/write locks, version stamps and notification primitives. The number of slots is configurable.
The following snapshot shows a writer and two readers using one segment:
Three slots are shown. The producer writes slot B while perception and diagnostics simultaneously read slot A without copying its payload. Slot A cannot be reused until both readers release their guards. ROS conversion and serialization occur in the diagnostic adapter.
Processes map the same shared-memory objects rather than sending payloads through ROS 2 for internal communication. They do not exchange raw pointers: each process accesses the slots through its own mapping.
Ownership and delivery semantics
The writer scans slots using nonblocking write-lock attempts. After filling a slot, it publishes a version and latest-slot index, releases the write lock and notifies waiting readers. A reader holds a slot's read lock through a move-only guard, which releases the lock when it leaves scope or when the caller explicitly releases it. Multiple readers can hold the same published slot, and a writer cannot overwrite it while those read locks remain held.
Once all readers of a slot release their guards, that slot becomes eligible for reuse. Shared storage makes payload access zero-copy; multiple slots allow production and reading of different frames to overlap, provided another slot is available. The slots hold successive payload versions, not separate copies for each reader.
Consumers have several access patterns through the same interface:
- Wait for data, then access it directly. Check the latest published slot first. If it contains an update since this reader's last acquisition, acquire a read guard without waiting for another publication. Otherwise, wait for an update, returning an empty guard on a notification timeout. Acquiring the slot's read lock can still block.
- Check for data without waiting. The nonblocking counterpart: check the latest published slot and return a read guard if it contains an update and its read lock is immediately available. Return an empty guard if there is no update or the slot is busy.
- Copy an update and release the slot. Check the latest published slot without waiting. If an update is immediately accessible, copy its payload into local storage and release the read lock before returning. Subsequent processing uses the local copy, not the shared slot. This is used for smaller state and configuration updates. If there is no update or the slot is busy, return false and leave the local copy unchanged.
- Access a specific version. Look for the requested version among the retained slots rather than selecting the latest one. Return a read guard if a matching slot can be locked immediately and its version still matches after locking. Otherwise, return an empty guard; the call does not wait for that version to become available.
The first three operations provide latest-value access, not a sequential queue. Each reader tracks its last-seen version to detect updates. If it last acquired version 10 and versions 11 and 12 have since been published, it selects the latest slot, version 12, rather than reading version 11 first. Intermediate versions can therefore be skipped, and unpinned slots can be recycled before a slow reader uses them. Selection follows the latest-slot index observed during the call; another publication can occur before the call returns.
Fixed capacity does not imply bounded waiting. If every slot is unavailable, the writer falls back to a blocking write lock. Slot count and reader hold time therefore matter to producer progress.
Use case: LiDAR decoding directly into SHM
Using shared memory only after decoding would still require a full-frame copy from an SDK-owned cloud into the shared region. I adapted the LiDAR driver and its output cloud representation so the decoder fills shared-memory slots directly, avoiding that full-frame transfer.
The driver accepts a cloud type as a template parameter and callbacks for
obtaining and returning clouds. Its standard cloud uses std::vector for
points and std::string for the frame ID. These containers can store their
data on the producing process's heap and hold pointers to that storage. Those
pointers refer to addresses in that process, not shared storage that another
process can access. Placing the container objects in SHM therefore does not
make their point or character data shared.
I supplied a flat cloud structure with an inline fixed-capacity point array and a fixed character array for the frame ID. The points and characters live inside the shared slot itself, so each process accesses them at the same offsets within its own mapping, without following pointers into another process's heap.
Fixed capacity does not mean every cloud has the same number of points. The slot reserves space for a maximum point count and records how many entries are in use. This keeps the slot's size fixed while allowing frame sizes to vary within that capacity. Dynamic storage inside shared memory is possible, but would require a shared allocator and references that remain valid across process mappings; this layout avoids that additional machinery.
A small container interface supplies the operations the decoder expects,
including resize(), clear() and emplace_back(), over the inline array.
I also changed the driver's frame-header assignment to write into the fixed
character array instead of assigning a std::string.
Because the point array has fixed capacity, it must be sized to accommodate the expected cloud size. The current container does not grow: resize requests are clamped, and points appended after capacity is reached are ignored without an explicit overflow report.
For each cloud, the handoff works as follows:
- The allocation callback acquires a writable shared-memory slot and gives
the driver a
shared_ptrto the cloud stored there. - The driver builds the cloud directly in the shared slot, storing both decoded points and associated metadata there.
- When the frame is complete, the driver calls the return callback with the completed cloud. In this integration, that callback does not publish the slot.
- When the driver releases its last reference to the completed cloud, the
shared_ptr's custom deleter publishes the slot. It updates the version and latest-slot index, releases the write lock and notifies waiting readers.
These lifetimes can overlap: the driver can acquire another writable slot before releasing its last reference to the previous cloud and publishing it.
The shared_ptr manages the SDK-side lifetime within the producer process; it
is not passed between processes. Other processes map the shared-memory object
and acquire their own read guards.
No completed cloud is copied from SDK-owned storage into SHM, and readers access that shared payload without another copy. The decoder still constructs individual point values before appending them; zero-copy here describes the cloud handoff, not network reception.
Where a copy is useful
A consumer that operates on the shared representation can process it directly while holding a read guard. The current perception integration instead uses a generated algorithm that expects a different array representation. The process converts the shared points into that working array, saves the source timestamp and explicitly releases the guard before running the algorithm. A downstream processing component uses the same copy-then-release pattern.
This retains a representation-conversion copy, but avoids keeping the source slot pinned during the subsequent computation. Shorter ownership reduces the time a reader prevents reuse of that particular slot; it does not by itself establish a bound on producer latency.
Keeping ROS 2 diagnostics and recording
A segment can serve several components at once: perception can read a sensor cloud while a diagnostic adapter reads the same stream for ROS 2 publication or bag recording. Each reader tracks updates independently, and reading a payload does not consume it or remove it for other readers. The producer does not need a separate output buffer for each reader.
The adapter is another reader of the framework, not the internal transport. Its ROS conversion copies data, and its read guards can delay slot reuse just like those of any application consumer.
Conclusion
I replaced the stack's internal ROS 2 interfaces with a common shared-memory transport, including direct LiDAR decoding into shared storage. A separate adapter preserves ROS 2 publication and recording.
Readers can delay slot reuse, and scoped guards handle normal execution rather than recovery from a process being killed. Restarting a segment does not automatically reconnect existing readers. Notification-mutex recovery is incomplete, and startup needs a bounded initialization wait and layout-compatibility checks.
This provides direct shared-payload access across the stack; quantifying the performance benefit requires representative latency, jitter and sustained-load measurements.
Technical deep dives