Case study · 2026-10-17
Tracing missing ROS 2 point clouds to a shared-memory transport limit
A synthetic point-cloud publisher, asymmetric transport tests, and Fast DDS diagnostics isolated a publisher-side shared-memory capacity limit behind incomplete local recordings.
Challenge
Missing point clouds in ROS 2 recordings needed to be isolated from sensor, transport, recorder, and storage failures.
Approach
Reproduce the loss with a synthetic publisher, compare best-effort and reliable QoS, remove disk writing, vary publisher and receiver segment capacity independently, and inspect the transport mechanism.
Result
- Publisher-side SHM capacity isolated in the local reproduction
- Local synthetic workload: 600/600 clouds recorded to tmpfs with a 16 MiB publisher segment
- Jetson: configured 16 MiB segment confirmed at runtime; approximately 10 Hz reception observed
Technologies
ROS 2 Humble • Fast DDS 2.6.11 • rmw_fastrtps_cpp • PointCloud2 • SQLite3
We were investigating missing point clouds in recordings from a Xavier/Hesai ROS 2 pipeline. To separate sensor-side failures from transport and recording losses, we built a synthetic PointCloud2 publisher on an x86_64 development host.
It reproduced incomplete delivery without the sensor. The experiments below isolate the cause of that local reproduction, followed by a runtime configuration and reception-rate check on the Jetson target.
Replacing the sensor with a controlled workload
The synthetic publisher did not need to reproduce every aspect of the LiDAR. Its purpose was narrower: generate a known stream of large PointCloud2 messages and let us count what arrived.
The test workloads covered two phases:
| Property | Initial QoS comparison | Segment-capacity tests |
|---|---|---|
| Points per cloud | 1,000; 49,152; 100,000 | 49,152 |
| Bytes per point | 16 | 16 |
| Point data per cloud | 16,000; 786,432; 1,600,000 bytes, respectively | 786,432 bytes, or 768 KiB |
| Requested publication rate | 10 Hz | 10 Hz |
| Publisher reliability | Best effort at all sizes; reliable at 49,152 and 100,000 points | Best effort |
| Publisher history depth | 5 | 5 |
| Clouds per run | 30 | 100; longer validation: 600 |
| Transport configuration | Default, no custom segment profile | Explicit segment profiles; longer validation changed only the publisher segment |
The environment was ROS 2 Humble, using rmw_fastrtps_cpp 6.2.10 and Fast DDS 2.6.11. Publisher and subscriber ran in separate processes, with localhost-only communication and isolated test domains.
The recorder used SQLite3 storage. Best-effort trials used an explicit best-effort, depth-100 subscription QoS override. Initial reliable trials instead used rosbag2's automatically matched subscription QoS, with history depth not explicitly controlled.
The synthetic publisher reproduced the loss without the physical sensor or vendor SDK, so neither was required to trigger the local failure. Each run sent a known number of clouds with sequential IDs, allowing us to compare sent and recorded counts and identify missing IDs.
Compare best-effort and reliable QoS
Before changing segment capacity, short untraced recording trials compared publisher reliability and cloud size using the default transport configuration. Each publisher sent 30 clouds at 10 Hz with history depth 5. No ros2 topic hz subscriber was present.
| Points per cloud | Publisher reliability | Recorded clouds | Missing IDs |
|---|---|---|---|
| 100,000 | Best effort | 23/30 | 6 internal gaps plus initial ID 0 |
| 100,000 | Reliable | 30/30 | None |
| 1,000 | Best effort | 30/30 | None |
| 49,152 | Best effort | 24/30 | 6 internal gaps |
| 49,152 | Reliable | 30/30 | None |
All five test runs completed 30 publish calls. Both reliable recordings contained all 30 clouds, while the larger best-effort workloads had missing clouds.
Reliable delivery supports retransmission of missing fragments. My interpretation is that this was the main reason the reliable recordings were complete. However, retransmission was not directly observed, and the recorder's history depth differed between configurations, so these tests did not conclusively isolate reliability's effect.
The later capacity tests investigated whether larger shared-memory segments reduced message loss. They used best-effort QoS throughout; reliable delivery was not retested with larger segments.
Remove the recorder
The local bag recordings reproduced the symptom: some clouds were missing.
But recording combines several stages—DDS reception, subscription handling, recorder buffering, and storage writes. To narrow the problem, we replaced the recorder with a subscriber that counted messages without writing a bag.
The subscriber also lost clouds, so neither the sensor nor bag writing was required to reproduce the local failure. We next investigated the publishing and delivery path.
Following the data through shared memory
Although the processes communicated locally, it was important to establish which transport actually carried the point clouds.
Earlier syscall tracing showed the publisher and recorder mapping the same Fast DDS shared-memory segments. The publisher process also generated a small amount of UDP traffic, but far too little to carry the full point-cloud workload.
This was consistent with shared-memory transport for the large payloads; it did not identify the purpose of the UDP traffic or measure shared-memory throughput.
Debugger inspection provided more direct evidence by observing point-cloud fragments in the shared-memory transport path. One cloud was sent as multiple Real-Time Publish-Subscribe (RTPS) protocol messages, including observed messages of 65,468 bytes containing DATA_FRAG submessages.
The cloud was fragmented inside the shared-memory transport path. This was RTPS fragmentation, not UDP/IP fragmentation.
“Shared-memory size” was not one setting
Several different resources could easily be confused:
- The available capacity of the
/dev/shmfilesystem. - The Fast DDS shared-memory transport segment.
- DDS history and receiver-side reassembly storage.
- The recorder’s cache.
The relevant resource in this reproduction was the publisher participant’s outgoing shared-memory transport segment. A DDS participant is the middleware entity through which a process joins a DDS domain; in these tests, the publisher and receiver used separate participants.
In Fast DDS 2.6.11 transport initialization, an unspecified segment size resolves to 512 KiB. The test cloud contained 768 KiB of point data alone, before serialized metadata.
That size comparison alone does not establish a failure: fragment buffers can be consumed and reused while a sample is being transferred. What matters is whether the segment can accommodate the outstanding fragments and other traffic before those buffers become reusable.
It was also not a shortage of free space in /dev/shm: the host had gigabytes available. More filesystem capacity does not automatically enlarge a participant’s configured transport segment.
Change one resource, on one side
We compared two explicit transport profiles. Both enabled UDPv4 and shared memory and disabled builtin transports. They differed only in segment_size:
- 512 KiB
- 16 MiB
The short recorder trials alternated between those configurations rather than testing each once:
| Test | Publisher segment | Receiver segment | Received or recorded |
|---|---|---|---|
| Bag recording, run 1 | 512 KiB | 512 KiB | 74/100 |
| Bag recording, run 2 | 16 MiB | 16 MiB | 100/100 |
| Bag recording, run 3 | 512 KiB | 512 KiB | 84/100 |
| Bag recording, run 4 | 16 MiB | 16 MiB | 100/100 |
| Subscriber only | 512 KiB | 16 MiB | 86/100 |
| Subscriber only | 16 MiB | 512 KiB | 100/100 |
With the publisher at 512 KiB, enlarging the receiver to 16 MiB still yielded 86/100 clouds. Reversing those settings yielded 100/100. The loss followed the publisher’s segment setting, not the receiver’s.
This makes sense once segment ownership is explicit. Each shared-memory-enabled DDS participant owns an outgoing transport segment. A receiving participant maps the sender’s segment and reads the referenced buffers there; it does not need to copy those fragments into its own outgoing segment.
The receiver still needs sufficient reassembly and history storage. But that is a separate allocation.
Connecting the results to the mechanism
A separate diagnostic run enabled native Fast DDS warnings. It produced:
Buffer is being invalidated, segment_size may be insufficient
The warning appeared 72 times, and the no-disk subscriber received 77 of 100 clouds. These counts are not directly comparable: a transport buffer is not necessarily a complete cloud. The main comparisons used untraced runs to avoid diagnostic overhead affecting the results.
The Fast DDS 2.6.11 implementation explains how insufficient capacity can cause loss. Under allocation pressure, the shared-memory manager can reclaim queued buffers that are not currently being processed, even if receivers have not yet consumed them. The receiving listener in that manager skips those invalidated buffers.
If a skipped buffer contains a required fragment, the receiver may be unable to reconstruct the cloud. Best-effort delivery does not request retransmission. This mechanism, together with the warnings and capacity tests, supports insufficient publisher-side segment capacity as the cause of the reproduced loss.
The capacity change—and what we actually validated
Increasing the publisher’s shared-memory transport segment to 16 MiB eliminated the observed loss in these tests while retaining best-effort QoS. This value was chosen to provide ample capacity, not to find the minimum required size; a smaller segment may also have been sufficient.
A longer, untraced validation run increased only the publisher’s shared-memory segment, leaving the recorder’s transport configuration at its defaults. It wrote the bag to tmpfs because disk space was limited. The run completed and recorded:
- 600 of 600 clouds
- IDs 0–599
- No internal gaps, duplicates, or backwards IDs
This was a delivery and recording-completeness check, not a sustained physical-disk benchmark. The short comparisons used ordinary temporary disk storage.
Complete delivery did not mean perfectly uniform timing. The mean recorded header spacing was about 100.45 ms, and the publisher reported timing overruns. Header spacing does not measure subscriber delivery latency.
Checking the configuration on the Jetson target
On the Jetson target, I applied the Fast DDS profile configuring a 16 MiB shared-memory transport segment and confirmed the enlarged segment at runtime. ros2 topic hz reported a received rate of approximately 10 Hz. These checks confirmed that the configuration took effect and the subscriber received clouds at approximately the expected rate; they did not establish recording completeness.
Capacity and integration limits
The 16 MiB segment was tested headroom, not a measured minimum or a universal recommendation. Different message sizes, traffic patterns, participants, or receiver stalls can require different capacity. Since segments are allocated per DDS participant, the memory cost also needs to be accounted for.
Any integration should preserve existing transport, discovery, and security configuration rather than replacing an application’s DDS profile wholesale.
Outcome and remaining validation
Increasing the publisher’s outgoing SHM segment to 16 MiB eliminated observed loss in the local tests, including a 600-cloud recording to tmpfs with no sequence gaps or duplicates. The asymmetric tests and version-matched transport inspection identify insufficient publisher-side segment capacity as the cause of the reproduced loss on this development host.
The local results point to insufficient publisher-side shared-memory capacity as a likely cause of the original Xavier/Hesai failure. The Jetson check confirmed that the enlarged segment was instantiated and reception was approximately 10 Hz, but did not establish whether the original recording loss was eliminated.