Case study index

Case study · 2026-10-17

Tracing missing ROS 2 point clouds to a shared-memory transport limit

A synthetic point-cloud publisher, asymmetric transport tests, and Fast DDS diagnostics isolated a publisher-side shared-memory capacity limit behind incomplete local recordings.

Challenge

Missing point clouds in ROS 2 recordings needed to be isolated from sensor, transport, recorder, and storage failures.

Approach

Reproduce the loss with a synthetic publisher, compare best-effort and reliable QoS, remove disk writing, vary publisher and receiver segment capacity independently, and inspect the transport mechanism.

Result

  • Publisher-side SHM capacity isolated in the local reproduction
  • Local synthetic workload: 600/600 clouds recorded to tmpfs with a 16 MiB publisher segment
  • Jetson: configured 16 MiB segment confirmed at runtime; approximately 10 Hz reception observed

Technologies

ROS 2 Humble • Fast DDS 2.6.11 • rmw_fastrtps_cpp • PointCloud2 • SQLite3

We were investigating missing point clouds in recordings from a Xavier/Hesai ROS 2 pipeline. To separate sensor-side failures from transport and recording losses, we built a synthetic PointCloud2 publisher on an x86_64 development host.

It reproduced incomplete delivery without the sensor. The experiments below isolate the cause of that local reproduction, followed by a runtime configuration and reception-rate check on the Jetson target.

Replacing the sensor with a controlled workload

The synthetic publisher did not need to reproduce every aspect of the LiDAR. Its purpose was narrower: generate a known stream of large PointCloud2 messages and let us count what arrived.

The test workloads covered two phases:

PropertyInitial QoS comparisonSegment-capacity tests
Points per cloud1,000; 49,152; 100,00049,152
Bytes per point1616
Point data per cloud16,000; 786,432; 1,600,000 bytes, respectively786,432 bytes, or 768 KiB
Requested publication rate10 Hz10 Hz
Publisher reliabilityBest effort at all sizes; reliable at 49,152 and 100,000 pointsBest effort
Publisher history depth55
Clouds per run30100; longer validation: 600
Transport configurationDefault, no custom segment profileExplicit segment profiles; longer validation changed only the publisher segment

The environment was ROS 2 Humble, using rmw_fastrtps_cpp 6.2.10 and Fast DDS 2.6.11. Publisher and subscriber ran in separate processes, with localhost-only communication and isolated test domains.

The recorder used SQLite3 storage. Best-effort trials used an explicit best-effort, depth-100 subscription QoS override. Initial reliable trials instead used rosbag2's automatically matched subscription QoS, with history depth not explicitly controlled.

The synthetic publisher reproduced the loss without the physical sensor or vendor SDK, so neither was required to trigger the local failure. Each run sent a known number of clouds with sequential IDs, allowing us to compare sent and recorded counts and identify missing IDs.

Compare best-effort and reliable QoS

Before changing segment capacity, short untraced recording trials compared publisher reliability and cloud size using the default transport configuration. Each publisher sent 30 clouds at 10 Hz with history depth 5. No ros2 topic hz subscriber was present.

Points per cloudPublisher reliabilityRecorded cloudsMissing IDs
100,000Best effort23/306 internal gaps plus initial ID 0
100,000Reliable30/30None
1,000Best effort30/30None
49,152Best effort24/306 internal gaps
49,152Reliable30/30None

All five test runs completed 30 publish calls. Both reliable recordings contained all 30 clouds, while the larger best-effort workloads had missing clouds.

Reliable delivery supports retransmission of missing fragments. My interpretation is that this was the main reason the reliable recordings were complete. However, retransmission was not directly observed, and the recorder's history depth differed between configurations, so these tests did not conclusively isolate reliability's effect.

The later capacity tests investigated whether larger shared-memory segments reduced message loss. They used best-effort QoS throughout; reliable delivery was not retested with larger segments.

Remove the recorder

The local bag recordings reproduced the symptom: some clouds were missing.

But recording combines several stages—DDS reception, subscription handling, recorder buffering, and storage writes. To narrow the problem, we replaced the recorder with a subscriber that counted messages without writing a bag.

The subscriber also lost clouds, so neither the sensor nor bag writing was required to reproduce the local failure. We next investigated the publishing and delivery path.

Following the data through shared memory

Although the processes communicated locally, it was important to establish which transport actually carried the point clouds.

Earlier syscall tracing showed the publisher and recorder mapping the same Fast DDS shared-memory segments. The publisher process also generated a small amount of UDP traffic, but far too little to carry the full point-cloud workload.

This was consistent with shared-memory transport for the large payloads; it did not identify the purpose of the UDP traffic or measure shared-memory throughput.

Debugger inspection provided more direct evidence by observing point-cloud fragments in the shared-memory transport path. One cloud was sent as multiple Real-Time Publish-Subscribe (RTPS) protocol messages, including observed messages of 65,468 bytes containing DATA_FRAG submessages.

The cloud was fragmented inside the shared-memory transport path. This was RTPS fragmentation, not UDP/IP fragmentation.

“Shared-memory size” was not one setting

Several different resources could easily be confused:

  • The available capacity of the /dev/shm filesystem.
  • The Fast DDS shared-memory transport segment.
  • DDS history and receiver-side reassembly storage.
  • The recorder’s cache.

The relevant resource in this reproduction was the publisher participant’s outgoing shared-memory transport segment. A DDS participant is the middleware entity through which a process joins a DDS domain; in these tests, the publisher and receiver used separate participants.

In Fast DDS 2.6.11 transport initialization, an unspecified segment size resolves to 512 KiB. The test cloud contained 768 KiB of point data alone, before serialized metadata.

That size comparison alone does not establish a failure: fragment buffers can be consumed and reused while a sample is being transferred. What matters is whether the segment can accommodate the outstanding fragments and other traffic before those buffers become reusable.

It was also not a shortage of free space in /dev/shm: the host had gigabytes available. More filesystem capacity does not automatically enlarge a participant’s configured transport segment.

Change one resource, on one side

We compared two explicit transport profiles. Both enabled UDPv4 and shared memory and disabled builtin transports. They differed only in segment_size:

  • 512 KiB
  • 16 MiB

The short recorder trials alternated between those configurations rather than testing each once:

TestPublisher segmentReceiver segmentReceived or recorded
Bag recording, run 1512 KiB512 KiB74/100
Bag recording, run 216 MiB16 MiB100/100
Bag recording, run 3512 KiB512 KiB84/100
Bag recording, run 416 MiB16 MiB100/100
Subscriber only512 KiB16 MiB86/100
Subscriber only16 MiB512 KiB100/100

With the publisher at 512 KiB, enlarging the receiver to 16 MiB still yielded 86/100 clouds. Reversing those settings yielded 100/100. The loss followed the publisher’s segment setting, not the receiver’s.

This makes sense once segment ownership is explicit. Each shared-memory-enabled DDS participant owns an outgoing transport segment. A receiving participant maps the sender’s segment and reads the referenced buffers there; it does not need to copy those fragments into its own outgoing segment.

The receiver still needs sufficient reassembly and history storage. But that is a separate allocation.

Connecting the results to the mechanism

A separate diagnostic run enabled native Fast DDS warnings. It produced:

Buffer is being invalidated, segment_size may be insufficient

The warning appeared 72 times, and the no-disk subscriber received 77 of 100 clouds. These counts are not directly comparable: a transport buffer is not necessarily a complete cloud. The main comparisons used untraced runs to avoid diagnostic overhead affecting the results.

The Fast DDS 2.6.11 implementation explains how insufficient capacity can cause loss. Under allocation pressure, the shared-memory manager can reclaim queued buffers that are not currently being processed, even if receivers have not yet consumed them. The receiving listener in that manager skips those invalidated buffers.

If a skipped buffer contains a required fragment, the receiver may be unable to reconstruct the cloud. Best-effort delivery does not request retransmission. This mechanism, together with the warnings and capacity tests, supports insufficient publisher-side segment capacity as the cause of the reproduced loss.

The capacity change—and what we actually validated

Increasing the publisher’s shared-memory transport segment to 16 MiB eliminated the observed loss in these tests while retaining best-effort QoS. This value was chosen to provide ample capacity, not to find the minimum required size; a smaller segment may also have been sufficient.

A longer, untraced validation run increased only the publisher’s shared-memory segment, leaving the recorder’s transport configuration at its defaults. It wrote the bag to tmpfs because disk space was limited. The run completed and recorded:

  • 600 of 600 clouds
  • IDs 0–599
  • No internal gaps, duplicates, or backwards IDs

This was a delivery and recording-completeness check, not a sustained physical-disk benchmark. The short comparisons used ordinary temporary disk storage.

Complete delivery did not mean perfectly uniform timing. The mean recorded header spacing was about 100.45 ms, and the publisher reported timing overruns. Header spacing does not measure subscriber delivery latency.

Checking the configuration on the Jetson target

On the Jetson target, I applied the Fast DDS profile configuring a 16 MiB shared-memory transport segment and confirmed the enlarged segment at runtime. ros2 topic hz reported a received rate of approximately 10 Hz. These checks confirmed that the configuration took effect and the subscriber received clouds at approximately the expected rate; they did not establish recording completeness.

Capacity and integration limits

The 16 MiB segment was tested headroom, not a measured minimum or a universal recommendation. Different message sizes, traffic patterns, participants, or receiver stalls can require different capacity. Since segments are allocated per DDS participant, the memory cost also needs to be accounted for.

Any integration should preserve existing transport, discovery, and security configuration rather than replacing an application’s DDS profile wholesale.

Outcome and remaining validation

Increasing the publisher’s outgoing SHM segment to 16 MiB eliminated observed loss in the local tests, including a 600-cloud recording to tmpfs with no sequence gaps or duplicates. The asymmetric tests and version-matched transport inspection identify insufficient publisher-side segment capacity as the cause of the reproduced loss on this development host.

The local results point to insufficient publisher-side shared-memory capacity as a likely cause of the original Xavier/Hesai failure. The Jetson check confirmed that the enlarged segment was instantiated and reception was approximately 10 Hz, but did not establish whether the original recording loss was eliminated.

References