The detections reach the dashboard

1. The one thing that was still missing

By the end of Week 13 every piece existed except the join. The detector produced boxes and published them on the standard vision_msgs/Detection3DArray topic; the dashboard drew a live point cloud. But the dashboard was not reading the boxes. You could see the perception output only in a separate, headless bird's-eye renderer on the server — never on the screen that is meant to be the product.

This week is that join, and nothing more glamorous: teach the HMI to subscribe to the detections and draw them. It is a small change in lines of code and a big one in what the demo means, because it is the first time the perception result appears where a driver would actually see it.

2. What changed in the app

The HMI already spoke rosbridge for the point cloud. Adding the boxes meant a second subscription over the same WebSocket and a second thing to draw:

• A second topic. Alongside /carla/lidar, the app now subscribes to /carla/detections (vision_msgs/msg/Detection3DArray). Same connection, same JSON transport — rosbridge just hands over a second stream.
• Parsing a detection. For each box the app reads the centre (bbox.center.position), the size (bbox.size), and the heading, recovering the yaw angle from the orientation quaternion. That becomes a simple footprint rectangle in the car's ground plane.
• Drawing it. The painter rotates each box's four corners by its yaw, maps them from sensor metres to screen pixels the same way the points are mapped, and strokes them in green with a faint fill. A chip in the top bar counts how many boxes are currently on screen.

The endpoint the app connects to is a build-time setting rather than a hard-coded address, which is what let the same app binary connect to a local rosbridge on the bench and to a tunnelled one in this run without editing a line.

3. Running it end to end

The interesting part is that all of this ran across the same split setup the project has lived with: the heavy compute on the GPU-less server, the AGL screen emulated on the laptop, and rosbridge as the bridge between them. Four processes, three of them on the laptop:

# 1. ON THE SERVER: loop the recorded bag through the detector,
#    with rosbridge serving /carla/lidar + /carla/detections on :9090
bash ~/carla-bridge/play_bag_detect.sh 5

# 2. ON THE LAPTOP: tunnel the server's rosbridge to localhost,
#    so the QEMU guest reaches it at ws://10.0.2.2:9090
ssh -N -L 127.0.0.1:9090:localhost:9090 <server>

# 3. ON THE LAPTOP: boot the AGL image in QEMU
./run-qemu.sh

# 4. ON THE LAPTOP: build + push the HMI into the guest, pointed at the tunnel
flutter run -d agl-qemu --dart-define=ROSBRIDGE_URL=ws://10.0.2.2:9090

The screenshot above is exactly that stack: the boxes in it were computed on the server from the recorded CARLA cloud, carried out as JSON over the tunnel, and drawn by the app inside the emulated AGL dashboard. The transport leg the project proved back in the early weeks — rosbridge into AGL — is the same one carrying the detections now. It just gained a second passenger.

4. The interface is the contract

There is a deliberate design choice hiding in the message type. The app does not know or care how the boxes were produced. It subscribes to vision_msgs/Detection3DArray — a ROS standard — and draws whatever detections arrive. Today those come from the classical PCL clustering node. Tomorrow they could come from a learned detector such as PointPillars. As long as the new detector publishes the same message on the same topic, the dashboard does not change at all.

5. Honest scope

Two things are worth stating plainly, so the screenshot is not read as more than it is.

• The boxes are still clusters, not named objects. Same caveat as last week: a box means points grouped together, so a wall or a pole can be boxed just like a car. Classification — saying what each box is — is still ahead.
• The detector runs beside the image, not inside it yet. In this run the perception node runs on the server and the AGL app renders what it publishes. Packaging the PCL detector into the AGL image itself, so the whole pipeline lives on the target, is the “ROS-embedded” step still on the list. What this week proves is the dashboard end: it can consume and draw real detections, live.

6. Where it stands, and what's next

Status: the perception result is now visible on the dashboard. The AGL Flutter HMI subscribes to the detector's standard detection topic and renders the 3D boxes over the live point cloud, verified end to end into an emulated AGL screen over a tunnelled rosbridge. The loop the project set out to build — cloud → detections → dashboard — is closed for the first time.

• Ship it. Fold the box-rendering into the application repository and the AGL demo so the packaged app draws detections out of the box.
• Embed the detector. Package the PCL node into the AGL image so the detections are produced on the target, not on a separate server.
• Make it smarter. Start the learned detector (PointPillars) as the parallel track, publishing the same message the dashboard already reads.

Week 13 gave the pipeline objects. Week 14 put them on the glass.

Links

• Application repository: github.com/shauryarane05/agl-lidar-hmi-app. The Flutter HMI (now drawing detections), the recorded rosbag, and the offline demo helper.
• Perception node: github.com/shauryarane05/agl-lidar-hmi-app/perception. The PCL detector publishing the vision_msgs/Detection3DArray the dashboard reads.