Bringing live perception from a self-driving car simulator onto the operating system that runs in real car dashboards — over ROS2 and rosbridge.
Modern cars are full of sensors. One of the most important is LiDAR — a spinning laser that measures the distance to everything around the vehicle, producing a 3D point cloud of the world. Software then turns that cloud into something useful: “there's a car 12 metres ahead, a pedestrian to the left.” Finally, the dashboard draws it on a screen so the driver (or the system) can see what the car sees.
This project builds that whole journey — but in simulation, so it can be developed and tested without a physical car:
Each box is a real, separate technology, and getting them to talk to each other — cleanly, reproducibly, and under real-world constraints — is the heart of the work.
It's worth being honest about the gap between the textbook deployment and the one we could actually run. The ideal is three machines, each doing the job it's suited for, on a real network:
We couldn't reach that: the target single-board computer (a Raspberry Pi 4B) wouldn't boot due to a hardware fault — a Jetson Nano has since been ordered as its replacement — and the only machine large enough to run CARLA was the GPU-less server. So everything collapsed onto one box, with AGL emulated in QEMU and a lightweight custom bridge. The table shows what's faithful to the goal and what is still a stand-in:
| Dimension | Ideal | What we built |
|---|---|---|
| Machines | workstation + target + GPU box | one server |
| AGL runs on | real automotive hardware | QEMU emulation (qemux86-64) |
| Sensor data | LiDAR point cloud → detections | LiDAR point cloud on AGL (+ odometry); PCL detections as PoC |
| Network | real LAN | QEMU slirp NAT via the host gateway |
| CARLA→ROS2 bridge | official carla_ros_bridge | custom two-process bridge |
| AGL client | HMI application (bird's-eye view) | standard-library WebSocket client |
What's genuinely proven: the transport leg — rosbridge → AGL — is identical to the ideal, and it now carries a live CARLA LiDAR point cloud all the way into an AGL VM (see Week 3). The remaining pieces (the HMI application, real target hardware, the final perception algorithms) all plug into that same proven pipe without changing its shape — they swap the cargo, not the channel.
This site documents the work in order. Start at Community Bonding and follow through the weeks.
Locking the architecture (and why rosbridge, not DDS), the two-Python-version puzzle, and standing up CARLA + the AGL/ROS2 toolchain on a GPU-less server.
Compiling the AGL+ROS2 image (and the recipe bug that blocked it), standing up rosbridge, a pure-stdlib WebSocket client, and proving an AGL VM can subscribe over QEMU networking.
Wiring the real simulator into the pipeline, fixing three runtime bugs, and watching the AGL dashboard track a self-driving car live at 20 Hz.
A Raspberry Pi target that wouldn't boot (Jetson Nano ordered), and a live CARLA LiDAR point cloud reaching AGL — produced on a server with no GPU at all.
Cleaning up the code into a public repository, and reviving the Jetson Nano after outdated firmware blocked Linux from booting — the target board is now building and booting.
Flashing the hand-built AGL image onto the Jetson and watching it boot on real silicon — then chasing down an unreadable SD card and a firmware that turned out to be a whole JetPack generation too new.
The SD-card battle, won: disproving my own firmware-mismatch theory and tracking the real cause to a failing 1.8V voltage-switch — a one-line device tree fix that boots AGL 21 to a login shell on real hardware.
Blocked on hardware in transit: repo cleanup, and settling the architectural question the project had been avoiding — three blockers that turned out to be one reflash, and why moving to NVMe makes the whole Week 6 device-tree saga irrelevant.
The move to NVMe, and the whole QEMU pipeline replicated on real hardware — the live CARLA LiDAR stream now lands on an AGL board over a real network instead of an emulated VM.
Building the real HMI: a Flutter bird's-eye LiDAR app on AGL, a hot-reload dev loop into a QEMU target, and a recorded CARLA point cloud replayed into it live over rosbridge. It renders real data, but the app overflows the screen.
Moving the dashboard onto the real Jetson Orin Nano. The app was the easy part: getting the compositor to draw meant fixing NVIDIA kernel mode-setting, and getting the app on screen meant learning how AGL hands out the display. It ends running full screen on real hardware, hot-reloading over a cable.
A real GPU workstation joins the project: CARLA now runs with a car I steer by hand, its LiDAR streaming live through the official carla_ros_bridge instead of a hand-rolled one. Plus the ordering rule that keeps it from crashing, a memory trap, and moving the dashboard onto the standard community ROS client.
Turning the dashboard from a laptop demo into something that packages the AGL way: a Jetson Orin workspace config, the recipe rebuilt on inherit flutter-app agl-app, and a recorded rosbag that makes the demo run fully offline. It ends with the first maintainer review of the recipe, and it passed.
The perception step: a PCL node segments the ground out of the CARLA LiDAR cloud and clusters what is left into 3D bounding boxes, nine to sixteen objects a frame. Shown as a bird's-eye animation, run fully offline from the recorded bag on a server with no display.
The boxes finally land on the glass: the AGL Flutter HMI now subscribes to the detector's Detection3DArray and draws the 3D boxes live over the point cloud — run end to end into an emulated AGL screen over a tunnelled rosbridge. Cloud → detections → dashboard, closed for the first time.
A lighter fortnight around exams: I finalised the recording into a self-contained reproducible bundle (LiDAR + two cameras + ground truth) and verified the whole perception pipeline now runs from it with no CARLA and no GPU: detections at ~32 Hz straight off the bag. Plus a round of polish on the HMI. Next: a clean, upstreamable app.
The dashboard now fuses three panels in one AGL screen: the LiDAR radar with detection boxes, plus a forward and an aerial camera, all over the same rosbridge and kept in sync with JPEG. Plus the packaging work to auto-start the demo inside the built image (a systemd unit and a build-tree recipe fix).