The project's opening diagram has always ended in the same box: an AGL dashboard application drawing what the car sees. Weeks 1 through 8 built and proved everything that leads up to that box. The simulator, the perception, the bridge, the transport, and finally the move onto real Jetson hardware. But the box itself, the HMI, stayed a stand-in. Right up to last week the AGL side of the pipeline was a 73-line standard-library WebSocket client that printed numbers to a terminal. It proved the data arrived. It did not draw anything.
This week that box got built.
1. A dev loop for the dashboard
The way you build a car HMI is not by rebuilding the whole operating system image every time you change a colour. The workflow my mentor pointed at is the one real embedded UI teams use: write the app in an editor on your laptop, and push it live onto the running target, updating in place while it runs. Flutter calls this hot reload.
The catch is that the target is AGL, and I did not want app iteration to be blocked on the Jetson being on my desk and flashed. So I set up the exact same loop against AGL running in QEMU on the laptop, using the same AGL release the Jetson runs (unagi 21.0.1, qemux86-64), with the same Flutter engine version baked into the image. What works here works there.
Registering the QEMU AGL target as a Flutter device took some plumbing: a small set of helper scripts that push the built app bundle in over SSH, launch the AGL Flutter embedder on the compositor, and tunnel the debug port back so hot reload can attach. The engine version on the laptop has to match the one inside the image exactly or the reload handshake fails. Once that lined up, the payoff was immediate. Change a line, press one key, and the running dashboard updates in about a third of a second. That is the loop the rest of the week was built on.
Reduced to its commands, the loop is short:
# boot AGL in QEMU, forwarding SSH (2222) and the Dart VM debug port (1234)
qemu-system-x86_64 -enable-kvm -m 2048 -bios OVMF.fd \
-drive file=agl-flutter.wic,format=raw,if=virtio \
-device virtio-gpu-pci -display gtk,gl=on \
-device virtio-net-pci,netdev=n \
-netdev user,id=n,hostfwd=tcp::2222-:22,hostfwd=tcp::1234-:1234
# register the running VM as a Flutter target (one time)
flutter config --enable-custom-devices
flutter custom-devices add # ping / install / run / forward-port, all over SSH
# the actual loop: build, push into AGL, launch, attach for hot reload
flutter run -d agl-qemu
# edit lib/main.dart, press r -> live in ~0.3 sWhy this matters This is not a throwaway convenience. The Jetson deployment next week uses the identical mechanism, just pointed at a real board over the network instead of a virtual one over local ports. Proving the loop on the emulator means the only new variable on the Jetson is the Jetson.
2. What the app draws
The application is a bird's-eye LiDAR view. The car sits at the centre, forward is up, and the surrounding returns are plotted as points on a set of range rings out to 50 metres, coloured by distance so nearby objects read warm and far ones read cool. A slow radar sweep rotates over the top so the display looks alive even when a frame is holding. A status line reports how many points are in the current frame and the update rate.
It has two sources. A simulated mode generates a synthetic rotating scan of a corridor with moving obstacles, which is useful as an always-on demo that needs nothing connected. And a live mode that subscribes over rosbridge and draws real sensor frames. The interesting engineering is in the live path, because CARLA's LiDAR arrives as a sensor_msgs/PointCloud2, which rosbridge hands over as a base64-encoded blob of binary. The app decodes that blob directly: it reads the x, y, z and intensity floats out of each point by their declared byte offsets, drops the ground and roof returns by height so the top-down view stays clean, and projects the rest to the ground plane. That decoder is what turns the raw pipe from the earlier weeks into an actual picture.
The heart of that decoder, in the app's Dart code:
// rosbridge delivers PointCloud2.data as base64; each point is point_step bytes
final bytes = base64Decode(msg['data']);
final bd = ByteData.sublistView(bytes);
for (int k = 0; k < n; k++) {
final off = k * step;
final x = bd.getFloat32(off + ox, Endian.little);
final y = bd.getFloat32(off + oy, Endian.little);
final z = bd.getFloat32(off + oz, Endian.little);
if (z < minZ || z > maxZ) continue; // drop the ground and roof returns
points.add(project(x, y)); // onto the bird's-eye plane
}3. Feeding it a real CARLA scan
To drive the live mode with genuine data rather than the synthetic scene, I needed a CARLA LiDAR recording. There were none, so I made one. On the server, CARLA plus the two-process bridge from Week 5 publishes /carla/lidar, and a rosbag recording captured a stretch of it to a file. That file can now be replayed forever, with no CARLA and no GPU required, which is exactly what a repeatable demo wants.
Two things had to be worked around to get the bag. CARLA on the GPU-less server renders in software, so its LiDAR only completes about a third of a rotation per second: the raw capture is genuine but slow. Replaying the bag on a loop with a speed multiplier smooths that back up to a livelier rate for the display. And the original bridge script wedged on this server at CARLA's synchronous-mode handshake, so I wrote an asynchronous variant that streams the sensor without that step. With those in place, the bag loops into rosbridge, and the app, pointed at that stream, comes up green and draws the recorded scan.
Recording the bag and replaying it into the app is a handful of commands:
# server: CARLA + the two-process bridge publish /carla/lidar (PointCloud2)
python lidar_to_sock.py & # CARLA raycast lidar -> raw bytes on a socket
python sock_to_pointcloud2.py & # -> ROS2 /carla/lidar
ros2 bag record -o carla_lidar /carla/lidar # Ctrl-C to stop
# replay it forever, sped up to smooth the slow software-render capture
ros2 launch rosbridge_server rosbridge_websocket_launch.xml # serves ws://:9090
ros2 bag play carla_lidar --loop --rate 5
# laptop: forward the app's :9090 to the server's rosbridge, then press "Go LIVE"
ssh -N -L 9090:localhost:9090 user@server
4. The honest problem: it overflows
The application renders, it goes live, and it draws real data. What it does not yet do is fit on the screen. As the screenshot shows, the radar's centre and the bottom control bar run off the edge of the display.
The cause is specific. AGL's compositor forces the application to full screen at a fixed 1920 by 1080, and it ignores requests from the app to run at any other size. On the laptop, once the window's title bars are accounted for, the bottom of that 1080-tall surface is pushed past the edge and clipped. The content is all being drawn correctly, it is simply drawn onto a canvas taller than the visible area. Scaling the whole emulator display to fit the screen makes the full picture visible, but that is a workaround at the display layer, not a fix in the app.
Stand and deliver This is the current honest state. The hard parts are done: the dev loop works, the PointCloud2 decoder works, and real CARLA data reaches the dashboard and renders. The remaining defect is a layout and display-fit problem, not a pipeline problem. It is annoying, it is visible, and it is next week's first job.
5. Where it stands, and what's next
Status: the HMI application exists. It runs on AGL, it hot-reloads for fast iteration, and it draws a real recorded CARLA LiDAR scan in a live bird's-eye view over the same rosbridge transport the whole project is built on. The pipeline's final box is no longer a placeholder.
- Stop the overflow. Make the app lay out correctly within the compositor's fixed full-screen surface, so the whole radar and its controls are always visible without scaling tricks at the display layer.
- Deploy on the Jetson. Move this exact dev loop and application off the emulator and onto the real Orin Nano board, the same way Week 8 moved the transport leg onto real silicon. The app is version-matched to the board's AGL image on purpose, so this is a change of address, not a rewrite.
Week 8 named the HMI application as the largest open piece of the whole project. This week it stopped being open. What is left is to make it sit properly on the screen and to run it where it is meant to run.