AGL boots on the Jetson — winning the SD-card battle

Week 5 ended mid-fight. The board booted my AGL kernel off the SD card, then drowned in I/O error, dev mmcblk0, sector 0 and never mounted the root filesystem. My leading theory was a firmware/OS generation mismatch — the board's firmware was JetPack 7, my kernel was JetPack 6 — and the plan was a full USB-recovery flash to pull the firmware back down to the matching generation. This week I did exactly that. It did not fix it. What followed was the real debugging.

1. The recovery flash — the fix I was sure of

What I did: got the board into Force Recovery Mode (it enumerated as 0955 NVIDIA Corp. APX), and ran the meta-tegra flash bundle, which ships its own matching firmware. It rewrote the board's firmware and OS together as one coherent JetPack-6 system.

Flashing completed
Coldbooting the device

I checked the version off the boot log afterwards, and it was genuinely downgraded:

Piece	Before	After flash
Board firmware	39.2.0 (JetPack 7.2)	36.5.0 (JetPack 6.2)
AGL kernel	5.15-l4t-r36.5	5.15-l4t-r36.5

Firmware and kernel were now the same generation. By my theory, the SD errors should have been gone.

2. Same errors — my theory was wrong

The board booted, loaded the kernel, and hit the exact same flood:

blk_update_request: I/O error, dev mmcblk0, sector 0 (READ)
mmcblk0: unable to read partition table

3. Ruling things out, one at a time

With the tidy explanation gone, I went back to first principles and knocked out every suspect I could:

• The card? No. Same card mounted perfectly in my laptop — every partition readable.
• The firmware? No. Just realigned it to the exact matching generation; identical errors.
• Power? No. Chased a red herring here — a 5 V/5 A USB-C brick that wouldn't even power the board on (the Orin Nano needs a negotiating USB-C PD supply or a 7–20 V barrel jack, not a dumb 5 V brick). Ruled power out as the cause of the reads failing.
• The OS boot chain? No. The bootloader reads the very same card fine — that's how it loads the kernel in the first place.

That last point was the whole key. The firmware reads the card. The kernel cannot. Same card, same slot, same board, seconds apart — the only thing that changes between them is how each one drives the SD bus.

4. The real root cause — a 1.8V voltage-switch that fails

An SD card can run its data lines at two signaling voltages: a slow, universally-safe 3.3 V, or a fast 1.8 V UHS high-speed mode (SDR50/DDR50/SDR104). The bootloader is conservative — it stays at 3.3 V, so it always works. The Linux kernel is ambitious — it boots, sees a UHS-capable card, and immediately tries to renegotiate the bus up from 3.3 V to 1.8 V for speed.

So the fix isn't firmware, or power, or the card. It's telling the kernel to be as conservative as the bootloader: never leave 3.3 V.

5. The fix — one node in the device tree

The SD/MMC controller is a node in the board's device tree — mmc@3400000. I patched it to forbid the 1.8 V switch and hold the power rail steady, and stripped out the UHS high-speed modes so the kernel never even tries them:

mmc@3400000 {
    status = "okay";
    no-1-8-v;                    /* never switch to 1.8V UHS */
    nvidia,vmmc-always-on;       /* keep the card's power rail steady */
    /* removed: sd-uhs-sdr25 / sdr50 / ddr50 / sdr104 */
};

Deploying it without a full reflash: normally the kernel's device tree comes from a flashed partition, so testing a change would mean rebuilding and reflashing every time. Instead I used an FDT override — I compiled the patched device tree, dropped it on the card's boot partition, and added one line to extlinux.conf pointing the bootloader at it:

LINUX /boot/Image
INITRD /boot/initrd
FDT /boot/kernel_tegra234-...-nv-super-fixed.dtb

Now the bootloader hands the kernel my device tree instead of the flashed one — a full test-fix loop in seconds instead of a rebuild.

6. It's alive — a login shell on real hardware

Reinserted the card, powered on, watched the serial console. No error flood. The kernel mounted root and dropped straight to:

Automotive Grade Linux 21.0.1 jetson-orin-nano-devkit ttyTCU0
jetson-orin-nano-devkit login:

Logged in as root and confirmed the whole stack end to end:

$ cat /etc/os-release
ID=agl
VERSION="21.0.1 (unagi)"
$ uname -r
5.15.185-l4t-r36.5-1033.33+g9c6d5c8154da
$ df -h /
/dev/mmcblk0p1   ...  /   (read-write)

7. Where the fix actually came from

There was no single guide for this — I want to be clear about that, because it's the part I'm most proud of. The mechanism (a failed 3.3 V→1.8 V switch) I reasoned out from the symptom: bootloader reads, kernel doesn't, only the bus signaling differs. nvidia,vmmc-always-on came from a matching NVIDIA developer-forum thread on Orin SD instability. no-1-8-v is a standard property from the upstream Linux MMC device-tree binding, not anything Jetson-specific. Assembling those three facts into the fix was the work.

And to a question my mentor asked — it's not specific to the Orin Nano "Super." The change targets the SDMMC controller node, which is identical across the whole Orin Nano / NX devkit family; only the device tree filename carries the "super" tag. It's a known Orin-family SD signal-integrity quirk, reported on plain Orin Nano boards too.

8. Where it stands, and what's next

Status: the target board is fully up — AGL boots to userspace on real silicon. That closes out the entire hardware bring-up arc that started back in Week 3 with a Raspberry Pi that wouldn't boot.

Two loose ends, then the real goal:

• Make the fix permanent. The device tree patch currently lives on one card via the FDT override; I'll bake it into the machine config on the build server so every image ships with it and boots cleanly with no manual step.
• Room to grow. The rootfs is tiny and nearly full — expand it to fill the card so there's space to build on.

Bare-metal debugging is a different discipline from writing code. The lesson that'll stick from this week isn't the device tree property — it's watching a clean, confident theory die on contact with the hardware, and having to trust the evidence over the story I'd already told myself.