Sending the kernel over USB from 9front

Boot via USB (Kernel) again

So far in my ClusterHAT journey, I was only able to control the HAT's USB ports from 9front. The options for booting were to either use an SD card, or boot via USB from a Linux host. In this part, let's tie it together by sending the boot payload from 9front, to enable a fully autonomous cluster.

The Pis have a rather unusual boot process, which is controlled by the GPU's firmware. While many in the open-source communities dislike that for various (probably justified) reasons, this is actually useful for dummies like me, having no clue how to boot up anything.

Let's have a look at what we need to run this boot procedure. While there's lots of good documentation in the Raspberry doc pages, the actual documentation for the protocol is the reference implementation usbboot. On 9front side, we'll use the nusb(2) library, a helper library for USB driver design. With that, let's give it a go!

Opening an USB device

When a Pi Zero cannot boot from SD card, it will enter the USB-boot device mode. On the host, we now need to open this device for sending commands, and ultimately the boot payload. The first thing to find out is, of course, which device to open. The ClusterHAT has 4 ports, which we expect to be active at roughly the same time. Also, we'd like to be able to send different boot payloads to the different Pis.

In the first iteration, I'm therefore skipping any automatic detection and expect the user to provide the pre-determined device number. After enabling a we can determine that by querying the USB driver:

cpu% hatctl p4 1
cpu% cat /dev/usb/ctl
[...]
ep6.0 enabled control rw speed high maxpkt 64 ntds 1 pollival 0 samplesz 0 hz 0 uframes 0 hub 5 port 1 rootport 1 addr 4 idle
vendor csp 0x0000ff vid 0x0a5c did 0x2764 Broadcom 'BCM2710 Boot' 701d1 xhci
ep6.1 enabled bulk w speed high maxpkt 512 ntds 1 pollival 0 samplesz 0 hz 0 uframes 0 hub 5 port 1 rootport 1 addr 4 idle
ep6.2 enabled bulk w speed high maxpkt 512 ntds 1 pollival 0 samplesz 0 hz 0 uframes 0 hub 5 port 1 rootport 1 addr 4 idle

We now have the "default pipe" or "control endpoint" of the Pi Zero 2 enumerated as ep6.0, and two USB function endpoints for bulk transfers as ep6.1 and ep6.2. (For the remainder of this article, I'll stick to the example of the Pi being enumerated as device "6".) We need both ep6.0 and ep6.1 for what we're about to do, and can access them in C by calling nusb like this:

#include "/sys/src/cmd/nusb/lib/usb.h"

typedef struct Pi
{
    Dev *ctl;
    Dev *bulkep;
} Pi;

int openPi(Pi *pi, char *n)
{
    /* expects the number of the already attached device,
       e.g. 6 if the pi is enumerated as ep6.0
    */
    Dev *d = getdev(n);
    if(d == nil){
        print("Could not getdev(%s): %r", n);
        return 1;
    }
    // Verify that we're trying to attach a Pi Zero
    if (d->usb->vid != 0xa5c || d->usb->did != 0x2764) {
        sysfatal("Not a supported Raspberry Pi!");
    }
    // Determine the USB Endpoint for Bulk transfers
    Ep *ep = nil;
    for (int i=0; i < nelem(d->usb->ep); i++){
        ep = d->usb->ep[i];
        if(ep == nil){
            continue;
        }
        if (ep->type == Ebulk && ep->dir == Eout) {
            break;
        }
    }
    print("Found bulk Endpoint at %d\n", ep->id);
    Dev *b = openep(d, ep);
    if(b == nil){
        sysfatal("Couldn't open bulk EP\n");
    }
    // Open the file used for bulk transfers
    if(opendevdata(b, OWRITE) < 0){
        sysfatal("couldn't open bulk data file\n");
    }

    pi->ctl = d;
    pi->bulkep = b;
    return 0;
}

Sending & receiving data via USB

I'd like to refer to the excellent USB in a NutShell page for a detailed introduction to USB matters. It's old and does not care about USB3 and beyond, but gives a good explanation of what's going on here. As teased above, we're going to write to two different endpoints.

As described here, the endpoints are closely related to the different transfer types. The "control transfer" is the core functionality used for sending commands and status information between host and device. We'll always use the control endpoint ep6.0 for this, which we've opened as pi->ctl. The nusb library provides the usbcmd() function as an abstraction around control transfers. For reading data back from the Pi, we'll use this helper function:

int pi_read(uchar *buf, int len, Pi *pi)
{
    int ret = usbcmd(pi->ctl, Rd2h|Rvendor, 0, len & 0xffff, len >> 16, buf, len);
    if (ret >= 0) {
        return len;
    } else {
        return ret;
    }
}

This call requests transfer of data from device to host (Rd2h) via a vendor-specific request (Rvendor & the following 0). We're transmitting the number of bytes to read via the len argument, and ask it to store the received data in buf. Since we'll only read small amounts of data, this is handled completely via control transfers.

On the other hand, we're going to have to send whole kernels over USB, which probably are going to be bigger than what control transfers allow. For this reason, the Pi Zero firmware provides the bulk endpoint, part of a USB function for receiving boot data. The helper function used for sending is using two steps:

int pi_write(uchar *buf, int len, Pi *pi)
{
    // Send size of payload
    int ret = usbcmd(pi->ctl, Rh2d|Rvendor, 0, len & 0xffff, len >> 16, nil, 0);
    if (ret != 0) {
        sysfatal("Failed writing to ctl file\n");
    }
    // Send payload in chunks
    int a_len = 0;
    while (len > 0) {
        int sending = MIN(len, 16 * 1024);
        ret = write(pi->bulkep->dfd, buf, sending);
        if (!ret) {
            break;
        }
        a_len += ret;
        buf += ret;
        len -= ret;
    }
    return a_len;
}

This is first doing a control transfer from host to device (Rh2d), but only sending the length, not any actual data. We'll be using the ep6.1 endpoint for that, which is opened as pi->bulkep in the C code. Specifically, there's a file abstraction provided by nusb, pi->bulkep->dfd, that we can write data to just like a regular file. (Note: I've stolen the 1024 chunk size from usbboot's calling sequence to libusb. No clue if it's necessary here...)

Boot process

The first step of booting is to send over the bootcode.bin payload. The info about the bootcode is stored in these structs:

typedef struct BootMessage
{
    int length;
    uchar signature[20];
} BootMessage;

typedef struct Payload
{
    BootMessage msg;
    uchar *buf;
} BootPayload;

We'll use the following code for sending the bootcode:

    BootPayload bootcode = openBootPayload(path);

    int res = pi_write((uchar*)&bootcode.msg, sizeof(bootcode.msg), &pi);
    if (res != sizeof(bootcode.msg)) {
        sysfatal("Failed writing bootcode size: %r\n");
    }

    res = pi_write(bootcode.buf, bootcode.msg.length, &pi);
    if (res != bootcode.msg.length) {
        sysfatal("Failed writing bootcode: %r\n");
    }
    int retcode = 0;
    res = pi_read((uchar*)&retcode, sizeof(retcode), &pi);
    if (res > 0 && retcode == 0) {
        print("Successfully read %d bytes\n", sizeof(retcode));
    } else {
        print("Failed : 0x%x\n", retcode);
    }

We're first transmitting the BootMessage containing the length of the payload, as well as an optional signature. While we're not using the signature here, the firmware still expects to receive the full 24 bytes.

Note: yes, these are two nested 2-step transfers. We're first transmitting the "length of the length" via control transfer, then the actual length via bulk transfer.

When the firmware knows how many bytes to expect, we can send the actual bootcode.bin file that we've read into a byte buffer. To end the first step of the boot protocol, we're requesting a return value.

When everything worked out fine to that point, the Pi resets itself for the second stage of the boot protocol. This means the USB device needs to be re-enumerated -> we'll need to close and re-open the Dev * structures.

When we have opened the second stage decriptors, we can continue with the protocol. Here, the firmware begins requesting specific files from us. We need to query the next step by reading structs like

typedef struct FileMessage
{
    int cmd;
    char fname[256];
} FileMessage;

As before, we'll transmit file size (cmd == 0) and file contents (cmd == 2) separately, only know we need to act on the Pi's request. This is done until the Pi is satisfied, in a loop like the following:

int booting = 1;
FILE *fp = nil;
while (booting) {
    res = pi_read((uchar*)&msg, 260, &pi);
    if (res < 0 || res != 260) {
        print("Error reading next boot request: %d, %r\n", res);
        sleep(1000);
        continue;
    }

    switch (msg.cmd) {
    case 0: // GetFileSize
        if (fp) fclose(fp);
        snprint(path, sizeof(path), "%s/%s", argv[2], msg.fname);
        fp = fopen(path, "rb");
        if (!fp) {
            print("Cannot open file %s\n", msg.fname);
            pi_write(nil, 0, &pi);
        } else {
            fseek(fp, 0, SEEK_END);
            fsize = ftell(fp);
            rewind(fp);

            res = usbcmd(pi.ctl, Rh2d|Rvendor, 0, fsize & 0xffff, fsize >> 16, nil, 0);
            if (res < 0) {
                sysfatal("Error sending size\n");
            }
        }
        break;
    case 1: // ReadFile
        if (!fp) {
            print("File %s not found\n", msg.fname);
            pi_write(nil, 0, &pi);
        } else {
            if (fsize == 0) {
                print("Empty file?\n");
            } else {
                uchar *buf = malloc(fsize);
                if (!buf) {
                    sysfatal("Error allocating buffer\n");
                }

                int read = fread(buf, sizeof(uchar), fsize, fp);
                if (read != fsize) {
                    sysfatal("Error reading file\n");
                }

                int written = pi_write(buf, fsize, &pi);
                free(buf);
                if (written != fsize) {
                    sysfatal("Error transfering file\n");
                }
            }
        }
        break;
    case 2:
        print("Done booting\n");
        booting = 0;
        break;
    }
}

This procedure iterates over the remaining files which are usually found in the FAT partition of a Pi SD card. Most notably, this'll be the config.txt, which tells the firmware which kernel to request for a specific Pi, followed by the 9pi3 kernel for our Pi Zero 2, and the cmdline.txt which will make 9front boot via TCP from the host.

Note the hardcoded 260 in the pi_read call. Turns out, sizeof(FileMessage) is 264 due to padding. The firmware is hardcoded to expect requests for 260 bytes however. Definitely did not take more than half an hour to find that mistake, thanks C...

This brings me one further step to my ideal system: I can now boot the kernel on the nodes via USB and fetch the userland via Wi-Fi. Now on to finding out how to boot the Pi Zero 1s without Wi-Fi - maybe implement USB networking?