Build Process

wlutil/build.py

The goal of building a workload is to produce a working boot binary and (optionally) a root filesystem to boot from. The same outputs are used for Spike, Qemu, and FireSim. The one exception is that Spike does not support a disk, so users may choose to create an initramfs-only version of their workload for Spike (that binary will boot on Qemu and FireSim as well). The build process proceeds as follows:

Build Parents

The first step is to make sure the workload’s base workload is ready. Marshal will first follow the dependency chain of bases and ensure that all dependencies are built before starting on the requested workload. Once the immediate parent is completed, Marshal begins the build process by create a copy of the parent’s root filesystem to use as the basis for the requested workload (the distros hard-code their rootfs’s to end the recursion).

Host Init

Before doing anything else, Marshal runs the workload’s host-init script (if any) to prepare the workload. This script is allowed to do anything it wants, so we must run it early in the process in case it changes anything from the linux kernel source to the root filesystem overlay.

Build Binary

wlutil/build.py:makeBin()

We build the boot binary before finishing the rootfs because we may need to boot the workload in Qemu in order to build it. This step is skipped if the user provided a hard-coded boot binary.

Create Final Linux Configuration

Users provide only kernel configuration fragments that must be processed to create the real linux configuration. We first run ‘make ARCH=riscv defconfig’ in the linux source directory (either default or user-provided). We then append configuration options to include an initramfs (CONFIG_BLK_DEV_INITRD and CONFIG_INITRAMFS_SOURCE), more on that below. We then call a script provided by Linux to combine the kernel fragments (riscv-linux/scripts/kconfig/merge_config.sh).

Build Platform Drivers

wlutil/build.py:makeDrivers() FireSim provides a number of non-standard devices that require custom linux drivers. In particular, the block device driver is needed in order to boot a working system. Instead of maintaining a custom fork of the linux kernel (and requiring users to keep in sync with it), we provide a custom initramfs that boots before your main system and loads the drivers.

The drivers for firesim are provided under boards/firechip/drivers. Marshal first runs make modules_prepare in the linux source tree, and then compiles each driver against the provided source. This happens on each new build to ensure they receive the latest kernel source and configuration (especially important if the workload provides a custom kernel). We currently do not support alternative drivers, so any custom linux kernel must be compatible with the default kernel with regard to these drivers.

Generate Initramfs

wlutil/build.py:makeInitramfs()

Because some drivers must be loaded in order to boot, we package them into a custom initramfs that is compiled into the kernel. Marshal generates this archive by staging several filesystems at wlutil/initramfs{disk, nodisk, drivers}:

  • disk/: contains a fully-functioning root filesystem with a busybox-based environment and an init script that knows to load drivers and look for a disk to boot from (either /dev/vda for qemu or /dev/iceblk for firesim).
  • nodisk/: contains just the init script to load drivers (it must be combined with a working root filesystem).
  • drivers/: contains the platform drivers built earlier.
  • devNodes.cpio: A pre-built archive containing the /dev/console and /dev/tty special files. These require a special procedure to create so we only do it once and commit the result.

Marshal combines the needed initramfs sources in a temporary directory into a single cpio archive and configures the kernel to include this archive at boot time.

Note that for nodisk workloads, we additionally include the entire contents of the workload’s rootfs into the initramfs. In this case, the init script in wlutil/initramfs/nodisk/init simply loads the drivers and calls the target’s /sbin/init to finish booting.

Linux Kernel Generation and Linking

With all of the dependencies finished, we can finally compile the Linux kernel and link it with the bootloader. While each workload can use a custom kernel source, all workloads use the same bootloader (for now), located at riscv-pk/. The final linked bbl+linux+initramfs is coppied into images/workloadName-bin.

Build Rootfs

wlutil/build.py:makeImage()

Add Files

Marshal internally converts both the files and overlay options into a list of FileSpec objects that describe the source and destination paths. We then mount the guest rootfs on disk-mount/ using guestmount (see applyOverlay() and copyImageFiles() in wlutil/wlutil.py).

Note

Guestmount was used to remove the need for root permissions, but it is somewhat slower and doesn’t play nice with Ubuntu. The mounting method can be changed via the mountImg() decorator in wlutil/wlutil.py.

Guest Init

Now that we have a working binary and root filesystem, we can run the user’s guest-init script (if provided). We configure the image to run this script on boot (see below for how), and boot exacly once in Qemu.

Run Script or Command

The final step is to apply the user’s run script or command options (if any). For simplicity, commands are converted into a run script (stored in wlutil/generated/_command.sh) before proceeding.

Run scripts are handled in a per-distro fashion (since distros acheive it in different ways). Marshal abstracts this by requesting that the distribution generate a “bootScriptOverlay” that we apply to the image. In Buildroot, this places the script in a known location and uses a hard-coded init script that runs it. Fedora has a systemd service that runs the script.