The goal of building a workload is to produce a working boot binary and (optionally) a root filesystem to boot from. The same outputs are used for Spike, Qemu, and FireSim. The one exception is that Spike does not support a disk, so users may choose to create an initramfs-only version of their workload for Spike (that binary will boot on Qemu and FireSim as well). The build process proceeds as follows:
The first step is to make sure the workload’s base workload is ready. Marshal will first follow the dependency chain of bases and ensure that all dependencies are built before starting on the requested workload. Once the immediate parent is completed, Marshal begins the build process by create a copy of the parent’s root filesystem to use as the basis for the requested workload (the distros hard-code their rootfs’s to end the recursion).
Before doing anything else, Marshal runs the workload’s
(if any) to prepare the workload. This script is allowed to do anything it
wants, so we must run it early in the process in case it changes anything from
the linux kernel source to the root filesystem overlay.
We build the boot binary before finishing the rootfs because we may need to boot the workload in Qemu in order to build it. This step is skipped if the user provided a hard-coded boot binary.
Create Final Linux Configuration¶
Users provide only kernel configuration fragments that must be processed to
create the real linux configuration. We first run ‘make ARCH=riscv defconfig’
in the linux source directory (either default or user-provided). We then append
configuration options to include an initramfs (CONFIG_BLK_DEV_INITRD and
CONFIG_INITRAMFS_SOURCE), more on that below. We then call a script provided by
Linux to combine the kernel fragments
Build Platform Drivers¶
FireSim provides a number of non-standard devices that require custom linux
drivers. In particular, the block device driver is needed in order to boot a
working system. Instead of maintaining a custom fork of the linux kernel (and
requiring users to keep in sync with it), we provide a custom initramfs that
boots before your main system and loads the drivers.
The drivers for firesim are provided under
make modules_prepare in the linux source tree, and then compiles
each driver against the provided source. This happens on each new build to
ensure they receive the latest kernel source and configuration (especially
important if the workload provides a custom kernel). We currently do not
support alternative drivers, so any custom linux kernel must be compatible with
the default kernel with regard to these drivers.
Because some drivers must be loaded in order to boot, we package them into a
custom initramfs that is compiled into the kernel. Marshal generates this
archive by staging several filesystems at
disk/: contains a fully-functioning root filesystem with a busybox-based environment and an init script that knows to load drivers and look for a disk to boot from (either
/dev/vdafor qemu or
nodisk/: contains just the init script to load drivers (it must be combined with a working root filesystem).
drivers/: contains the platform drivers built earlier.
devNodes.cpio: A pre-built archive containing the
/dev/ttyspecial files. These require a special procedure to create so we only do it once and commit the result.
Marshal combines the needed initramfs sources in a temporary directory into a single cpio archive and configures the kernel to include this archive at boot time.
Note that for nodisk workloads, we additionally include the entire contents of
the workload’s rootfs into the initramfs. In this case, the init script in
wlutil/initramfs/nodisk/init simply loads the drivers and calls the
/sbin/init to finish booting.
Linux Kernel Generation and Linking¶
With all of the dependencies finished, we can finally compile the Linux kernel
and link it with the bootloader. While each workload can use a custom kernel
source, all workloads use the same bootloader.
The final linked sbi+linux+initramfs is coppied into
Marshal internally converts both the
overlay options into a
FileSpec objects that describe the source and destination paths. We
then mount the guest rootfs on
disk-mount/ using guestmount (see
Guestmount was used to remove the need for root permissions, but it
is somewhat slower and doesn’t play nice with Ubuntu. The mounting method can
be changed via the
mountImg() decorator in
Now that we have a working binary and root filesystem, we can run the user’s
guest-init script (if provided). We configure the image to run this script
on boot (see below for how), and boot exacly once in Qemu.
Run Script or Command¶
The final step is to apply the user’s
run script or
command options (if
any). For simplicity, commands are converted into a run script (stored in
wlutil/generated/_command.sh) before proceeding.
Run scripts are handled in a per-distro fashion (since distros acheive it in different ways). Marshal abstracts this by requesting that the distribution generate a “bootScriptOverlay” that we apply to the image. In Buildroot, this places the script in a known location and uses a hard-coded init script that runs it. Fedora has a systemd service that runs the script.