Custom Workloads

While FireMarshal provides some basic built-in workloads for getting started (described in Quick Start (Built-In Workloads)), the real power lies in its ability to define custom workloads.

A custom workload takes the form of a workload configuration file, and a directory containing any files needed by the workload. Additionally, workloads can be based on a parent workload to avoid duplicate work. These files can be anywhere you like, but FireMarshal must be able to find the workload descriptions. See Workload Search Paths for details on how FireMarshal searches for workloads.

We now walk through two examples of building a custom workload. For full documentation of every option, see Workload Specification.



In this example, we produce a 2-node workload that runs two benchmarks: quicksort and spam-filtering. This will require installing a number of packages on Fedora, as well as cross-compiling some code. The configuration is as follows:

  "name" : "example-fed",
  "base" : "fedora-base.json",
  "overlay" : "overlay",
  "host-init" : "",
  "guest-init" : "",
  "jobs" : [
      "name" : "qsort",
      "outputs" : [ "/root/run_result.csv" ],
      "run" : ""
      "name" : "pySort",
      "outputs" : [ "/root/run_result.csv" ],
      "run" : ""

The name field is required and (by convention) should match the name of the configuration file. Next is the base (fedora-base.json). This option specifies an existing workload to base off of. FireMarshal will first build fedora-base.json, and use a copy of its rootfs for example-fed before applying the remaining options. Additionally, if fedora-base.json specifies any configuration options that we do not include, we will inherit those (e.g. we will use the linux-config option specified by fedora-base).

Next come a few options that specify common setup options used by all jobs in this workload. The overlay option specifies a filesystem overlay to copy into our rootfs. In this case, it includes the source code for our benchmarks (see example-workloads/example-fed/overlay). Next is a host-init option, this is a script that should be run on the host before building. In our case, it cross-compiles the quicksort benchmark (cross-compilation is much faster than natively compiling).


# This script will run on the host from the workload directory
# (e.g. workloads/example-fed) every time the workload is built.
# It is recommended to call into something like a makefile because
# this script may be called multiple times.
echo "Building qsort benchmark"
cd overlay/root/qsort


Next is guest-init, this script should run exactly once natively within our workload. For example-fed, this script installs a number of packages that are required by our benchmarks. Note that guest-init scripts are run during the build process; this can take a long time, especially with fedora. You will see linux boot messages and may even see a login prompt. There is no need to login or interact at all, the guest-init script will run in the background. Note that ends with a poweroff command, all guest-init scripts should include this (leave it off to debug the build process).

# set -x

# This is an example of the sort of thing you might want to do in an init script.
# Note that this script will be run exactly once on your image in qemu.

# Note: you will see a bunch of fedora boot messages and possibly even a login
# prompt while building as this script runs. Don't worry about the login promt,
# your script is running in the background.

# In this case, we will use fedora's package manager to install something (the
# full-featured 'time' command to replace the shell builtin). We also use pip
# to install a python package used by one of the benchmarks. You can also
# download stuff, compile things that don't support cross-compilation, and/or
# configure your system in this script.

# Note that we call poweroff at the end. This is recomended because this script
# will be run automatically during the build process. If you leave it off, the
# build script will wait for you to interact with the booted image and shut
# down before it continues (which might be useful when debugging a workload).

echo "Installing the real time tool (not the shell builtin)"
dnf install -y time

echo "Installing the 'algorithms' python package for the PySort benchmark"
pip install algorithms


Finally, we specify the two jobs that will run on each simulated node. Job descriptions have the same format and options as normal workloads. However, notice that the job descriptions are much shorter than the basic descriptions. Jobs implicitly inherit from the root configuration. In this case, both qsort and spamBench will have the overlay and host/guest-init scripts already set up for them. If needed, you could override these options with a different base option in the job description. In our case, we need only provide a custom run option to each workload. The run option specifies a script that should run natively in each job every time the job is launched. In our case, we run each benchmark, collecting some statistics along the way, and then shutdown. Finishing a run script with poweroff is a common pattern that allows workloads to run automatically (no need to log-in or interact at all).

set -x

# This script will be run every time you boot the workload. In this case we're
# running a benchmark and recording some timing information into a csv that can
# be extracted later. Also note that we call poweroff at the end, if you would
# prefer to interact with the workload after it's booted, you can leave that off.

cd root/qsort
/usr/bin/time -f "%S,%M,%F" ./qsort 10000 2> ../run_result.csv

We can now build and launch this workload:

./marshal build workloads/example-fed.json
./marshal launch -j qsort workloads/example-fed.json
./marshal launch -j spamBench workloads/example-fed.json

You will see each job launch and can inspect the live input. The outputs and uart log will be saved in in the runOutputs directory with a name like runOutputs/example-fed-launch-2020-01-23--01-33-41-X17CZDQ4P3DW1QRI.

Bare-Metal Workload


FireMarshal was primarily designed to support linux-based workloads. However, it provides basic support for bare-metal workloads. Take test/bare.yaml as an example:

  "name" : "bare",
  "base" : "bare-base.json",
  "host-init" : "",
  "bin" : "hello",
  "testing" : {
    "refDir" : "refOutput"

This workload creates a simple “Hello World” bare-metal workload. This workload simply inherits from the “bare” distro in its base option. This tells FireMarshal to not attempt to build any linux binaries or rootfs’s for this workload. It then includes a simple host-init script that simply calls the makefile to build the bare-metal boot-binary. Finally, it hard-codes a path to the generated boot-binary. Note that we can still use all the standard FireMarshal commands with bare-metal workloads. In this case, we provide a testing specification that simply compares the serial port output against the known good output of “Hello World!”.

Next Steps

For more examples, you can look in the test/ directory. The full set of workload configuration options is documented at Workload Specification. You can also customize much of the default behavior of FireMarshal through config files or environment variables (see FireMarshal Configuration). Finally, the available commands and their options are documented in FireMarshal Commands.