Integration with Forges

Now that you have installed CI-tron, you quite likely would like to expose your test machines (DUTs) on your projects’ GIT forges.

The forge integration has been designed with the following goals in mind:

Sane, but partially-or-fully overridable defaults: Allows the same job to run on every DUT in the farm, with the possibility to opt-in more control only when and where needed.

As seamless as possible: running on a CI-tron runner should be as similar to running on the default CI runners as possible;

The default boot configuration can be found in executor/server/src/valve_gfx_ci/executor/server/templates/boots_db.yml.j2.

executor/server/src/valve_gfx_ci/executor/server/templates/boots_db.yml.j2

#
# This file defines how to boot the various computers supported by CI-tron.
#
# Its goal is to provide sane boot defaults that enables running the same
# job on all the supported hardware, while also supporting jobs selectively
# opting-out of the defaults to get a more complete control over their boot
# configuration.
#
# To achieve this goal, we make use of the feature that enables splitting lists
# into categories. Categories that are not referenced by a job will be
# inherited automatically while uncategorized elements are evaluated first, so
# as to reduce the risks of user confusion. With that in mind, here are the
# rules on how to add support for new hardware:
#
# * Only use one of the following categories (no re-naming allowed):
#
#   * 50-platform: Anything necessary to boot the machine before reaching the
#       bootloader stage, or firmware that is expected by the platform and is
#       not otherwise part of the linux-firmware project, such as the signed
#       and non-redistributable firwmare files found on Qualcomm hardware.
#
#   * 50-bootloader: Anything necessary to load the environment as specified by
#       the `kernel`, `initramfs`, and `dtb` deployment options. Place in this
#       category all the bootloader configuration files.
#
#   * 50-kernel-modules: Anything necessary to make the default kernel boot to
#       a usable environment (ie. kernel modules, firmware, ...). Nothing from
#       this category should be included by default by a job that overrides the
#       default kernel.
#
# * Make the default deployment boot into a boot2container environment,
#     and be overridable without needing to set a category. In other words,
#     set the default boot2container initramfs in the `uncategorised` section.
#

{% macro b2c_url(goarch) -%}
https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.9.16/downloads/initramfs.linux_{{ goarch | replace("386", "amd64") }}.cpio.xz
{%- endmacro %}

{% macro kernel_url(goarch) -%}
https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.9.16/downloads/linux-{{ goarch | replace("386", "x86_64") | replace("amd64", "x86_64") }}
{%- endmacro %}

{% macro dtb_url(goarch) -%}
https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.9.16/downloads/linux-{{ goarch | replace("386", "x86_64") | replace("amd64", "x86_64") }}.dtbs.cpio.xz
{%- endmacro %}

dhcp:
  ipxe:
    match:
      user_class: iPXE
      architecture:
        - x86
        - x86_64
        - arm32
        - arm64
    defaults:
      dhcp:
        options:
          bootfile: {{ job.http.url }}/boot/boot.ipxe
          hostname: "{{ dut.hostname }}"
      storage:
        http:
          50-bootloader:
            - path: /boot/boot.ipxe
              data: |
                #!ipxe

                set semicolon:hex 3b
                set ampersand:hex 26
                set pipe:hex 7C
                set cmdline {% if dhcp_request.firmware.name == "UEFI" %}initrd=initrd{% endif %} {{ job.deployment.kernel.cmdline }}

                echo
                echo Downloading the kernel
                echo <ipxe> kernel {{ job.http.path_to("kernel") }}
                echo <ipxe> cmdline ${cmdline}
                kernel {{ job.http.path_to("kernel") }} ${cmdline}

                echo
                echo Downloading the initrd
                echo <ipxe> initrd --name initrd {{ job.http.path_to("initramfs") }}
                initrd --name initrd {{ job.http.path_to("initramfs") }}

                echo Booting!
                boot

      kernel:
        url: "{{ kernel_url(dhcp_request.architecture.goarch) }}"
      initramfs:
        url: "{{ b2c_url(dhcp_request.architecture.goarch) }}"

  uboot:
    match:
      firmware: uboot
      architecture:
        - x86_64
        - arm32
        - arm64
        - riscv64
    defaults: &uboot-defaults
      storage:
        tftp:
          50-bootloader:
            - path: &uboot_script_path /boot.scr.uimg
              format:
                uboot:
                  architecture: {{ dhcp_request.architecture.value | default("unknown")}}
                  compression: none
                  os: linux
                  type: script
              data: |
                echo Loading the kernel
                setenv bootargs '{{ job.deployment.kernel.cmdline | replace('\n', ' ') }}'
                tftpboot ${kernel_addr_r} {{ job.tftp.path_to("kernel") }}

                # The best compression ratio I found was 5.2x (XZ with x86_64), so let's go with 8x
                # NOTE: We use a PoT kernel_comp_size to align the subsequent artifact as much as possible
                setexpr kernel_comp_addr_r ${kernel_addr_r} + ${filesize}
                setexpr kernel_comp_size ${filesize} * 8
                echo Loading the compressed kernel at ${kernel_comp_addr_r}:${kernel_comp_size}

                {% if job.deployment.dtb %}
                  setenv fdt_file {{ job.tftp.path_to("dtb") }}
                {% endif %}

                setexpr fdt_addr ${kernel_comp_addr_r} + ${kernel_comp_size}
                echo Loading the fdt ${fdt_file} at ${fdt_addr}
                if test -n "${fdt_file}" && tftpboot ${fdt_addr} ${fdt_file}
                then
                  setexpr ramdisk_addr_r ${fdt_addr}} + ${filesize}
                else
                  setenv ramdisk_addr_r ${fdt_addr}
                  setenv fdt_addr ${fdtcontroladdr}
                fi

                echo Loading the ramdisk at ${ramdisk_addr_r}
                tftpboot ${ramdisk_addr_r} {{ job.tftp.path_to("initramfs") }}
                setenv ramdisk_size ${filesize}

                echo Booting!
                booti ${kernel_addr_r} ${ramdisk_addr_r}:${ramdisk_size} ${fdt_addr}
                bootz ${kernel_addr_r} ${ramdisk_addr_r}:${ramdisk_size} ${fdt_addr}

            - path: /pxelinux.cfg/.*
              data: |
                default ci-tron
                label ci-tron
                kernel {{ job.tftp.path_to("kernel") }}
                append {{ job.deployment.kernel.cmdline }}
                initrd {{ job.tftp.path_to("initramfs") }}
                {% if job.deployment.dtb %}
                  fdt {{ job.tftp.path_to("dtb") }}
                {% endif %}

          50-platform:
            # Given that U-boot may potentially depend on the updated DTB
            - path: /dtbs?/(.*)
              url: "{{ dtb_url(dhcp_request.architecture.goarch) }}"
              format:
                archive:
                  match: boot/dtbs/\1
      dhcp:
        options:
          bootfile: *uboot_script_path
          hostname: "{{ dut.hostname }}"
      kernel:
        url: "{{ kernel_url(dhcp_request.architecture.goarch) }}"
      initramfs:
        url: "{{ b2c_url(dhcp_request.architecture.goarch) }}"

  raspberrypi_arm64_downstream:
    match:
      mac_address: &raspberrypi_mac_addrs
        - b8:27:eb:.*
        - 2c:cf:67:.*
        - d8:3a:dd:.*
        - dc:a6:32:.*
        - e4:5f:01:.*
      # NOTE: One can use the UUID field from the DHCP query to identify an RPi 4 and 5.
      # Src: https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#DHCP_OPTION97
      # uuid: "35695052-.*"
    defaults:
      storage:
        tftp:
          50-bootloader:
            - path: /config.txt
              data: |
                [all]
                arm_64bit=1
                enable_uart=1
                kernel={{ job.tftp.path_to("kernel") }}
                initramfs {{ job.tftp.path_to("initramfs") }} followkernel

                [pi5]
                dtoverlay=vc4-kms-v3d-pi5
                dtparam=pciex1_gen=3
                dtoverlay=disable-bt
                dtparam=uart0_console

                [pi4]
                dtoverlay=disable-bt
                dtoverlay=vc4-kms-v3d-pi4

                [pi3]
                dtoverlay=disable-bt
                dtoverlay=vc4-kms-v3d
                # NOTE: File sizes should be limited to 32MB not to exceed the
                # maximum TFTP transfer size using the default 512 bytes block size

                [pi2]
                dtoverlay=disable-bt
                dtoverlay=vc4-kms-v3d
                # NOTE: File sizes should be limited to 32MB not to exceed the
                # maximum TFTP transfer size using the default 512 bytes block size

            - path: /cmdline.txt
              data: >-
                modules_load=overlay,vc4,v3d,snd_soc_hdmi_codec,i2c-brcmstb,i2c-bcm2835,nbd {{ job.deployment.kernel.cmdline}}

          50-platform:
            # This allow serving all the binaries, DTBs, and DTB overlays required to boot the wanted kernel
            - path: /(.*)
              # NOTE: Zip files are more efficient for seek times
              url: &rpi_firmware "https://github.com/raspberrypi/firmware/archive/b154632e320b87ea95c6ce8b59f96dbbe523ecf1.zip"
              format:
                archive:
                  match: firmware-.*/boot/\1
      dhcp:
        options:
          hostname: "{{ dut.hostname }}"

          # Keep the spaces! Src: https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#dhcp-request-reply
          43: "Raspberry Pi Boot   "
      kernel:
        url: *rpi_firmware
        format:
          archive:
            match: firmware-.*/boot/kernel8.img
      initramfs:
        categories:
          50-kernel-modules:
            - url: *rpi_firmware
              format:
                archive:
                  extension: "cpio"
                  keep:
                    - path: firmware-.*/(modules/.*-v8\+/(modules|kernel/(fs/overlayfs|sound|drivers/(video|gpu|media/cec/core|i2c/busses/i2c-.*|block/nbd.ko))).*)
                      rewrite: usr/lib/\1
        uncategorised:
          - url: "{{ b2c_url('arm64') }}"

  bios_chainload_ipxe:
    match:
      firmware: bios
    defaults:
      storage:
        tftp:
          50-bootloader:
            - path: &bios_ipxe_path /boot/ipxe.kpxe
              url: "https://downloads.gfx-ci.steamos.cloud/ipxe-dut-client/2024-01-30_20-41-29-mupuf-i386-undionly.kpxe"
      dhcp:
        options:
          bootfile: *bios_ipxe_path
          hostname: "{{ dut.hostname }}"

  uefi_tftp_chainload_ipxe:
    match:
      firmware: uefi
      protocol: tftp
      architecture:
        - x86_64
        - arm32
        - arm64
    defaults:
      storage:
        tftp:
          50-bootloader:
            - path: &efi_ipxe_path /boot/ipxe.efi
              url: "https://downloads.gfx-ci.steamos.cloud/ipxe-dut-client/2024-01-30_20-41-29-mupuf-{{ dhcp_request.architecture.ipxe_buildarch }}-snponly.efi"
      dhcp:
        options:
          bootfile: *efi_ipxe_path
          hostname: "{{ dut.hostname }}"

  uefi_http_chainload_ipxe:
    match:
      firmware: uefi
      protocol: http
      architecture:
        - x86_64
        - arm32
        - arm64
    defaults:
      storage:
        http:
          50-bootloader:
            - path: /boot/ipxe.efi
              url: "https://downloads.gfx-ci.steamos.cloud/ipxe-dut-client/2024-01-30_20-41-29-mupuf-{{ dhcp_request.architecture.ipxe_buildarch }}-snponly.efi"
      dhcp:
        options:
          bootfile: {{ job.http.url }}/boot/ipxe.efi
          hostname: "{{ dut.hostname }}"

  # For platforms that are not supported by iPXE, let's assume that the bootloader
  # is U-Boot and let it boot using the extlinux/pxelinux flow
  uefi_tftp_using_uboot:
    match:
      firmware: uefi
      protocol: tftp
      architecture:
        - riscv64
    defaults: *uboot-defaults


fastboot:
  hdk_8550:
    match:
      variables:
        product: kalama
    defaults:
      fastboot:
        header_version: 4
        os_version: 14.0.0
        os_patch_level: 2023-10
      kernel:
        url: https://fs.mupuf.org/hdk8650/linux-6.8-hdk8650-no-fw.gz
        cmdline:
          50-platform:
            - pd_ignore_unused clk_ignore_unused
            - nvme_core.default_ps_max_latency_us=0      # Disable NVME power management
            - g_cdc.dev_addr={{ dut.mac_address }}
            - androidboot.serialno={{ dut.id }}
      initramfs:
        categories:
          50-platform:
            - url: https://fs.mupuf.org/hdk8650/sm8550-hdk-firmware.cpio.xz
        uncategorised:
          - url: "{{ b2c_url("arm64") }}"
      dtb:
        url: "https://fs.mupuf.org/hdk8650/sm8650-hdk.dtb"

  hdk_8650:
    match:
      variables:
        product: pineapple
    defaults:
      kernel:
        url: https://fs.mupuf.org/hdk8650/2025-01-23-msm-gpu-fault-fixes-msm-next-6dbabc6/linux-arm64
        cmdline:
          50-platform:
            - pd_ignore_unused clk_ignore_unused
            - nvme_core.default_ps_max_latency_us=0      # Disable NVME power management
            - g_cdc.dev_addr={{ dut.mac_address }}
      initramfs:
        categories:
          50-platform:
            - url: https://fs.mupuf.org/hdk8650/sm8650-hdk-firmware.cpio.xz
        uncategorised:
          - url: "{{ b2c_url("arm64") }}"
      dtb:
        url: 'https://fs.mupuf.org/hdk8650/2025-01-23-msm-gpu-fault-fixes-msm-next-6dbabc6/linux-arm64.dtbs.cpio.xz'
        format:
          archive:
            match: 'boot/dtbs/qcom/sm8650-hdk.dtb'

We currently only support GitLab, but support for more forges is planned and welcomed.

GitLab

To expose your DUTs on a GitLab instance, you will need to add define it in MarsDB, in the gitlab: element. Check out MarsDB’s YAML code block for the details.

.ci-tron-job-v1

Warning

The interface is not final and is not considered stable. Make sure to pin the commit from which you import this file from, along with explicitly setting CI_TRON_JOB_TEMPLATE.

The low-level CI-tron integration GitLab CI job which you can use as the base for your CI jobs.

Warning

This job is more suited towards low-level bootloader/kernel/initrd testing. If you are looking for a more transparent way to run your GitLab jobs on bare-metal, please have a look at .ci-tron-b2c-job-v1 instead.

Including the job template

If your project is hosted on gitlab.freedesktop.org, use this:

# WARNING: it is strongly advised to use a commit hash instead of ``main``
variables:
  CI_TRON_TEMPLATE_PROJECT: &ci-tron-template-project gfx-ci/ci-tron
  CI_TRON_JOB_TEMPLATE_PROJECT_URL: https://gitlab.freedesktop.org/$CI_TRON_TEMPLATE_PROJECT
  CI_TRON_JOB_TEMPLATE_COMMIT: &ci-tron-template-commit main

include:
  - project: *ci-tron-template-project
    ref: *ci-tron-template-commit
    file: '/.gitlab-ci/dut.yml'

Projects hosted elsewhere unfortunately have to specify the full URL in the include:

# WARNING: it is strongly advised to use a commit hash instead of ``main``
# WARNING: be extra careful to always keep the variables and the include URL in sync!
variables:
  CI_TRON_JOB_TEMPLATE_PROJECT_URL: https://gitlab.freedesktop.org/gfx-ci/ci-tron
  CI_TRON_JOB_TEMPLATE_COMMIT: main

include:
  - remote: 'https://gitlab.freedesktop.org/gfx-ci/ci-tron/-/raw/main/.gitlab-ci/dut.yml'

Introductory example

stages:
  - integration testing

variables:
  # WARNING: it is strongly advised to use a commit hash instead of ``main``
  CI_TRON_JOB_TEMPLATE_PROJECT_URL: https://gitlab.freedesktop.org/gfx-ci/ci-tron
  CI_TRON_JOB_TEMPLATE_COMMIT: main

include:
  - remote: 'https://gitlab.freedesktop.org/gfx-ci/ci-tron/-/raw/main/.gitlab-ci/dut.yml'

# Print Hello World in the console log, then power off
test:
  stage: integration testing
  extends:
    - .ci-tron-job-v1
  tags:
     - cpu:arch:x86_64
     - farm:$FARM_NAME
  variables:
    CI_TRON_PATTERN__SESSION_END__REGEX: '^.*It''s now safe to turn off your computer\r?$'
    CI_TRON_PATTERN__JOB_SUCCESS__REGEX: 'Execution is over, pipeline status: 0\r?$'
    CI_TRON_KERNEL__URL: https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.9.15.1/downloads/linux-x86_64
    CI_TRON_INITRAMFS__B2C__URL: https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.9.15.1/downloads/initramfs.linux_amd64.cpio.xz
    CI_TRON_KERNEL_CMDLINE__SALAD: "SALAD.machine_id={{ dut.id }} console={{ dut.local_tty_device }},115200"
    CI_TRON_KERNEL_CMDLINE__B2C_NTP_PEER: "b2c.ntp_peer=ci-gateway"
    CI_TRON_KERNEL_CMDLINE__B2C_RUN: b2c.run="-t {{ fdo_proxy_registry }}/gfx-ci/ci-tron/executorctl:latest echo Hello world"
    CI_TRON_KERNEL_CMDLINE__B2C_POWEROFF: b2c.poweroff_delay=15

Timeouts

CI-tron exposes multiple sorts of timeouts:

OVERALL: This limits the overall time the job can take to complete.

INFRA_SETUP: This limits how long the setup phase, used to download artifacts or create NBD drives, can take.

BOOT_CYCLE: This limits how long one boot -> test -> shutdown cycle can take.

CONSOLE_ACTIVITY: This limits the time between we receive 2 characters in the serial output.

FIRST_CONSOLE_ACTIVITY: This limits the time it can take to receive the first console log.

INFRA_TEARDOWN: This limits how long the tearing down phase, used for the client to download the generated artifacts, can take.

CI_TRON_TIMEOUT__OVERALL__MINUTES: Number of minutes the job can take to complete, across reboots. Like the other timeouts below, minutes is a floating point number; eg. 0.5 = 30 seconds.

CI_TRON_TIMEOUT__BOOT_CYCLE__MINUTES: Number of minutes one boot -> test -> shutdown cycle can take.

CI_TRON_TIMEOUT__BOOT_CYCLE__RETRIES: Number of times the BOOT_CYCLE timeout can trigger a reboot before causing the end of the job.

CI_TRON_TIMEOUT__CONSOLE_ACTIVITY__MINUTES: Number of minutes the job can go without receiving any new console activity, after receiving the first byte.

CI_TRON_TIMEOUT__CONSOLE_ACTIVITY__RETRIES: Number of times the CONSOLE_ACTIVITY timeout can trigger a reboot before causing the end of the job.

CI_TRON_TIMEOUT__FIRST_CONSOLE_ACTIVITY__MINUTES: Number of minutes the job can wait for the first byte to arrive on the console.

CI_TRON_TIMEOUT__FIRST_CONSOLE_ACTIVITY__RETRIES: Number of times the FIRST_CONSOLE_ACTIVITY timeout can trigger a reboot before causing the end of the job.

Watchdogs

Watchdogs are user-defined timeouts that can be used to enforce sections of execution make progress within the expected period.

In the following attributes, replace SHORT_NAME with whatever name you want to give to your watchdog.

CI_TRON_WATCHDOG__SHORT_NAME__MINUTES: Number of minutes that can happen between the CI_TRON__BOOT_CYCLE__WD_START_REGEX regular expression is matched and the CI_TRON__BOOT_CYCLE__WD_STOP_REGEX one is matched. The timeout gets automatically reset to when CI_TRON__BOOT_CYCLE__WD_RESET_REGEX is matched.

CI_TRON_WATCHDOG__SHORT_NAME__RETRIES: Number of times this timeout can be reset through a reboot before causing the end of the job.

Console Patterns

Console patterns are regular expressions that, when matched, influence the execution of the job.

CI_TRON_PATTERN__SESSION_END__REGEX

This variable is required

A regular expression that, when matched, indicates that the job execution is over

CI_TRON_PATTERN__JOB_SUCCESS__REGEX

This variable is required

A regular expression that, when matched, indicates that the job succeeded

CI_TRON_PATTERN__JOB_WARN__REGEX: A regular expression that, when matched, indicates that something went a little off during the execution of the job. The job can still complete, but the GitLab job will fail when matched, even if CI_TRON_PATTERN__JOB_SUCCESS__REGEX is matched.

CI_TRON_PATTERN__SESSION_REBOOT__REGEX: A regular expression that, when matched, indicates that the test machine requires a reboot. This will increase the BOOT_CYCLE timeout’s retry counter, which may abort the run if the current value exceeds CI_TRON_TIMEOUT__BOOT_CYCLE__RETRIES.

CI_TRON_WATCHDOG__SHORT_NAME__START_REGEX: A regular expression that, when matched, starts the SHORT_NAME watchdog, as set by CI_TRON_WATCHDOG__SHORT_NAME__MINUTES.

CI_TRON_WATCHDOG__SHORT_NAME__RESET_REGEX: A regular expression that, when matched, resets the SHORT_NAME watchdog timeout to its original value.

CI_TRON_WATCHDOG__SHORT_NAME__STOP_REGEX: A regular expression that, when matched, disables the SHORT_NAME watchdog until CI_TRON_WATCHDOG__SHORT_NAME__START_REGEX is matched again.

Container Image Store

CI-tron has the ability to download, cache, and share container images with test machines.

Warning

Due to the global/shared nature of the image store, you will need to provide the platform that should be used to pull the container as it may differ from the platform used to run CI-tron.

Additionally, since other jobs may pull containers during the execution of your job, you really should reference the image not by its label (which may be overwritten by a later pull), but by its image id: {{ job.imagestore.public.short_name.image_id }}.

The information needed for the DUT to mount the image store can be found at {{ imagestore.mount("public").nfs }}. It contains the following fields:

type: The filesystem mount’s type (always NFS for now);
src: The filesystem mount’s source;
opts: The filesystem mount options;
to_b2c_filesystem(name): A helper that converts these mount options into a Boot2container-compatible cmdline argument: b2c.filesystem={name},...

The following attributes can be used to list the containers that should be pulled, along with their pull options. Replace SHORT_NAME with whatever name you want to give to your container. This name can then be used to get the imageID of the container by using its lower-cased short name, like so {{ job.imagestore.public.short_name.image_id }}.

CI_TRON_IMAGESTORE__PUBLIC__IMAGES__SHORT_NAME__NAME

This variable is required

The container image you want to pull, fully-qualified or with docker:// as a prefix.

CI_TRON_IMAGESTORE__PUBLIC__IMAGES__SHORT_NAME__PLATFORM

This variable is required

Typical values: linux/amd64, linux/arm64/v8, linux/arm/v6, or linux/riscv64

The platform of the container that should be pulled. Useful for multi-architecture container manifests.

CI_TRON_IMAGESTORE__PUBLIC__IMAGES__SHORT_NAME__TLS_VERIFY

Accepted values: true, false

A boolean to specify whether to require HTTPS and verify certificates when contacting the container registry during the pull. Defaults to true.

CI_TRON_IMAGESTORE__PUBLIC__IMAGES__SHORT_NAME__PULL_POLICY

Default value: relaxed_always

A string representing the pull policy to use. This may one of the following:

always: Always try pulling the image

relaxed_always: Re-use a previous pull if it is less than 5 minutes old, otherwise try pulling the image again

missing: Only pull the image if it is unknown

Network Block Devices

CI-tron allows you to create fresh Network Block Devices (NBD) for the job.

CI_TRON_NBD__SHORT_NAME__SIZE

This variable is required Format: d+[kMGTP]i?B?

Instantiate an NBD server that exposes a network block device of the wanted size, in bytes. You may also use the convenience exponents k, M, G, T, and P. Follow that by B if you want metric sizes, or iB if you want a power-of-two size. If no B is specified, we default is to use power-of-two sizes.

For example, the following values 1048576, 1024kiB, 1M, and 1MiB are equal, while 1MB < 1MiB.

CI_TRON_NBD__SHORT_NAME__MAX_CONNECTIONS

default: 5

Maximum number of NBD connections that could be initiated by the client to this NBD server.

The information needed for the DUT to connect to the NBD server can be found at {{ job.nbd.short_name }}. It contains the following fields:

name: Name of the NBD resource;
max_connections: The content of CI_TRON_NBD__SHORT_NAME__MAX_CONNECTIONS;
hostname: Hostname of the NBD server exporting this resource;
tcp_port: TCP port of the NBD server exporting this resource;
to_b2c_nbd(name): A helper that converts these parameters into a Boot2container-compatible cmdline argument: b2c.nbd={name},...

Collection Of Lists

CI-tron jobs have this unusual concept of a list split as a collection of lists. This feature is used throughout the job configuration to partially inherit or override sets of items from the default configuration.

The concept is simple: lists are splits in multiple categories, and a category is inherited from the defaults only if it is not mentioned at all. Additionally, there is an extra category called uncategorised which is the default category if the user choses not to name their list.

When the final list is built, the categories are sorted by name then their content is copied in order. Finally, the content of uncategorised is added.

Warning

Some users of collections of lists may decide to prioritize uncategorised elements. In the case of the HTTP and TFTP server, this is done to reduce the surprise factor and make it easier to override defaults paths without having to know about the category they may be inherited from.

LIST_NAME__SHORT_NAME__KEY: Add the element KEY to the uncategorised part of the LIST_NAME list. The keys are then sorted within this list based on their SHORT_NAME before their values being inserted in the list.

LIST_NAME__RAW__SHORT_NAME

Rather than letting the template manage the elements in the uncategorised category of the LIST_NAME list, this enables users to define the list themselves as a list of YAML fragments, ordered by SHORT_NAME.

This can be used a way to prevent inheritance of defaults by simply setting the value to ''.

Warning

Using LIST_NAME__RAW__SHORT_NAME completely overrides any LIST_NAME__SHORT_NAME__KEY that may be set.

LIST_NAME__CATEGORY__CAT_NAME__SHORT_NAME__KEY: Add the element KEY to the CAT_NAME category of the LIST_NAME list. The keys are then sorted within this list based on their SHORT_NAME before their values being inserted in the list.

LIST_NAME__CATEGORY__CAT_NAME__RAW__SHORT_NAME

Rather than letting the template manage the elements in the CAT_NAME category of the LIST_NAME list, this enables users to define the list themselves as a list of YAML fragments, ordered by SHORT_NAME.

This can be used a way to prevent inheritance of defaults by simply setting the value to ''.

Warning

Using LIST_NAME__CATEGORY__CAT_NAME__RAW__SHORT_NAME completely overrides any LIST_NAME__CATEGORY__CAT_NAME__SHORT_NAME__KEY that may be set.

Default boot configuration

By default, CI-tron aims to provide a Boot2Container environment to its users. This means that if all you care about is userspace testing, you do not need to worry about bootloaders, kernels, device trees, or initramfses.

In the event you actually care about your boot process, you may configure your kernel using CI_TRON_KERNEL, kernel command line using CI_TRON_KERNEL_CMDLINE, DTBs using CI_TRON_DTB, or initramfses using CI_TRON_INITRAMFS.

Additionally, the following categories may be inherited from the defaults in CI_TRON_KERNEL_CMDLINE, CI_TRON_INITRAMFS, CI_TRON_HTTP_ARTIFACT, or CI_TRON_TFTP_ARTIFACT:

50-platform: Anything necessary to boot the machine before reaching the bootloader stage, or firmware that is expected by the platform and is not otherwise part of the linux-firmware project, such as the signed and non-redistributable firwmare files found on Qualcomm hardware.

50-bootloader: Anything necessary to load the environment as specified by the kernel, initramfs, and dtb deployment options. Place in this category all the bootloader configuration files.

50-kernel-modules: Anything necessary to make the default kernel boot to a usable environment (ie. kernel modules, firmware, …). Nothing from this category should be included by default by a job that overrides the default kernel.

Artifacts

CI-tron may serve artifacts to the test machine in various ways and various times.

Rather than duplicating this content throughout the documentation, let’s define it here first.

ARTIFACT__PATH

This variable is only applicable to artifacts served via HTTP/TFTP

A regular expression that, when it matches the GET request’s path, will return the wanted artifact, as selected by ARTIFACT__DATA, or ARTIFACT__URL.

If parts of the path are wrapped in parentheses, whatever content is matched may be referenced by URL, ARTIFACT__FORMAT__IDX__ARCHIVE__MATCH, ARTIFACT__FORMAT__IDX__ARCHIVE__KEEP__SHORT_NAME__PATH, and ARTIFACT__FORMAT__IDX__ARCHIVE__KEEP__SHORT_NAME__REWRITE.

ARTIFACT__DATA

Back the content of the artifact with the specified string.

If the data to return is more complex than a small string, you may specify it in parts using ARTIFACT__DATA__IDX.

Warning

This argument takes priority over ARTIFACT__URL if set.

ARTIFACT__DATA__IDX

Back the content of the artifact by aggregating all the DATA variables, sorted by IDX. IDX does not have to be a number, it can be any string.

If the data to return simple-enough, you may specify it using it in parts using ARTIFACT__DATA.

Warning

Do not mix ARTIFACT__DATA__IDX and ARTIFACT__DATA as the result is unspecified.

Warning

This argument takes priority over ARTIFACT__URL if set.

ARTIFACT__URL

An HTTP(S) URL to the artifact you want to use, or a link to a job artifact from the current pipeline, specified using the glartifact://$jobname[$filepath][?default=$defaulturl] format, with the following percent-encoded fields:

$jobname: name of the job from the current CI pipeline you want to use the artifact for, with any / and % character replaced with %2F and %25 respectively. The artifacts are taken from the last successful execution of the job;

$filepath (optional): The path to a single file inside the job’s artifact archive, with any ? and % character replaced with %3F and %25 respectively. If omitted, the whole archive is included;

$defaulturl (optional): A fallback URL in case $jobname is not found in the current CI pipeline, with any & and % character replaced with %26 and %25 respectively. If omitted, an error is returned in that case.

Warning

This argument is ignored if ARTIFACT__DATA or ARTIFACT__DATA__IDX is set.

Note

Parts of the URL may reference parts of the request URL, akin to an nginx url_rewrite.

Example: To answer any GET request matching /linux-firmware/$path with the corresponding file in the linux-firmware GIT repository, you could set the following variables:

CI_TRON_HTTP_ARTIFACT__FW__PATH: /linux-firmware/(.*)

CI_TRON_HTTP_ARTIFACT__FW__URL: https://web.git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/\1

Example: glartifact://

Use as a kernel the file releases/linux-x86_64 located in the linux: [amd64, x86_64] job’s artifacts archive from the current CI pipeline. If the job is missing from the pipeline, default to the kernel located at the specified URL:

CI_TRON_KERNEL__URL: glartifact://linux: [amd64, x86_64]/releases/linux-x86_64?default=https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.9.16/downloads/linux-x86_64

U-Boot formatting

If you would like the artifact returned by the HTTP server to be a U-Boot uimage, you should at least specify ARTIFACT__FORMAT__IDX__UBOOT__TYPE, with IDX the position in the formatting pipeline.

ARTIFACT__FORMAT__IDX__UBOOT__TYPE

This variable is required if wanting to serve a U-Boot image

Wrap the artifact returned by ARTIFACT__PATH as a u-boot image of the specified type. Supported values can be found at https://github.com/u-boot/u-boot/blob/master/include/image.h, without the IH_TYPE_ prefix and in lowercase.

ARTIFACT__FORMAT__IDX__UBOOT__ARCHITECTURE

Supported values: x86, x86_64, arm32, arm64 (default), riscv32, riscv64

Wrap the artifact returned by ARTIFACT__PATH as a u-boot image, indicating that the CPU architecture is the specified value.

ARTIFACT__FORMAT__IDX__UBOOT__COMPRESSION

Supported values: none

Wrap the artifact returned by ARTIFACT__PATH as a u-boot image, indicating that the compression type is the specified value.

ARTIFACT__FORMAT__IDX__UBOOT__OS

Supported values: linux

Wrap the artifact returned by PATH as a u-boot image, indicating that the Operating System is the specified value.

Archive formatting

If the artifact returned by the HTTP server is an archive, you may modify its content on the fly to, either:

Return one file from the archive by using ARTIFACT__FORMAT__IDX__ARCHIVE__MATCH

Create a new archive which contains a subset of the files present in the original archive by using ARTIFACT__FORMAT__IDX__ARCHIVE__KEEP__SHORT_NAME__PATH. The returned files may be re-located to a different location by using ARTIFACT__FORMAT__IDX__ARCHIVE__KEEP__SHORT_NAME__REWRITE.

In either case, the path of the files present in the original archive may reference parts of the request URL. This is useful when trying to find the corresponding file in an archive.

ARTIFACT__FORMAT__IDX__ARCHIVE__EXTENSION

Supported values: none, cpio, iso, tar, zip.

Name of the archive format that should be created as a result of this formatting operation. Leave the value unset, or set to none if you just want to extract a single file from the archive using ARTIFACT__FORMAT__IDX__ARCHIVE__MATCH.

ARTIFACT__FORMAT__IDX__ARCHIVE__COMPRESSION

Supported values: none, gz, bz2.

Compression format that should be applied to the archive that should be created as a result of this formatting operation. Leave the value unset, or set it to none if you do not want to compress the archive.

ARTIFACT__FORMAT__IDX__ARCHIVE__MATCH

Rather than creating a new archive, return the first file that matches the regular expression specified. Files are evaluated in the same order as the archive defines them.

Note

Parts of the specified path may reference parts of the request URL, akin to an nginx url_rewrite.

Example: Answer any TFTP read request matching /dtb/$path or /dtbs/$path with the file located at boot/dtbs/$path in the archive found at the specified URL, then wrap it in a U-Boot image:

CI_TRON_TFTP_ARTIFACT__DTB__PATH: /dtbs?/(.*)

CI_TRON_TFTP_ARTIFACT__DTB__URL: https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.9.15.1/downloads/linux-arm64.dtbs.cpio.xz

CI_TRON_TFTP_ARTIFACT__DTB__FORMAT__0__ARCHIVE__MATCH: boot/dtbs/\1

CI_TRON_TFTP_ARTIFACT__DTB__FORMAT__1__UBOOT__TYPE: flatdt

ARTIFACT__FORMAT__IDX__ARCHIVE__KEEP__IDX__PATH

Allow filtering the content of the source archive to only contain the file paths that match the specified regular expression.

This value may be paired with ARTIFACT__FORMAT__IDX__ARCHIVE__KEEP__IDX__REWRITE in order to change the path of the files kept.

Note

Parts of the specified path may reference parts of the request URL, akin to an nginx url_rewrite, similarly to ARTIFACT__FORMAT__IDX__ARCHIVE__MATCH.

ARTIFACT__FORMAT__IDX__ARCHIVE__KEEP__IDX__REWRITE

Change the location of the files matched by ARTIFACT__FORMAT__IDX__ARCHIVE__KEEP__IDX__PATH using this substitution regular expression.

Note

Parts of the specified path may reference parts of the request URL, akin to an nginx url_rewrite.

Example: Create a CPIO archive that contains all the Raspberry-Pi-firmware-provided modules for the arm64 kernel, but moved to usr/lib/modules/$KERNELVERSION-v8+/ rather than the original firmware-$VERSION/modules/$KERNELVERSION-v8+/ and use it as an initramfs.

CI_TRON_INITRAMFS__RPIFW__URL: https://github.com/raspberrypi/firmware/archive/b154632e320b87ea95c6ce8b59f96dbbe523ecf1.zip

CI_TRON_INITRAMFS__RPIFW__FORMAT__ARCHIVE__EXTENSION: cpio

CI_TRON_INITRAMFS__RPIFW__FORMAT__ARCHIVE__KEEP__0__PATH: firmware-.*/(modules/.*-v8+/.*)

CI_TRON_INITRAMFS__RPIFW__FORMAT__ARCHIVE__KEEP__0__REWRITE: usr/lib/1

HTTP Server

CI-tron allows you to instantiate an HTTP server which can expose read-only artifacts at any path wanted.

The URL to this web server can be referenced by using {{ job.http.url }}.

You may also get the full URL to the test environment artifacts by using {{ job.http.path_to("$target") }} with $target being any of:

kernel

initramfs, initramfs.IDX

dtb, dtb.IDX

CI_TRON_HTTP_ARTIFACT

A collection of lists of artifacts. The uncategorised part of the list is evaluated first in lexicographic order of the keys, followed by every category (in lexicographic order), themselves sorted in lexicographic order.

Example

To answer any GET request matching /linux-firmware/$path with the corresponding file in the linux-firmware GIT repository, you could set the following variables:

CI_TRON_HTTP_ARTIFACT__FW__PATH: /linux-firmware/(.*)

CI_TRON_HTTP_ARTIFACT__FW__URL: https://web.git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/\1

TFTP Server

CI-tron allows you to instantiate a TFTP server which can expose read-only artifacts at any path wanted.

You may also get the full URL to the test environment artifacts by using {{ job.tftp.path_to("$target") }} with $target being any of:

kernel

initramfs, initramfs.IDX

dtb, dtb.IDX

CI_TRON_TFTP_ARTIFACT

A collection of lists of artifacts. The uncategorised part of the list is evaluated first in lexicographic order of the keys, followed by every category (in lexicographic order), themselves sorted in lexicographic order.

Example

Answer any TFTP read request matching /dtb/$path or /dtbs/$path with the file located at boot/dtbs/$path in the archive found at the specified URL, then wrap it in a U-Boot image:

CI_TRON_TFTP_ARTIFACT__DTB__PATH: /dtbs?/(.*)

CI_TRON_TFTP_ARTIFACT__DTB__URL: https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.9.15.1/downloads/linux-arm64.dtbs.cpio.xz

CI_TRON_TFTP_ARTIFACT__DTB__FORMAT__0__ARCHIVE__MATCH: boot/dtbs/\1

CI_TRON_TFTP_ARTIFACT__DTB__FORMAT__1__UBOOT__TYPE: flatdt

Boot configuration

As we saw in the Default boot configuration section, CI-tron provides a default kernel that is suitable for a Boot2container environment. You may however decide to override it using CI_TRON_KERNEL, CI_TRON_INITRAMFS, and CI_TRON_DTB. For Fastboot-devices, you may instead supply the complete boot image using CI_TRON_FASTBOOT__BOOT_IMAGE.

You may also want to control the kernel command line. To do so, you will need to use CI_TRON_KERNEL_CMDLINE

CI_TRON_KERNEL: An artifact to override CI-tron’s default kernel with.

Example

CI_TRON_KERNEL__URL: https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.9.15.1/downloads/linux-arm64

CI_TRON_KERNEL_CMDLINE

A collection of lists of strings that allow configuring the kernel command line in parts. The final command line will be the concatenation of all the fragments, ordered by their short name.

Example

Given the following variables set:

CI_TRON_KERNEL_CMDLINE__02_SECOND: world

CI_TRON_KERNEL_CMDLINE__01_FIRST: hello

The generated command line will be hello world.

CI_TRON_INITRAMFS

A collection of lists of artifacts which will be provided as initramfses to the test machine.

These initramfses may be downloaded via HTTP or TFTP using:

{{ job.tftp.path_to("$target") }}

{{ job.http.path_to("$target") }}

With target being:

initramfs: a concatenation of all the initramfses specified in the list;

initramfs.N: The N-th initramfs specified. Useful when the bootloader supports more than one initramfs and downloads them through TFTP which can have a size limit as low as 32MB.

CI_TRON_DTB

A collection of lists of artifacts which may be used as device tree binaries for the bootloader / kernel to use. If more than one DTB is provided, the behavior will differ depending on the boot chain.

These DTBs may be downloaded via HTTP or TFTP using:

{{ job.tftp.path_to("$target") }}

{{ job.http.path_to("$target") }}

With target being:

dtb: a concatenation of all the DTB specified in the list;

dtb.N: The N-th DTB specified.

Warning

Some boot methods only allow specifying one DTB. In this case, only the first DTB will be used.

Fastboot

For Fastboot-based devices, it is possible to override the Default boot configuration of how the boot image should be generated, or by providing a complete boot image using :attr:CI_TRON_FASTBOOT__BOOT_IMAGE.

Most of the following parameters are named directly after Google’s mkbootimg.py command line parameters which, unfortunately, do not really come with documentation.

CI_TRON_FASTBOOT__HEADER_VERSION: Default value: 0

CI_TRON_FASTBOOT__BASE: Default value: 0x10000000

CI_TRON_FASTBOOT__KERNEL_OFFSET: Default value: 0x00008000

CI_TRON_FASTBOOT__RAMDISK_OFFSET: Default value: 0x01000000

CI_TRON_FASTBOOT__DTB_OFFSET: Default value: 0x01f00000

CI_TRON_FASTBOOT__TAGS_OFFSET: Default value: 0x00000100

CI_TRON_FASTBOOT__BOARD: Default value: ""

CI_TRON_FASTBOOT__PAGESIZE: Default value: 2048

CI_TRON_FASTBOOT__OS_VERSION: Default value: 0

CI_TRON_FASTBOOT__OS_PATCH_LEVEL: Default value: 0

CI_TRON_FASTBOOT__BOOT_IMAGE: The complete boot image that you would like to boot on your fastboot device.

Warning

This parameter renders all the other Fastboot parameters useless, since it completely bypasses image generation.

User variables

If you wish to define variables that are related to job configuration but don’t want to accidentally influence the template generation, you may prefix them with CI_TRON__.

Just like any other variable, they may be referenced using {{ environ.CI_TRON__VARIABLE_NAME }}.

Executorctl parameters

CI-tron does not yet have true native support for GitLab. This job is an attempt to hide the fact that we need to call executorctl to queue the job. Here are variables that can be used to tune this behaviour.

CI_TRON_JOB_TEMPLATE

default: ${CI_TRON_JOB_TEMPLATE_PROJECT_URL}/-/raw/${CI_TRON_JOB_TEMPLATE_COMMIT}/.gitlab-ci/ci-tron-job-dut-v1.yml.j2

Path or URL to the job template that will be used to convert GitLab environment variables to the expected YAML job description CI-tron expects.

CI_TRON_EXECUTORCTL_IMAGE

default: registry.freedesktop.org/gfx-ci/ci-tron/executorctl:…

Container image to use when queuing the job in CI-tron.

CI_TRON_JOB_TEMPLATE_PROJECT_URL

This variable is required if CI_TRON_JOB_TEMPLATE is unset

URL (eg. https://gitlab.freedesktop.org/gfx-ci/ci-tron) of the project that is hosting the job template. Mandatory if you did not override CI_TRON_JOB_TEMPLATE, ignored if you did.

CI_TRON_JOB_TEMPLATE_COMMIT

This variable is required if CI_TRON_JOB_TEMPLATE is unset

Commit-like of the job template that should be used. Mandatory if you did not override CI_TRON_JOB_TEMPLATE, ignored if you did.

CI_TRON_JOB_SCRIPT_PATH: Path where the job script is saved; set this to a path in your job artifacts if you want to keep the job script.

CI_TRON_JOB_SCRIPT_IGNORE_VARS_MATCHING: Regex of names of variables that contain values that should not be written down in the job script.

CI_TRON_EXECUTORCTL_EXTRA_ARGS__SHORT_NAME: Extra parameters that will be passed to executorctl, in the same way a collection of lists would.

.ci-tron-b2c-job-v1

The Boot2container-based job which enables a more seamless integration with GitLab CI.

This job is inheriting from .ci-tron-job-v1, and thus all the attributes it specifies are available. Please however avoid setting any variable called CI_TRON__B2C_ outside of the ones documented under.

Support matrix

Gitlab Feature	Support	Notes
after_script	⚠	Unsupported and will break in the future
artifacts	✅
before_script	⚠	Unsupported and will break in the future
cache	✅
dependencies	✅	Not as efficient as using an HTTP artifact, can be improved
interruptible	✅
image	❌	Use the `CI_TRON__B2C_IMAGE_UNDER_TEST` variable instead
needs	✅	Not as efficient as using an HTTP artifact, can be improved
pages	❌	Unsupported. Use a general-purpose runner for this.
release	❌	Unsupported. Use a general-purpose runner for this.
resource_group	✅
retry	✅
run	❌	Unsupported, but may be supported in the future
services	⚠	Unsupported. May be emulated using `CI_TRON_HTTP_ARTIFACT__B2C_CFG__DATA__03_MY_SERVICE=b2c.run_service=...`
script	❌	Use the `CI_TRON__B2C_EXEC_CMD` variable instead
timeout	⚠	Use the `CI_TRON_TIMEOUT__OVERALL__MINUTES` variable to set the timeout, then ensure `timeout` is set to a value greater than it by at least 5 minutes.
variables	✅

Introductory example

Here is a simplistic example of how to integrate a CI-tron-provided DUT in your in GitLab CI pipelines:

.gitlab-ci.yml:

stages:
  - integration testing

variables:
  # WARNING: it is strongly advised to use a commit hash instead of ``main``
  # WARNING: be extra careful to always keep the variables and the include URL in sync!
  CI_TRON_JOB_TEMPLATE_PROJECT_URL: https://gitlab.freedesktop.org/gfx-ci/ci-tron
  CI_TRON_JOB_TEMPLATE_COMMIT: main

include:
  - remote: 'https://gitlab.freedesktop.org/gfx-ci/ci-tron/-/raw/main/.gitlab-ci/dut.yml'

pre-test:
  stage: integration testing
  variables:
    GIT_STRATEGY: none
  image: registry.freedesktop.org/gfx-ci/ci-tron/executorctl:latest
  script:
    - mkdir -p result
    - echo file1 > result/file1
    - echo file2 > result/file2
  artifacts:
    paths:
      - result

# Concatenate the content of file1 and file2 then save it in file3.
# Check out the generated artifacts to see file3.
.test:
  stage: integration testing
  needs:
    - job: pre-test
      artifacts: true
  extends:
    - .ci-tron-b2c-job-v1
  variables:
    GIT_STRATEGY: none
    CI_TRON__B2C_IMAGE_UNDER_TEST: 'registry.freedesktop.org/gfx-ci/ci-tron/machine-registration:latest'
    CI_TRON__B2C_EXEC_CMD: 'cat result/file1 result/file2 | tee result/file3'
  artifacts:
    paths:
      - result

# Run .test on an amd64 DUT
test-amd64:
  extends:
    - .test
  tags:
    - cpu:arch:x86_64
    - farm:$FARM_NAME

# Run .test on an arm64 DUT
test-arm64:
  extends:
    - .test
  tags:
    - cpu:arch:aarch64
    - farm:$FARM_NAME

Container images

Note

Test machines should access containers through proxy registries that de-duplicate image downloads and thus decrease boot time and increase boot reliability.

You may use the registry.to_local_proxy() function to convert an image name to one pointing towards its designated proxy, according to the rules configured in Registry config.yml’s format.

Example: {{ registry.to_local_proxy('${CI_TRON__B2C_CUSTOM_IMAGE}') }}

CI_TRON__B2C_IMAGE_UNDER_TEST

This variable is required

The image you want to run, fully-qualified or with docker:// as a prefix.

CI_TRON__B2C_EXEC_CMD

This variable is required

The command or list of commands that should be executed, using the shell specified by CI_TRON__B2C_EXEC_SHELL.

This variable may contain multiple lines, but should not contain double ", nor unescaped single quotes.

CI_TRON__B2C_EXEC_SHELL

default: sh

The shell that should be used to execute the CI_TRON__B2C_EXEC_CMD commands.

CI_TRON__B2C_EXEC_SHELL_FLAGS

default: -euc

Shell flags that should be set before executing the CI_TRON__B2C_EXEC_CMD in the CI_TRON__B2C_EXEC_SHELL.

CI_TRON__B2C_MACHINE_REGISTRATION_IMAGE

default: registry.freedesktop.org/gfx-ci/ci-tron/machine-registration:latest

The container image to use to check that the test machine running the job really is the one we expected it to be.

CI_TRON__B2C_MACHINE_REGISTRATION_CMD

default: check

Machine registration command to execute. Could be set to setup --tags TAG1,TAG2,... instead to only check for the presence of the wanted tags and hiding un-wanted hardware.