This is the third post in our "embedded Rust testing" series. This post covers how to run a hardware testsuite based on defmt-test on real hardware with GitHub Actions. We'll be using the testsuite introduced in the first article in the series, so check that out if you haven't yet!

Intro

Employing Continuous Integration pipelines to ensure that all code is tested automatically before being merged can greatly improve software reliability and reduce the maintenance burden. Thanks to free CI offerings for open-source projects, it has become a standard tool that is used throughout the entire Rust ecosystem.

However, there's a catch for embedded projects: If your library depends on specific, non-standard hardware like microcontrollers, it cannot be fully tested using these standard CI services. We have outlined some ways of testing generic drivers in the second article in this series, but some libraries such as microcontroller HALs, require having the targeted hardware available to achieve meaningful test coverage.

Luckily, some CI providers allow the use of self-hosted CI runners that can be hooked up to the hardware to be tested. In this article, we'll be setting up a self-hosted GitHub Actions runner that can be used to run tests on real hardware.

The Hardware

We'll be running the testsuite for the nrf-hal project from the first article on an nRF52840-DK.

In order for the tests to work, the development board has been wired as explained in that article, and the testsuite was confirmed to pass locally.

The CI runner hardware will be a Raspberry Pi 4, but there's no specific reason to choose that. Other hardware should work just as well.

The Problem

Looking at GitHub's documentation for self-hosted runners, we're greeted with a big warning:

Warning: We recommend that you only use self-hosted runners with private repositories. This is because forks of your repository can potentially run dangerous code on your self-hosted runner machine by creating a pull request that executes the code in a workflow.

That's right: if we just add a self-hosted runner to an open-source repository, literally anyone could open a Pull Request and run arbitrary code on our hardware. Depending on how exactly you decide to run the runner software (in an unprivileged container, in a VM, as root on bare metal), this is either very bad or extremely bad, and might let an adversary destroy the hardware hooked up to the machine, or take control of the machine itself.

Luckily, there's a workaround that we can deploy to avoid this issue, which we'll detail in the next section.

Creating a GitHub Actions setup

In summary, we're going to be setting up:

• A private repo, hooked up to our self-hosted runner
• A Private Access Token that allows triggering workflows in the private repo
• An SSH key that lets us push code to the private repo
• A GitHub Actions workflow that pushes to the private repo, triggers a CI pipeline, and then reports back the result
• Integration with bors.tech, which ensures that CI passes before a PR is merged

Together, this will prevent the self-hosted runner from running unapproved code. A reviewer has to approve a Pull Request with bors, which will then push to the staging branch, triggering our hardware CI pipeline.

Note that we won't get into how to actually host the runner software on your hardware, since we didn't come up with a very good solution for that (more on that later).

Repository Setup

We start by creating an additional private repo in which we will be running the hardware tests. In my case, I've created a nrf-hal-ci repository under my personal account, but it can also live in the same organization that hosts the public repository.

The public repository needs an SSH key with access to the private repo, so let's create one now:

ssh-keygen -t rsa -f ssh

Because this will be used by CI, don't enter a passphrase.

In the private CI repo, add the public part of the SSH key as a Deploy Key, and check "Allow write access":

In the public repo, add the private part of the SSH (starting with -----BEGIN OPENSSH PRIVATE KEY-----) key under Settings -> Secrets as a new Repository Secret called SSH_KEY.

Additionally, the public repository will need a secret Personal Access Token for the GitHub API so that it can remotely trigger a workflow in the private repository. In your GitHub settings, go to Developer Settings, Personal access tokens, and click Generate new token. The token must have the workflow scope, which will also check the repo scope. Add this token as a secret of the public repository, called PRIVATE_CI_PERSONAL_ACCESS_TOKEN.

Now, the public repository should have the following secrets:

We can now start writing our Workflow.

Proxy Workflow

First, we need to write a job to trigger the hardware workflow and add it to the regular GitHub Actions pipeline:

  hil:
    runs-on: ubuntu-latest
    if: github.event_name == 'push'

The job can only be run on push events and has to be skipped for pull_request events, otherwise it would fail, because the secrets are not available there.

    steps:
    - uses: actions/checkout@v2
      with:
        fetch-depth: 0  # everything

Because we want to push the repo contents to the private repo, we need to fetch the entire git history in this step, or the push will fail.

    - name: Install SSH key
      uses: shimataro/ssh-key-action@v2
      with:
        key: ${{ secrets.SSH_KEY }}
        known_hosts: not needed
    - name: Push changes to CI repository
      run: |
        git checkout -b hil-run
        git remote add ci git@github.com:<PRIVATE_REPO_OWNER>/<PRIVATE_REPO_NAME>.git
        git push -f -u ci hil-run
        sleep 5

We install the SSH key and use it to git push the code to be tested to the private repository. Note the sleep 5 at the end – this is needed because we're also pushing the GitHub Actions workflow configuration that we want to trigger in the next step, and GitHub sadly does not immediately process the added or changed workflow after its configuration is pushed.

We use the hil-run branch here, because we want to avoid re-triggering workflows that run on pushes to eg. the master branch. While this isn't really a problem (their outcome is ignored), it still wastes concurrent builds that you could use for other projects, and it can also use up paid GitHub Actions time, since this is a private repository. You might need to adjust your existing workflow to only run on pushes to select branches in order for this to work.

Replace <PRIVATE_REPO_OWNER>/<PRIVATE_REPO_NAME> with the right GitHub repository. In this example, this would be jonas-schievink/nrf-hal-ci.

    - name: Trigger and wait for HIL workflow
      uses: jonas-schievink/workflow-proxy@v1
      with:
        workflow: HIL
        ref: hil-run
        repo: <PRIVATE_REPO_OWNER>/<PRIVATE_REPO_NAME>
        token: ${{ secrets.PRIVATE_CI_PERSONAL_ACCESS_TOKEN }}
        repost-logs: true

This step actually triggers the workflow. It uses my workflow-proxy action, which also allows reposting the workflow logs to the invoking workflow, which makes them viewable for contributors. Note that the "HIL" (hardware-in-the-loop) workflow hasn't actually been written yet. We'll do that in the next step.

Just like in the previous step, replace <PRIVATE_REPO_OWNER>/<PRIVATE_REPO_NAME> with the right GitHub repository.

Hardware Testing Workflow

Now, let's write the workflow that will actually run the tests on our hardware. This will only run in the private repository, and utilize the self-hosted runner hardware that we've set up. Put this into a separate workflow file in .github/workflows, because it needs a different trigger event than the other workflows.

name: HIL

on:
  workflow_dispatch:

The workflow name must match the workflow key used in the workflow-proxy step above. Our workflow will be triggered exclusively through the workflow_dispatch event, we don't want to accidentally run it on the original (public) repository, since that would fail, so don't specify other events here.

env:
  CARGO_TERM_COLOR: always

jobs:
  hil:
    runs-on: self-hosted
    steps:
    - uses: actions/checkout@v2

The important part here is runs-on: self-hosted, which makes this job use our dedicated runner hardware. Jobs in this workflow can also run on GitHub-provided runners, which can be useful if you want to compile your code on a more powerful machine.

    - uses: actions-rs/toolchain@v1
      with:
        profile: minimal
        toolchain: stable
        override: true
        target: thumbv7em-none-eabihf
    - name: Install native dependencies
      run: |
        sudo apt-get update
        sudo apt-get install -y libusb-1.0-0-dev pkg-config

Depending on how you deploy the self-hosted runner, the workflow environment might not have all packages installed that you'd find on GitHub-hosted runners, so some extra steps might be required here. In this case, I'm using the Docker image myoung34/github-runner, which is missing some packages required by probe-run.

    - name: Install Rust tooling
      run: cargo install probe-run flip-link

To actually flash and run the defmt-test testsuite, we install probe-run, while flip-link is used to link the executables while allowing reliable stack overflow detection.

    - name: Run tests on hardware
      working-directory: nrf52840-hal-tests
      run: cargo test

Because defmt-test and probe-run natively integrate with Cargo, all we have to do is to run cargo test in the testsuite directory to build and run all tests on the development kit. Should a test fail, probe-run will detect this and make the cargo test invocation fail as well (and thus the workflow).

You can see a successful CI run using this configuration here. The full code changes necessary to set this up for the nrf-hal repository can be viewed in Pull Request #302.

bors

Since this workflow does not run when a Pull Request is opened, but we still want to run it once a PR has been approved and before it is merged into the main repository, we also have to set up a mergebot like bors. Otherwise, the untested code would get merged into the main repo, where the tests could then fail.

Conclusion

While not exactly simple, this recipe should allow setting up reliable hardware-based CI for any project. It can also be adopted to non-Rust projects if needed.

One useful improvement would be having a convenient way to deploy the runner software on custom hardware, without giving it too many permissions. However, even without that, only approved Pull Requests will be granted access to the runner, so this may not be a problem depending on your needs.

Sponsor this work

Knurling-rs is mainly funded through GitHub sponsors. Sponsors get early access to the tools we are building and help us to support and grow the knurling tools and courses. Thank you to all of the people already sponsoring our work through the Knurling project!