Do you need help with coding?

In this blog post we'll explore how to structure a procedural macro, AKA proc-macro, crate to make it easier to test. We'll show different testing approaches:

unit tests, for code coverage;
compile-fail tests, to ensure good error messages; and
end-to-end tests, to validate behavior

We'll also share some techniques for debugging the seemingly inscrutable error messages generated by buggy procedural macros.

⚠️ This post assumes you know the basics of creating a procedural macro crate and are familiar with the libraries syn, a Rust code parser, and quote, a Rust code generator – a refresher section on those concepts is included below.

Procedural macros operate at the syntax level, as source code transformations, before any type checking happens. The output of procedural macros is "Rust code" that cannot be executed in library-level unit tests (#[test] functions). Furthermore, when cross compiling, procedural macros are executed on the "build machine" but their output Rust code may only compile for the "target machine" (e.g. a microcontroller); in that case one cannot execute the output of the procedural macro on their development machine. These characteristics make procedural macros relatively hard to test so one must use an approach that's different from the one used to test regular libraries.

proc-macro refresher

NOTE skip this section if you are familiar with TokenStream, syn and quote

`TokenStream`

Procedural macros operate on token streams (which are close to what a lexer / tokenizer would output). A function-like procedural macro is defined like this:

use proc_macro::TokenStream;

#[proc_macro]
pub fn foo(input: TokenStream) -> TokenStream {
    dbg!(input)
}

And used like this:

foo!(x + y)

x + y is the TokenStream input to the proc-macro. The debug representation (dbg!) of that token stream looks like this (simplified):

[Ident("x"), Punct('+'), Ident("y")]

The output of this proc-macro is the same as its input so foo!(x + y) expands into x + y

`syn`

The syn crate provides parsing functionality. It parses a TokenStream into a Rust Abstract Syntax Tree (AST), which is easier to work with than "raw tokens".

For instance, we can parse the previous token stream x + y into a binary expression, which is represented by the syn type ExprBinary.

use syn::{ExprBinary, parse_macro_input};

#[proc_macro]
pub fn foo(input: TokenStream) -> TokenStream {
    let expr: ExprBinary =
        parse_macro_input!(input as ExprBinary);
    dbg!(expr);
    todo!()
}

The debug representation of expr looks like this (very simplified):

ExprBinary {
    left: Expr::Path(ExprPath([Ident("x")])),
    op: BinOp::Add,
    right: Expr::Path(ExprPath([Ident("y")]))
}

Which is less like an array ("stream") and more like a tree.

When the input changes to x + (y - z), the parsed expression becomes a nested tree:

ExprBinary {
    left: Expr::Path(ExprPath([Ident("x")])),
    op: BinOp::Add,
    right: Expr::Paren(ExprParen(ExprBinary {
        left: Expr::Path(ExprPath([Ident("y")])),
        op: BinOp::Sub,
        right: Expr::Path(ExprPath([Ident("z")])),
    }))
}

which can be graphically represented as:

Abstract Syntax Tree

`quote`

The quote! macro in the quote crate lets you ergonomically generate token streams. You write Rust syntax inside the quote! macro and you get back a TokenStream.

use quote::quote;

#[proc_macro]
pub fn foo(input: TokenStream) -> TokenStream {
    quote!(xyz + 1)
}

The above proc-macro expands into xyz + 1 regardless of its input.

In practice, you'll want to return a token stream based on the input so you'd use quote's quasi-quoting feature.

use syn::{ExprBinary ,parse_macro_input};
use quote::quote;

#[proc_macro]
pub fn foo(input: TokenStream) -> TokenStream {
    let expr: ExprBinary =
        parse_macro_input!(input as ExprBinary);
    let left = expr.left;
    let right = expr.right;
    quote!(#right - #left)
    //     ^^^^^^   ^^^^^ interpolate these inputs
}

This updated version of the proc-macro swaps the input binary expression and turns into subtraction. foo!(x + y) produces y - x, whereas foo!(y * x) produces x - y.

That's the basic gist of procedural macros: at the user-facing level they take a token stream and return a token stream but, internally, they usually contain parsing (syn) and quasi-quoting (quote).

The example

To illustrate the testing strategy we'll partially re-implement the functionality of the contracts crate. The crate provides design by contract functionality in the form of item-level attributes.

The goal of the design by contract paradigm is to enforce invariants with runtime assertions. These "contracts" (assertions) in addition to unit tests increase the confidence in the software. For example, if you were implementing std::Vec<T>, one invariant that needs to be upheld in several of its operations is that the length of the collection never exceeds its capacity.

struct Vec<T> {
    pointer: NonNull<T>, // heap allocation
    length: usize, // number of items in the collection
    capacity: usize, // size, in items, of the heap allocation
}

impl<T> Vec<T> {
    pub fn push(&mut self, item: T) {
        assert!(self.length <= self.capacity);

        // do the actual push, which may reallocate the
        // heap allocation and change `capacity`

        assert!(self.length <= self.capacity);
    }
}

With the contracts crate, one can write those assertions more concisely as:

impl<T> Vec<T> {
    // the invariant is checked on function entry and exit
    #[invariant(self.length <= self.capacity)]
    pub fn push(&mut self, item: T) {
        // .. do the actual push ..
    }
}

The other kind of assertion that appears often in practice are preconditions, specially in constructors. Newtypes like NonNull upheld some invariant like "this pointer is not null"; when constructing these newtypes that property needs to be validated. For example, the unsafe new_unchecked constructor of NonNull could be written as:

pub struct NonNull<T> {
    pointer: *mut T,
}

impl<T> NonNull<T> {
    pub unsafe fn new_unchecked(pointer: *mut T) -> Self {
        debug_assert!(!pointer.is_null());
        Self { pointer }
    }
}

The input validation (debug assertion) can be turned into a contract attribute to make it stand out from the code and easier to search for.

impl<T> NonNull<T> {
    #[debug_requires(!pointer.is_null())]
    pub unsafe fn new_unchecked(pointer: *mut T) -> Self {
        Self { pointer }
    }
}

In this post, we'll only concern ourselves with the "requires" feature of the contracts crate, which we'll rename to "precondition" to differentiate it from the original.

The syntax we'll implement is shown below:

#[contracts]
#[precondition(!list.is_empty())]
#[precondition(check_some_other_property(list))]
fn process(list: &[Item]) -> Output {
    // function body
}

Apart from the "requires" to "precondition" rename, we'll have a single proc-macro named #[contracts] – the original contracts provides several proc-macros: #[requires], #[ensures], etc.

The semantics of each #[precondition] is to assert that its argument, e.g. !list.is_empty(), evaluates to true on function entry, before the body of the function is executed.

We'll focus on tests in this post and omit the implementation details but you can find a fully working implementation in this repository.

The pipeline

We'll structure the procedural macro as a pipeline where each stage transforms the output from the preceding stage. On one end of the pipeline we have the user input and at the other end we have the "expanded Rust code". We'll strive to test each stage of the pipeline in isolation.

procedural macro pipeline

The ? character in the diagram means "fallible" – these passes can early return with an error. The latter passes are "infallible".

In code, the above pipeline looks like this:

use proc_macro::TokenStream;

#[proc_macro_attribute]
pub fn contracts(
    args: TokenStream,
    item: TokenStream,
) -> TokenStream {
    let ast = parse(args, item);
    let model = analyze(ast);
    let ir = lower(model);
    let rust = codegen(ir);
    rust
}

Let's look deeper into each stage.

Parse

In this stage, we parse the user input, which are Rust tokens (TokenStream), into an Abstract Syntax Tree (AST). We'll use the syn crate to do the parsing. In our example, the AST is a single function item (e.g. fn f() { .. }).

pub type Ast = syn::ItemFn;

pub fn parse(args: TokenStream, item: TokenStream) -> Ast {
    // ..
}

Small aside: the user input is split in two arguments – this is mandated by the top-level #[proc_macro_attribute] function:

args are the Rust tokens passed as an argument to the attribute, e.g. the x in #[contracts(x)]
item are the Rust tokens on top of which the attribute "sits on"

If we were to represent that in Rust syntax it would look like this:

#[contracts(<args>)]
<item>

Back to the parse function: although not reflected in the signature, this function is fallible. I decided to use the proc-macro-error crate, which provides a rustc-like abort! API and produces rustc-like error messages, instead of the usual Result based approach.

In this stage, we can test that parse can indeed accept the expected syntax without failing (abort!-ing). We'll write that test as a regular #[test] function defined somewhere in the proc-macro library (e.g. src/lib.rs).

To synthesize input for this function we use the quote! macro, which accepts Rust code and returns Rust tokens (TokenStream).

#[test]
fn valid_syntax() {
    parse(
        /* args: */ quote!(),
        /* item: */ quote!(
            #[inline]
            #[precondition(x % 2 == 0)]
            fn even_to_odd(x: u32) -> u32 {
                x + 1
            }
        ),
    );
}

That test covers the "happy path" of this stage.

Diagnostics (error paths)

The parse function has a few error paths:

using #[contracts] on an item that's not a function
passing an argument to the #[contract] attribute

We'll cover these paths using compile-fail tests. Those kind of tests are not part of Rust's built-in test infrastructure but one can use the trybuild crate.

The usual setup is to create a test file (e.g. ui.rs) in the tests directory, which where integration tests live.

#[test]
fn ui() {
    let t = trybuild::TestCases::new();
    t.compile_fail("tests/ui/*.rs");
}

This ui tests will treat all files under the tests/ui directory as compile-fail tests. For each compile-fail test, one must provide a Rust source file and a .stderr file that contains the expected error message.

To test error path (1) I wrote this item-is-not-a-function.rs file:

use contracts::contracts;

#[contracts]
struct S;

fn main() {}

and this accompanying item-is-not-a-function.stderr file:

error: item is not a function

  = help: `#[contracts]` can only be used on functions

 --> $DIR/item-is-not-a-function.rs:4:1
  |
4 | struct S;
  | ^^^^^^^^^

The ui test will enforce that the implementation of the proc-macro produces the error message in .stderr when the associated .rs file is compiled.

To cover error path (2) I used this other Rust program (has-expr-argument.rs):

use contracts::contracts;

#[contracts(true)]
fn f() {}

fn main() {}

and this expected compiler error message:

error: this attribute takes no arguments

  = help: use `#[contracts]`

 --> $DIR/has-expr-argument.rs:3:13
  |
3 | #[contracts(true)]
  |             ^^^^

Syntax errors?

We don't cover lexer errors – unmatched delimiters, missing semicolons, etc. – with compile-fail tests in a proc-macro crate. Lexer errors are caught by rustc before proc-macro code is executed so proc-macros don't have to deal with that kind of errors.

Analyze

In the next stage, we'll map the AST into the proc-macro "domain model", that is into types that more closely reflect the semantics of the proc-macro. (I like to think of proc-macros as a way to extend Rust code with a Domain-Specific Language (DSL) so the name "domain model" seems fitting to me.) For lack of a better term, I'll call this the 'analyze' stage.

use syn::{Expr, ItemFn};

struct Model {
    preconditions: Vec<Expr>,
    item: ItemFn,
}

fn analyze(ast: Ast) -> Model {
    // ..
}

Our domain Model is going to be a function item (syn::ItemFn) plus some precondition expressions (syn::Expr). The AST stage function item contains #[precondition] attributes, but after this stage, those attributes are stripped from the function item and processed into precondition expressions.

At this stage, we can test that #[precondition] attributes are extracted correctly.

#[test]
fn can_extract_precondition() {
    let model = analyze(/* ast: */ parse_quote!(
        #[precondition(x)]
        fn f(x: bool) {}
    ));

    let expected: &[Expr] = &[parse_quote!(x)];
    assert_eq!(expected, model.preconditions);

    assert!(model.item.attrs.is_empty());
}

That's reflected in two checks:

model.preconditions contains the expression arguments of the #[precondition] attributes that were in the AST
model.item must not contain #[precondition] attributes

To synthesize input for this test we use the parse_quote! macro, which is similar to quote! but returns a syn type. In this case, we use it to produce an instance of our Ast type, which is an alias for syn::ItemFn (a function item). We also use the parse_quote! macro to generate the expected values of the test.

Another thing worthwhile to check here is that attributes unrelated to our proc-macro are not removed or reordered. Reordering attributes can cause other proc-macros to break when proc-macros are composed. The test for that is shown below:

#[test]
fn non_dsl_attributes_are_preserved() {
    let model = analyze(/* ast: */ parse_quote!(
        #[a]
        #[precondition(x)]
        #[b]
        fn f(x: bool) {}
    ));

    let expected: &[Attribute] = &[
        parse_quote!(#[a]),
        parse_quote!(#[b]),
    ];
    assert_eq!(expected, model.item.attrs);
}

Diagnostics

Like the previous stage, the 'analyze' stage is also fallible. We'll test error paths using compile-fail tests like we did previously.

First error path: the argument of #[precondition] is not an expression.

Test case: precondition-is-not-an-expression.rs

use contracts::contracts;

#[contracts]
#[precondition(struct)]
fn f() {}

fn main() {}

Expected compiler error (.stderr file):

error: expected an expression as argument

  = help: example syntax: `#[precondition(argument % 2 == 0)]`

 --> $DIR/precondition-is-not-an-expression.rs:4:15
  |
4 | #[precondition(struct)]
  |               ^^^^^^^^

Second error path: no #[precondition] was specified.

(Arguably, this should be a warning and not an error but one cannot emit warnings from proc-macros on the stable channel as of Rust 1.55 so we'll make do with an error)

Test case: zero-contracts.rs

use contracts::contracts;

#[contracts]
fn f() {}

fn main() {}

Expected compiler error:

error: no contracts were specified

  = help: add a `#[precondition]`

 --> $DIR/zero-contracts.rs:3:1
  |
3 | #[contracts]
  | ^^^^^^^^^^^^
  |

Lower

In this stage, we transform the domain model into a codegen "intermediate representation" (IR).

Although we could directly transform the model into Rust code (Rust tokens), it's not easy to do assertions on Rust tokens so it would not be easy to test the logic in a single 'model to Rust' stage.

By introducing an IR that's amenable to testing, we can shift most of the logic to the first 'lower' stage and do unit testing there on the produced IR. The later 'codegen' stage should be kept as "dumb" as possible so we can make do without unit tests there.

We'll lower model.preconditions into Assertions, an IR representation of an assert! expression.

struct Ir {
    assertions: Vec<Assertion>,
    item: ItemFn,
}

struct Assertion {
    expr: Expr,
    message: String,
}

fn lower(model: Model) -> Ir {
    // ..
}

At this stage, we'll test that preconditions are turned into assertions and that the assertion messages have the format we want.

#[test]
fn produces_assertion_for_precondition() {
    let mut model = Model::stub();
    model.preconditions.push(parse_quote!(x));

    let ir = lower(model);

    assert_eq!(1, ir.assertions.len());

    let assertion = &ir.assertions[0];
    let expected: Expr = parse_quote!(x);
    assert_eq!(expected, assertion.expr);
    assert_eq!(
        "violation of precondition `x`",
        assertion.message,
    );
}

impl Model {
    fn stub() -> Self {
        Self {
            preconditions: vec![],
            item: parse_quote!(
                fn f() {}
            ),
        }
    }
}

Codegen

In this last stage, we'll turn the IR into Rust tokens.

type Rust = TokenStream;

fn codegen(ir: Ir) -> Rust {
    // ..
}

Rust tokens are hard to work with so the only test here is a basic check that the output Rust tokens can be parsed as a function item.

#[test]
fn output_is_function_item() {
    let ir = Ir {
        assertions: vec![Assertion {
            expr: parse_quote!(x),
            message: "message".to_string(),
        }],
        item: parse_quote!(
            fn f() {}
        ),
    };
    let rust = codegen(ir);

    assert!(syn::parse2::<ItemFn>(rust).is_ok());
}

This may look like a trivial test but later we'll see that it comes in handy sometimes.

End-to-end testing

To validate the behavior of the proc-macro we'll use "end-to-end" tests that exercise the entire pipeline. We'll place these tests in the tests directory.

This is the tests/precondition.rs test file.

use contracts::contracts;

#[contracts]
#[precondition(input)]
fn f(input: bool) -> i32 {
    0
}

#[test]
fn pass() {
    f(true);
}

#[test]
#[should_panic = "violation of precondition `input`"]
fn fail() {
    f(false);
}

#[precondition(input)] must check that the input argument is true when the function f is called. If input is false then the program panics before the body of the function is executed.

The pass test exercises the runtime happy path of #[precondition] where the assertion passes.

The fail test exercises the runtime error path of #[precondition] where the assertion fails. In this test we use the #[should_panic] attribute to indicate that the code inside the test function must panic in order for the test to pass. We also include the expected panic message in the #[should_panic] attribute.

In this example application, we can run the expanded code on the build (developer's) machine; that makes end-to-end testing easier. Some procedural macros may produce code that's only meant to be run on a target machine whose architecture is different that the developer's machine, like a microcontroller – that is the case for defmt, a logging framework, and RTIC, a bare-metal multitasking framework. In those cases you'll need to get creative with your end-to-end tests and use emulators (e.g. QEMU), hardware-in-the-loop (HIL) testing and/or snapshot testing.

"Debugging" proc-macros

That last 'codegen' stage can be a source of hard to debug issues, which is the other reason for keeping it as simple as possible.

Panics

For instance, if you forget a semicolon somewhere you may get this kind of error when you run the integration tests:

$ export RUST_BACKTRACE=1

$ cargo t --test precondition
error: custom attribute panicked
 --> tests/precondition.rs:3:1
  |
3 | #[contracts]
  | ^^^^^^^^^^^^
  |
  = help: message: unexpected end of input, expected semicolon

The proc-macro panicked but setting RUST_BACKTRACE won't show you the backtrace. Your debugging tool here is that seemingly trivial unit test we wrote for the 'codegen' stage.

$ export RUST_BACKTRACE=1
$ cargo t --lib -- codegen::tests
running 1 test
test codegen::tests::output_is_function_item ... FAILED

failures:

---- codegen::tests::output_is_function_item stdout ----
thread 'codegen::tests::output_is_function_item' panicked (..)
stack backtrace:
(..)
   3: contracts::codegen::(..)
             at ./src/codegen.rs:31:26
(..)

That's the backtrace frame (function) were the bug was: I forgot a semicolon in this line:

let stmt: Stmt = parse_quote!(assert!(#expr, #message) );
//                                                    ^ here

Compilation errors

In other cases, the proc-macro will run fine (not panic) but produce Rust code that doesn't compile due to a codegen bug. The error message will often have subpar span information so it'll be hard to tell where the problem comes from.

$ cargo t --test precondition
error[E0308]: mismatched types
 --> tests/precondition.rs:5:22
  |
3 | #[contracts]
  |            - help: consider removing this semicolon
4 | #[precondition(input)]
5 | fn f(input: bool) -> i32 {
  |    -                 ^^^ expected `i32`, found `()`
  |    |
  |    implicitly returns `()`

My suggestion here is to use Rust-Analyzer's 'expand macro recursively' feature, which recently gained support for proc-macro attributes. Note that, at the time of writing, proc-macro attribute support is behind an experimental flag (rust-analyzer.experimental.procAttrMacros) that needs to be explicitly enabled.

Here's the tests/precondition.rs file again:

#[contracts] // <- use 'expand macro recurvisely' on this line
#[precondition(input)]
fn f(input: bool) -> i32 {
    0
}
// ..

Rust-Analyzer will display this macro expansion:

// Recursive expansion of contracts! macro
// ========================================

fn f(input:bool) -> i32 {
  assert!(input,"violation of precondition `input`");
  {
    0
  };
}

Which makes it easier to spot the semicolon that needs to be removed.

Alternatively, you can use the cargo-expand tool, which uses the unstable -Z unpretty compiler flag and thus requires nightly. The tool expands the whole file, not just one proc-macro, and, in this example, it will also expand the assert! macro cluttering the output.

$ cargo expand --test precondition
#![feature(prelude_import)]
#[prelude_import]
use std::prelude::rust_2018::*;
#[macro_use]
extern crate std;
fn f(input: bool) -> i32 {
    if !input {
        {
            ::std::rt::begin_panic(
                "violation of precondition `input`",
            )
        }
    };
    {
        0
    };
}
(.. omitted other 42 lines of code ..)

Try it out!

Try out the testing strategy and the debugging techniques presented in this blog post by adding these features to the example repository:

add a #[postcondition(expr)] contract (to the existing proc-macro!) that asserts expr on function exit
add an #[invariant(expr)] contract that asserts expr on both function entry and exit
extend the annotated function's API documentation with a description of all the contracts the function has
extend the contract syntax with if guards, e.g. #[precondition(expr if cond)] only asserts expr if cond evaluates true
add "debug" variants of the contracts, e.g. #[debug_precondition]
- these contracts are only be checked when debug-assertions are enabled, i.e. their assertions are stripped when the release compilation profile is used

Conclusion

That's it! We hope this blog post have given you some ideas about how to test and debug your next proc-macro crate.

Considering Rust for your next project? Need to get your team up to speed up with Rust? Wondering how to best get your Rust project started or want make sure you are using the best practices and the right architecture / patterns? Already writing Rust and could use a code review? Would like to offload some development work or sponsor our work on open source projects like Rust-Analyzer or Knurling?

Ferrous systems is here for all your Rust needs: from training to consulting, spanning code reviews, development packages, async "Rust experts" support and mob sessions. Shoot us an e-mail and we'll figure out how we can help you.

Structuring, testing and debugging procedural macro crates