In this post, we will discuss an interesting technique for testing test coverage, and the associated Rust crate — cov-mark.
The two goals of the post are:
-
Share the knowledge about a specific testing approach.
-
Show a couple of Rust tricks for writing libraries.
This post is an independent sequel to A Trick for Test Maintenance one.
cov-mark crate provides two macros, hit! and check!, which can be used like this:
fn safe_divide(dividend: u32, divisor: u32) -> u32 {
if divisor == 0 {
cov_mark::hit!(divide_by_zero);
return 0;
}
dividend / divisor
}
#[test]
fn test_safe_divide_by_zero() {
cov_mark::check!(divide_by_zero);
assert_eq!(safe_divide(92, 0), 0);
}hit! takes a mark name as a parameter and, at runtime, records that this mark was hit.
check! verifies that the specified mark will be hit when executing the enclosing block.
It creates a guard object which checks the mark in Drop.
If mark is not hit, check! will fail the test.
Motivation
Such manual coverage marks are complementary to the traditional code coverage tools. The main benefit they bring to the table is that it becomes easier to co-evolve tests and code, marks help with maintenance.
Larger scale projects often suffer from the "infinite test suite problem" — there are just so many tests that finding a specific test for a given piece of code is challenging.
This, in turn, can lead to test bifurcation, where the same code has effectively two independent sets of tests, added at different times.
With marks, if you see hit!(some_mark) in the code, you can quickly find the corresponding test by grepping for check!(some_mark).
This is especially helpful if you are contributing to a new (or a well forgotten) code base.
Another common problem is testing for things that shouldn’t happen. Let’s say that you want to write a test that checks that people, who opted-out of emails, don’t get emails. The test itself is straightforward:
-
Create a user profile with opt-out checked.
-
Do something that would trigger sending an email to the user.
-
Verify that no email was sent.
The problem with this is that there might actually be many reasons why email was not sent:
-
The user might have an empty email address.
-
The email system might be not initialized.
-
Sending emails might be disabled globally for whatever reason.
So, the test might pass by accident even if we forget to actually check the specific opt-out!
cov-mark allows you to check that, during the execution of this test, we’ve actually looked at the checkbox, and that that was the reason why the email was not send.
These problems become much worse when code changes over time. Even if tests and code live in a single module initially, one day you might move the code but forget to move the tests. When writing new tests, you can test that the test fails if you comment-out tested code. But, if you later change something else, the test might magically become evergreen. Having explicit and tested coverage marks helps with catching such code drift at the moment it happen.
Finally, if you are writing something mission critical, explicit coverage marks might help with checking that every condition and control flow variation have associated tests.
In rust-analyzer, we have been using coverage marks from early on. We use them primarily to avoid the infinite test suite problem, so they are used sparingly, we don’t try to mark each and every branch. The experience has been positive so far — adding the marks is slightly annoying, but they do help with reading the code afterwards, and it has caught a couple of non-trivial problems during refactors.
Implementation
Here’s the full implementation of cov-mark:
#[macro_export]
macro_rules! hit {
($ident:ident) => {{
#[cfg(test)]
{
extern "C" {
#[no_mangle]
static $ident: $crate::__rt::AtomicUsize;
}
unsafe {
$ident.fetch_add(1, $crate::__rt::Ordering::Relaxed);
}
}
}};
}
#[macro_export]
macro_rules! check {
($ident:ident) => {
#[no_mangle]
static $ident: $crate::__rt::AtomicUsize =
$crate::__rt::AtomicUsize::new(0);
let _guard = $crate::__rt::Guard::new(&$ident);
};
}
#[doc(hidden)]
pub mod __rt {
pub use std::sync::atomic::{AtomicUsize, Ordering};
pub struct Guard {
mark: &'static AtomicUsize,
value_on_entry: usize,
}
impl Guard {
pub fn new(mark: &'static AtomicUsize) -> Guard {
let value_on_entry = mark.load(Ordering::Relaxed);
Guard { mark, value_on_entry }
}
}
impl Drop for Guard {
fn drop(&mut self) {
if std::thread::panicking() {
return;
}
let value_on_exit = self.mark.load(Ordering::Relaxed);
assert!(
value_on_exit > self.value_on_entry,
"mark was not hit"
)
}
}
}A hit!(mark_name) is implemented using an atomic counter:
static mark_name: AtomicUsize = AtomicUsize::new(0);This counter should be accessed by both hit! and check! macro, which might live in different modules.
There’s no pure Rust way to make the two to refer to the same name, without explicitly declaring it in some commonly accessible module.
Fortunately, Rust uses the C linkage model, so we can use the linker to do dependency injection for us.
The check! macro defines a counter using a #[no_mangle] attribute.
The hit! macro then declares it using an extern "C" block.
The linker then weaves the two together.
If we make a typo in the mark name, we’ll get a linker error.
Strictly speaking, this linker trick exists outside of the definition of the Rust language, but sometimes it can be used quite effectively to cut down on dependencies.
It has a couple of serious drawbacks though.
First, mark names must be globally unique.
Second, if we accidentally use an existing symbol name in hit!, we will try to increment something which might not be an integer at all!
Strictly speaking, the unsafe block inside the hit! macro can cause undefined behavior.
However, hitting such a symbol clash by chance without triggering a linker error seems improbable, so I think it is acceptable to not make the caller type unsafe.
If we really want to fix this, we can add a unique prefix to a symbol name, but that would require a procedural macro.
The body of hit! is guarded with a #[cfg(test)], so no marks are present in the final artifact.
The check! macro creates a guard object:
let _guard = $crate::__rt::Guard::new(&$ident);We use prefix to avoid unused variable error.
We however need to use _guard and not just — a plain underscore will drop the guard immediately, and not at the end of the enclosing block.
Inside the Guard, we check that the counter was incremented.
In general, especially when tests are run concurrently, we can’t guarantee that the mark will be hit exactly once.
There’s also a slight chance of false negative here if the mark is hit by an unrelated thread.
Inside Drop, we also check if the current thread is already panicking.
This is a common pattern for asserting drops — we don’t want to mask the original test failure, if any (panicking during a panic aborts the process).
We use Relaxed ordering to access the counter.
As we are accessing only a single memory location, modification order is enough.
Note also that all "runtime" code used by our macros is hidden inside mod rt module.
Specifically, the rt module (rt stands for runtime) also re-export AtomicUsize from the standard library.
I like to use this pattern to make sure that $crate::__rt in macros refers to the names I want, without fear that the caller might shadow the std name.
It also gives a nice overview of the code that is injected into the caller.
That’s all there is for the implementation.
Really, the code in cov-mark crate itself is not that interesting, it’s the general pattern that matters.
Feel free to use the crate as is, or copy its code it and tweak it to your liking!
We offer consulting services and Rust trainings — contact us for a quote and consider subscribing to our Trainings newsletter! We also organize a remote embedded Rust conference, Oxidize Global, that will be taking place in July 2020.
