What is defmt?

The defmt library is described in the defmt book as:

A highly efficient logging framework that targets resource-constrained devices, like microcontrollers.

It's important to note that, despite the most common use-case, defmt doesn't target microcontrollers exclusively, and nor does it target Arm systems in particular. It is useful wherever you need a high efficiency logging framework.

As with so many things, what we call defmt is actually many different things wearing a trenchcoat. Under the general umbrella of defmt we have:

1. The Logging API for libraries to emit log messages:

   info!("Received buffer {=[u8]} at {=u32:us}", buffer, rx_timestamp);
   

   These calls are similar to, but not fully compatible with, the std::fmt machinery used in println! and friends. In particular, defmt supports =u32 and similar to specify the type of the value being printed, but std::fmt does not. We added this to save space because then the type of the value doesn't need to be serialised into the logging frame.

2. The Logging Implementation, contained in the defmt crate. This includes macros which emit magically named symbols that intern the format strings and other metadata using The String Interner (5), then call into the registered global defmt transport using The Logging ABI (3) to send messages using The Encoding Implementation (4).

3. The Logging ABI: a stable mechanism for connecting a defmt transport to logging calls made in libraries (leaving the library independent of the whichever logging implementation the application author has chosen). This is done using the Logger trait and the global_logger! macro.

4. The Encoding Implementation: A mechanism for emitting log messages as both interned strings (at compile time) and log frames (at run time). As this mechanism can change over time, it is identified with a Defmt Version value. A Defmt Version is a string, but currently each prior version has had a simple numeric value; the current Defmt Version is "4". This version number unambiguously identifies both the The String Interner (5) and the The Wire Format (6), which then allows the defmt logs to be reconstituted.

   The Defmt Version produced by the defmt crate is indicated in the ELF symbol table by a symbol with a name like _defmt_version_ = X. As of defmt-0.3.10, that symbol is called _defmt_version_ = 4 to indicate Defmt Version 4. Note that it does not indicate the version number of the defmt crate. It would, in theory, be possible for one version of the defmt crate to offer multiple Defmt Versions, controlled with feature flags, although it is highly likely that only one could be selected at a time to avoid conflicts.

   Adding a new Defmt Version is not considered a breaking change, because of this mechanism to self-describe which version it is using, although it may require users to update any host tools (like probe-rs) to use a new enough version of defmt-decoder. Dropping support within defmt-decoder for an older wire format is considered a breaking change to that crate.

5. The String Interner: a mechanism for embedding defmt interned strings and metadata in such a way that they do not occupy space in the target's flash, but can be read by a host-side tool like defmt-decoder. In Defmt Version 4, the data is stored as JSON formatted symbol names within an ELF file, such that each is allocated a unique 'memory address' (which acts as the interned string's ID).

   These symbols are placed within a NOLOAD section and so do not occupy any storage space on the target. Here is an example of a symbol from Defmt Version 4, as viewed with objdump:

   00000002 g O .defmt 00000001 {"package":"dk","tag":"defmt_debug","data":"Initializing the board","disambiguator":"15307126232750674618","crate_name":"dk"}
   

   We can see it lies at memory address 0x0000_0002, thus giving it an ID of 2. The JSON encoded object tells of which package generated the message, the log level, and the string itself.

6. The Wire Format: A mechanism for encoding defmt log messages into defmt log frames (the wire format). Any given Defmt Version may have multiple options for how this encoding is performed, which will be indicated using symbols within the .demft section, like _defmt_encoding_ = rzcobs.

7. The Format Trait: a trait, defmt::Format, for letting user-defined data types encode themselves into the The Wire Format. This then allows them to be passed as arguments to the defmt logging macros in The Logging API (1). There is an associated derive-macro which can automatically implement this trait on any struct or enum comprising only values that implement that trait (in a similar vein to derive(Debug) for core::fmt::Debug).

8. The Reference Logging Transport called defmt-rtt, which sends encoded defmt frames over Segger's Real Time Transport (RTT). RTT is an in-memory ringbuffer which is written to by the target's CPU and read from by a debug probe under the control of a host machine. RTT is usually used to send strings (like a serial port) but we use it to send encoded defmt frames.

9. The Reference Logging Parser called defmt-parser, which can read defmt log frames (e.g. as received over RTT via a debug probe connected to an MCU that is using defmt-rtt as its logging transport) and turn them into Rust structures.

10. The Reference Decoder Library called defmt-decoder, which can read structures that represent defmt log frames (e.g. as output by defmt-parser) and turn them into formatted Unicode strings.

11. The Reference Parser Binary called defmt-print, which uses defmt-decoder to decode standard input, and prints formatted log messages to standard output.

There is a delicate balance between these components. As one example, it is important that a tool that uses defmt-decoder (e.g. the popular probe-rs programming tool) uses a version that knows about the various different Defmt Versions that it might encounter. In addition, is important that a defmt transport like defmt-rtt is compatible with the defmt logging crate being used - even if you are using a driver crate that hasn't been updated for some time and was written to use an older version of the defmt library. Because the logging library and the transport communicate through magically named macro-generated extern "C" functions, we must limit users to a single instance of the defmt crate in the dependency tree. This means major version bumps on the defmt crate have ecosystem-splitting effects that we really want to avoid.

What does a 1.0 mean?

A 1.0 release is a commitment to stability, as demonstrated to great effect by the Rust Project itself.

The current Rust language is the result of a lot of iteration and experimentation. The process has worked out well for us: Rust today is both simpler and more powerful than we originally thought would be possible. But all that experimentation also made it difficult to maintain projects written in Rust, since the language and standard library were constantly changing.

The 1.0 release marks the end of that churn. This release is the official beginning of our commitment to stability, and as such it offers a firm foundation for building applications and libraries. From this point forward, breaking changes are largely out of scope (some minor caveats apply, such as compiler bugs).

When we started, defmt was a novel idea and no-one was certain it would work out in practice. The response has been remarkable:

• There are currently 491 crates on crates.io which depend on defmt at the time of writing (mostly libraries using defmt to emit logs, but also some defmt transports)
• There are currently over 5,500 repositories on Github which depend on defmt (a mixture of libraries and binaries)

However, to get there, defmt has had to change and adapt. The defmt crate has had around 19 releases, getting as far as v0.3.10. This version implements Defmt Version 4, whilst our latest defmt-decoder was v0.4.0, which can interpret Defmt Version 4 and Defmt Version 3.

Whilst we have been effectively stable since the v0.3 release, the v1.0 release of defmt will mark an official end to any churn, and the beginning of our commitment to stability for the defmt ecosystem.

The defmt 1.0 release

1. Libraries will be able to specify defmt = "1" in their Cargo.toml file. Subsequent releases of defmt will follow Cargo's guidelines on Semantic Versioning. There is currently no intention to ever produce a defmt 2.0 release.

2. There will be a final 0.3-series release of defmt, which imports defmt = "1" and re-exports a defmt-0.3 compatible API (the so-called 'semver trick').

3. Any library currently using defmt = "^0.3" in their Cargo.toml file can replace that line with defmt = "1", and everything will continue to work, unless you are locked to a version of 0.3 prior to the final 1.0-compatible version.

4. A binary can link against both a library using defmt = "^0.3" and a library using defmt = "1" (this was not possible for the defmt-0.2 to defmt-0.3 upgrade).

   Note however that any dependency locking to a specific earlier of defmt (e.g. using defmt = "=0.3.10") will conflict with the use of defmt-1.0.

5. The defmt-decoder will be updated to 1.0, which will be a breaking change for any binary or library using the existing version. These breaking changes will be documented, with common fixes identified. A PR will be produced for probe-rs to update to defmt-decoder 1.0. Subsequent releases of defmt-decoder will will follow Cargo's guidelines on Semantic Versioning.

6. The ABI between the defmt library and the transport implementations (like defmt-rtt) will be stabilised as-is and documented in the defmt book.

7. The defmt-rtt transport will be updated to 1.0, but any application using defmt-rtt = "^0.3" can be updated to use defmt-rtt = "^1.0" with no further changes required to the program.

8. The list of Defmt Versions supported by defmt-decoder (and hence defmt-print) will be documented, including the period of time for which the Knurling Project is guaranteeing to support each of those formats with support and updates in defmt-decoder.

9. A mechanism will be documented whereby users can add their own custom Defmt Versions. A set of Defmt Versions will be reserved for private use, along the lines of the Unicode Private Use Area. This will allow users to fork the defmt crate and defmt-decoder crates for private use, confident that the production versions will never start producing output with a conflicting Defmt Version. Once fully developed, there is the option to upstream such changes, but where significant differences are to be found, we might want to optionally enable the new encoding to make upgrades easier for existing users.

   For example, a user might wish to change the wire format so that every defmt log frame includes a single byte to indicate the logging level, rather than that information only being available inside the interned metadata. They could temporarily fork the defmt and defmt-decoder crates and prototype these changes using a PUA Defmt Version (e.g. Version 1000). At a later time, this could be added to the defmt crate as part of, say Defmt Version 7, but as an optional feature enabled in much the same way that RZCOBS encoding is enabled currently (with a symbol called _defmt_encoding = rzcobs).

10. The defmt-parser library will remain an unstable, internal-only tool. The defmt-decoder will be the stable mechanism to turn encoded binary data into Unicode text with optional ANSI escape-sequence-based formatting.

Next Steps

We have already published the release candidate crates defmt-1.0-rc.1 and defmt-0.3.100-rc.1 so you can review all the documentation online. We also have a git branch that you can use for testing - we highly recommend you do this, using a cargo patch:

[patch.crates-io]
defmt = { git = "https://github.com/knurling-rs/defmt", branch = "defmt-1.0", version = "0.3.100" }

Any crate in your tree using defmt = "0.3" should automatically be upgraded to 0.3.100, which means you are in fact using the latest 1.0 release.

The 1.0 crates proper should be published on or around the 27 January, but, anyone still using defmt = "^0.3" won't get upgraded to 1.0 until we publish the semver-trick crate (defmt 0.3.100). That will happen on 05 February, and everyone will (in theory) seamlessly move over to this new world of defmt 1.0, automatically.

Please give the release candidate crates a trial, and if you observe any issues, please let us know in the issue tracker, and if you have questions about the defmt-1.0 release, you can join in our Github Discussion on the topic.

Onwards, to the future of defmt!