rust-analyzer is an experimental compiler frontend for the Rust programming language. The ultimate goal for this project is to provide the perfect IDE experience for Rust, with all IDE features working flawlessly while editing code.

I began working on rust-analyzer in August of 2018. Later, in February of 2019, the rls-2.0 working group was formed after the Rust all hands in Berlin, and rust-analyzer became a part of this effort. The purpose of the working group is to experiment with possible approaches to building the IDE-capable compiler. The acquired knowledge then could be used to decide the overall direction for rustc. Note that the rls-2.0 is a misnomer: we don't have any immediate plans for replacing the existing Rust Language Server.

This post talks about what happened to rust-analyzer in between the all-hands and today, discusses future plans, and also announces the rust-analyzer Open Collective. Note that this post is written by me, @matklad, and expresses my personal views.

Achievements

Rust-analyzer is a team effort and we have done a lot of cool stuff since February. Here, I'd love to highlight some of the bits which I find especially interesting:

Florian Diebold (@flodiebold) continued the work on the type system and spearheaded integration with chalk – the new Rust trait solver. This work had a direct impact on users of rust-analyzer: completions now work for Deref! chalk integration also proves that trait solving, which is an inherently hard problem for IDEs, can work extremely quickly. Finally, rust-analyzer throws a ton of real-life Rust code at chalk, which helped to uncover some performance issues within Chalk itself.

Edwin Cheng (@edwin0cheng) did an amazing job implementing macro_rules. This lead to a huge usability improvement, because large parts of the standard library and crates.io ecosystem make use of macros. Macros by example are an odd-one-out feature of the Rust language and use unusual name resolution rules. Interaction between the module system and macros is the hardest bit of the language from the IDE point of view. One of the big breakthroughs since the all-hands was making macro-aware name resolution incremental (PR #1055), using an idea by @nikomatsakis.

Igor Matuszewski (@Xanewok) implemented an integration with cargo watch, which was later improved by Edwin Cheng and Ryan Cumming (@etaoins). This gives inline errors in the editor with relatively little effort, although performance is less than ideal for larger projects. This feature is exciting because it shows an interesting approach for integrating existing rls and rust-analyzer. Specifically, currently rls links to rustc, and that produces some maintenance work to keep rls, Clippy (which is also linked in) and rustc in agreement on the internal compiler's API. What we can do instead is run rustc out of the rls process (by shelling out to Cargo). This should not only make rls easier to maintain, but also will allow sharing save-analysis data processing logic between rust-analyzer and rls (and, indeed, even merge the two projects).

We have also focused heavily on performance, to make sure that new powerful features don't make rust-analyzer slower. I am especially proud of fixing issue #1331, which revealed three independent high-level shortcomings of rust-analyzer. This issue is about custom processing of the Enter key in the editor. In the long run, we want to auto-indent the cursor, but for now we do simpler things like automatic continuation of doc comments. This feature is interesting because it interacts with user's typing and should be really fast. A perfect IDE should be able to do both:

• long-running background analysis like error detection
• without blocking high-priority syntax-related requests like "how to indent the cursor after Enter"

This requirement is a good example of why you need a special architecture for IDEs, and can't "just" use a traditional compiler. In rust-analyzer, we have a rather fancy scheduling and cancellation system to be able to do this.

Of course, there are some high-level bits which are still missing, primarily around macros.

First, we need to expand procedural macros. Roman Golyshev (@fedochet), an intern at JetBrains working on IntelliJ Rust, did a master's thesis on supporting proc macros in IntelliJ. The core idea is to write a thin nightly shim around proc macros, and interact with it via IPC. The shim achieves two goals:

• only a small section of code uses the private nightly API, which significantly reduces maintenance burden,
• proc-macros are executed in a separate, potentially sandboxed process, and can't break the IDE.

I hope to use this approach in rust-analyzer as well.

Second, rust-analyzer currently doesn't handle conditional compilation (#[cfg] attributes). Conditional compilation is also a thorny issue for IDEs: while a compiler can just discard deactivated pieces of the source code, an IDE should provide at least basic, but, ideally, full support for them. The required infrastructure is already implemented in the rust analyzer: we can view the same source file under different #[cfg] settings simultaneously. What is missing is the actual interpretation of #[cfg]'s.

Third, while we can complete things generated by macros, we can't yet do completion inside macros. In general, this is an unsolved problem, because the input language of a macro can be arbitrary. Take a look at proc-caesar for a fun demonstration of this. But we certainly can, and should give completions in the common case.

During the course of the past months, my personal estimate for success of rust-analyzer approach has shifted from "there's a chance that this is possible" to "this will work if we put enough effort". Note that I don't have a deep knowledge of rustc, so I can't say for sure if this is a reasonable path forward for rustc, given its current state. Together with Rust compiler team, we should form a unified view for the long-term development of the projects.

So, what should we do next?

Plans

Despite being an experimental, "build it from source yourself" project, rust-analyzer is already used by some folks as a daily driver for rust. And, I must say, it did become significantly more usable recently! So one obvious way forward is to put more effort into making it a better product for users: * streamlining installation and update process, * tweaking defaults towards users, and not developers of rust-analyzer itself, * clearly separating "this is fully implemented and should just work" from more experimental features.

However, here lies a problem: only some components of rust-analyzer (chalk, salsa, ongoing rustc_lexer work) are genuinely reusable libraries. A lot of other code is currently written with experimental "let's see how far we can go in a few months" mindset. If we continue in this fashion, we will eventually build a compiler that is as good for batch compilation as rustc is good for IDEs. We should put more effort into "librarification": extracting bits of rust-analyzer and rustc into production-ready reusable libraries. That will require a lot of refactoring work on the rustc side, and I hope we can allocate more time to this!

Another open question is the relationship between existing rls and rust-analyzer. They use significantly different architectures, so neither is better than the other for all use-cases at this point. In theory, rust-analyzer should be capable of doing everything that rls can do with much better performance, but there are a lot of yaks to shave until that becomes reality. Nonetheless, I am personally intrigued by the "out of process rustc" idea.

All these grand plans mean a lot of work, and this work should be funded somehow.

Open Collective

We have decided to try out the Open Collective platform for rust-analyzer:

https://opencollective.com/rust-analyzer/

Open Collective focuses on collecting money for on-going projects, rather than for individuals.

The "core contributors" (admins) of the collective at the moment are

• Aleksey Kladov (that's me!): initial developer of rust-analyzer
• Florian Diebold: an extremely prolific contributor who joined early on and basically owns all type-related things
• Florian Gilcher: Rust team member a and a co-founder of Ferrous Systems

Ferrous Systems acts as a fiscal host to the rust-analyzer Open Collective. This means it provides a bank account as well as bookkeeping services to the collective. Ferrous Systems currently employs Aleksey Kladov, core maintainer and initial developer of the rust-analyzer project. For that reason, some of the funds may directly be used to pay part of Aleksey’s wages through Ferrous Systems. This is only for rust-analyzer related work. rust-analyzer funds are stored in a separate bank account and will be solely used for that purpose. On top of the information on the OpenCollective page, Ferrous Systems will publish a regular update on funds in this account. Ferrous Systems has also sponsored the rust-analyzer project in the past.