Fascinating discussion on the differences between a compiler's AST and IDEs (&comments from the IntelliJ Rust folks) https://internals.rust-lang.org/t/towards-a-second-edition-of-the-compiler/5582
Related Posts
I would have thought that invoking a C compiler would be a solved problem. Looking at Rust's cc crate there's a remarkable long tail of corner cases to fix.
Exotic CPUs, microarchitectures, compiler differences, operating system differences, etc.
https://github.com/rust-lang/cc-rs/blob/main/CHANGELOG.md
ASTs typically discard comments, and that's usually what you want.
The only time (AFAICS) that preserving comments is useful is for writing a code formatter.
Could you write a formatter in terms of a list of lexemes? A CST is a non-trivial bit of code for one use case.
I've heard of 'blub languages', where you don't realise that other languages have better abstractions until you've experienced them.
I think the same thing happens with individual features. I've seen several C++ folks miss variadic generics in Rust, but I've not written enough C++ to feel it.