Discussion: Redesigning the sniper printer

Hi!

Over the past few months, I've been working out kinks and bugs in the sniper printer quite frequently. It's a very cool piece of software that does some impressive things, but I have noted several problems with the design that, in my opinion, cannot be overcome with the current architecture. I'd therefore like to discuss a complete redesign.

For context, I'll first provide a brief description of what the sniper printer does and how it does it, then I'll move on to the problems, and finally my proposed solution.

### What is the purpose of the sniper printer?

The sniper printer provides a means to parse source code into a Spoon AST, perform some transformations, and then print it back to file with minimal formatting changes to non-transformed parts. In the literature, this has been referred to as _high-fidelity code transformation_.

For applications where the output of a Spoon transformation is actually stored as production code, such as for a [merge tool](https://github.com/KTH/spork) or an [automatic repair tool](https://github.com/spoonlabs/sorald), retaining formatting is absolutely critical. Imagine getting a pull request that fixes some static analysis warning, and also reformats the entire file containing the warning. That probably doesn't get accepted.

### How does the sniper do high-fidelity printing?

The very short explanation is this: Alongside the Spoon AST, we've got a tree of source fragments. The `SniperJavaPrettyPrinter` (sniper) extends the `DefaultJavaPrettyPrinter` (DJPP), and whenever DJPP wants to print something, the sniper searches in the source fragments for something that matches what the DJPP wants to print. The sniper maintains a stack of _contexts_, which lets it know precisely where to search for the next fragment.

### Why do we need to redesign the sniper?

The current design, as cool as it is and as well as it works _most_ of the time, has three serious flaws. The first and largest flaw is that I don't believe it can ever work perfectly, by nature of its design. Essentially, we do transformations on the AST without regards to formatting, and then the sniper tries to recreate the formatting after the transformations have been applied. The more transformations that are applied, the less likely the sniper is to be able to match the source fragments correctly. The largest problems are with the small details, especially parentheses and whitespace, which often cause issues.

The second flaw is that the architecture is incredibly complex. The interactions between the DJPP and the sniper's extensions of it are very intricate, and debugging cases where the sniper prints something just a bit strangely can be very difficult.

The third flaw is that the sniper's dependence on a printing context. This can make it difficult to start printing anywhere in the source tree, and fails miserably if you combine different Spoon ASTs into one (e.g. what happens in a merge tool).

### Solutions?

I think we should draw inspiration from other works here. For example, the [IntelliJ platform has the _Program Structure Interface_](https://plugins.jetbrains.com/docs/intellij/psi.html?from=jetbrains.org), which contains formatting information alongside AST-like information. [Another work](https://doi.org/10.1016/j.entcs.2005.04.037) used what they refer to as a _Literal-Layout AST (LL-AST), which is simply an AST augmented with formatting nodes (e.g. whitespace).

Obviously, we can't turn the Spoon AST into an LL-AST, that would be unworkable for users. What I suggest is to _maintain_ an LL-AST in the background, such that any transformation on the Spoon AST is also applied to the LL-AST. Then, the sniper printer would simply print exactly what's in the LL-AST.

The obvious upsides of this is that it makes printing rather trivial, and the complexity is now moved to the transformations. I think that will be a lot easier to reason about and debug. I also think that it is a workable solution for making the sniper printing as close to perfect as possible.

The obvious downside of this is potential performance overhead, but there is already significant overhead in the current sniper printer. Although I do believe that maintaining a separate LL-AST would increase that overhead, it's entirely opt-in from a user perspective (don't use the high-fidelity mode, don't get the performance overhead).

Does anyone have any thoughts on this? My intention is to start prototyping the "backing LL-AST idea" in the coming weeks to see if it's a workable solution in practice, but before that I'd really like to discuss this with some bright minds. Maybe I've missed something :)