Discussion: Implement the formatter as a GDExtension

[khellste]

Now that the project no longer depends on Topiary, there is less reason for the formatter to be implemented as an external binary.

I propose a full rewrite in GDExtension/C++, which would have the following advantages:

• The formatter can be distributed as a single Godot addon. No need to download and install a separate binary.
• It runs natively in the Godot editor without shelling out to a separate process. (Though we should still support external editors using godot --headless -s some_cli_wrapper.gd.)
• Formatter primitives (like perhaps the GDScript AST) can be exposed to @tools in the Godot scripting environment, allowing users to more easily build custom tooling on top of the formatter.

I realize it's a big undertaking, but I think having the formatter as a standalone Godot addon would be worth it. I'm willing to take a stab at it if others on the project see value in it too.

[R2robot]

Would it still be usable from external editors?

[khellste]

Would it still be usable from external editors?

It should be doable using godot --headless -s some_new_cli_wrapper_script.gd -- arg1 arg2 arg3, where ideally the wrapper script is just a thin wrapper that marshalls the command line arguments into some formatter options and then calls into the formatter GDExtension. (Added a mention of this to the original issue.)

[R2robot]

It should be doable using godot --headless -s some_new_cli_wrapper_script.gd -- arg1 arg2 arg3

I've never run anything through godot --headless before. Does that launch a new instance of Godot, or will it use the currently running instance? If it runs a new one, it sounds like it would be a bit heavy to run for my current config which formats on save.... and I save a lot.

[khellste]

Yes, headless mode launches a new instance of Godot, which does sound slower. So you make a good point.

Maybe for the CLI we would release a standalone C++ binary (likely as fast as Rust, maybe faster), which could share almost all its code with the GDExtension.

[NathanLovato]

Some feedback on this. In short:

• Compiling rust to GDExtension is an option to consider
• I think GDExtension should be a compilation target only
• Exposing the AST should not be accounted for for this

Compiling rust to GDExtension is an option to consider

I didn't think about it yesterday when you brought this up on Discord, but actually we can compile rust code to a GDExtension without rewriting into another language. It's worth at least considering and evaluating before rewriting the entire thing into another language because it's probably only a small fraction of the work (and no risk of regressions).

The formatter is already a library that can be wrapped around. The CLI itself only imports the formatter and linter for compilation. Godot rust gdextension support: https://godot-rust.github.io/

Given that, a complete rewrite in C++ should be motivated before proceeding (I wouldn't want anyone to take on that much workload and risk facing many regressions for little practical gain). I've considered rewriting in C++ before rewriting the formatter in Rust, in case the Godot team would eventually want this to be an official part of Godot. For now it's not the case/it seems they'd rather keep this external anyway.

Another reason is using the same toolchain and coding practices as Godot for companies who'd want to customize the formatter alongside the engine. But that, like exposing the AST, is the kind of thing that almost nobody uses in practice; there have been no requests or needs expressed in that direction.

I think GDExtension should be a compilation target only

Users rely on the CLI; removing the CLI and making the program GDExtension-only (for example, by using Godot APIs and types for formatting itself) would make running the formatter through the CLI 3 to 4 orders of magnitude slower because you'd have to spawn a copy of Godot each time. Even headless it's a couple hundred ms at least on a fast computer.

It'd also be a breaking change to user workflows (requiring adding a copy of Godot in CI, changing their format commands in external code editors, etc.).

As a bonus it's way less work to keep the existing cli, formatter/linter libs, and compilation target. And it's possibly already compatible with gdextension.

Exposing the AST should not be accounted for for this

For the reason briefly mentioned above + it's unrelated to the formatter. It shouldn't come into account when considering turning the formatter into a gdextension. It's a separate potential project and body of work that's not necessarily reduced by compiling the formatter as a GDExtension or rewriting it.

---

I'm bringing up these points based on the state of the conversation. We're left with 2 benefits for users from a GDExtension, for users of the Godot addon only:

1. No need to run one command to download the formatter anymore.
2. The formatter doesn't run as a separate process anymore. That's generally transparent to users. For most people it'd shave off ~15ms running time on Windows and ~1-3ms on unix systems. (note: If the issue is the cost of spawning the process on Windows, the formatter could be tweaked to work as a daemon; that's not a lot of work and would address this part)

That comes at the cost of extra initial work (at least writing the addon in GDExtension, testing on different platforms...) probably some extra maintenance, and more compilation time (can't iterate on the addon and test it instantly anymore).

I'm open to it and I appreciate it if you do want to take on this workload for improving user experience in that way. I'm underlying this just so you're sure that you want to put in the time for these. A C++ programmer came by recently eager to port the formatter to C++ collaboratively and quickly abandoned the project. I would imagine upon realizing the actual scope of the work.

We can discuss the implementation and estimate the time different approaches would take if you'd like. In any case, I'd like to discuss and agree on the implementation before you invest a lot of time into it.

With that said, I want to thank you for bringing this up and considering contributing.

[khellste]

I hadn't considered godot-rust but it sounds like it might be the best way to avoid a massive rewrite full of regressions.

I have no Rust experience, but from a little Googling it seems it might look something like this:

• Use gdext to write a thin GDExtension wrapper around the formatter. Here is a guide for my reference.
• I can start with a basic singleton interface. Something like GDScriptFormatter.format_gdscript(source: String, options: Dictionary) -> String would be a good start.
• Pause here to see how well that worked, and worry about linter/addon support later.

Thoughts?

[NathanLovato]

Starting there to see how well it works is a good plan.

If you write it procedural rust code and avoid using too many abstractions the language provides it's pretty close to other systems programming languages minus the memory allocation code (memory allocations are implied when you instantiate a struct or Vec etc., done on the stack or heap for you). If you have any questions or need a hand along the way please reach out anytime!

The main things that can be a bit hard at first is the concept of owning borrowing and how rust constrains how you can access and mutate things as part of its compile time memory access checks. But I'm not sure you'll run much into it with a GDExtension, interfacing with Godot (most of the code will touch things on the Godot side which manages its own memory).