Buck2 should be built, tested, developed, and released with... Buck2!

Like it says on the tin. I *think* this is already the day to day at Meta (I assume), but this should be how it works in public, too! I think this is a difficult, long-term goal — but it is, I think, a worthy one. Actually, it's not even that far off right now, you can do it — and it's actually super easy thanks to `reindeer` vendoring, now! I was really impressed, and the build was quick — but it isn't quite as easy as it should be, perhaps, and I think we should strive to make it the main way to build the code. Right now, as of this ticket, it's also one that is probably not worth prioritizing at the expense of many other things. Mainly? I don't think we can capitalize quite yet on a lot of the most ideal benefits (caching, RBE) until several things get polished off. Also, there are lots of open questions to answer and related quandaries about features, support, and future plans.

Some things I can think of off the top of my head that need to be fixed before we can really really buy into this even being practical:

- No jemalloc or optimizations in the bootstrapped build, right now
- Both are pretty critical (optimization is mandatory, jemalloc could maybe be replaced with something simpler if we try?)
- There needs to be a release schedule and some binaries we can start using to bootstrap things. Otherwise, we can't even point someone to a reasonable binary to start with. #292
- There should be a publicly available build cache, at minimum.
- It should be populated by CI, at minimum. And used by subsequent CI steps, so builds are snappy as hell and fast.
- All platforms, on every commit.
- What caching technology? Where's the object storage go? Who runs the build servers? Can we use GitHub Actions to serve things? I don't know!
- Most of the integration tests don't seem to be open source yet. Those need to be ported and slowly made open source, and put under BUCK files.
- But there's no real test harness in the OSS version! I don't know what the status of things like the internal `tpx` are (I heard mumbles from Neil about potential open sourcing), or if the current `buck2 test` functionality is considered sufficient. But that's really important in the long run since a harness is kind of needed to understand how to rerun tests functionally.
- And also, what should a CI integration for Buck2 even *look* like? I actually have no clue about the prior art here for something like Bazel.
- Presumably, we'll at least need to integrate it into GitHub of course with something like BXL, too
- At the absolute minimum, we should test the buck2-on-buck2 build in GitHub Actions, since it seems like we don't do so already. I can write a patch for this. #315
- There's probably tons of optimizations to be done in RBE to make it faster in both caching and execution
- And more friendly to systems like BuildBuddy, or OTEL tracing in BuildBarn; like #256
- Cargo is... a whole thing.
- Are we going to release buck2 on crates.io? Should we? Unfortunately there's no namespacing on crates.io so as the package set proliferates we'll keep adding things, and we'll probably want automation to release all +60 packages(!!!!!!!!!) correctly.
- I honestly feel crates.io is *really* a chore at this scale of so many packages, because every crate has a little overhead, but the crate is the unit of parallel compilation in practice, so... It's not like you actually get any value out of having to specify an identical version field for 60 packages with all the same metadata...
- Should cargo always be supported as a viable build method? I think it's actually nice, but only if it doesn't create an absurd number of code paths or something or make things worse off (more on that in a minute)
- Maybe Cargo *can* be supported, but only for building the literal binary? That would make it nicer for people with `cargo` but maybe things like complex integration tests for example could still require a full bootstrap build — in this ideal world, the bootstrap build would be the release build after all. So kind of like an inversion of the current situation; buck2 for serious builds and cargo if you just really want to try this build for some reason.
- Or maybe we shouldn't, and eventually we could abandon first-party cargo support entirely?[^1] I also think this should be considered and could be nice. After all, buck2 isn't meant for *Rust* programmers, but only Rust programmers care about cargo or would care to use it, I think. It's sort of just a necessity right now. And you still want binaries even if you release on crates.io, because again, not everyone writes Rust or uses cargo! And then you wouldn't need to toil over things like release automation, etc. Just a thought.

About the build complexity thing: how many ways can buck2 even be *built* right now? There's `fbcode_build`, `buck_oss_build`... external OSS cargo builds? I think also Meta might do *internal* Cargo builds for reasons too? Am I missing some, I don't actually know? There's kind of a lot going on here and it does have some ramifications.[^2] If we actually stopped using Cargo, it would be great if it were possible to slim down everything in the `shim/` layer so we could simplify that to "OSS Buck2 build" and "Meta-Internal Buck2 build".

[^1]: I believe the generated rules from Reindeer do use `cargo` to compile crates, but that is of course hidden from the user. This is more about running `cargo build` or `buck2 build` when you want a fresh buck2 binary.

[^2]: I actually wanted to do a change to use `CARGO_PKG_VERSION` in one spot in #256 when emitting build metadata, but oops — it isn't always available in other builds like `buck_oss_build`, since it's not using cargo! So that's a chore, where does the version number come from? How do you sync it with cargo builds? Generally having so many build variants makes small things like this annoyingly complex.

All these basically mean that like, I would only use buck2 to bootstrap buck2 if I was quite literally involved in fixing these! Once they're fixed, everyone can be happy, but until then it's a rough world. That's how it is sometimes, but the goal, of course, is to change that!

---

But, aside from the benefits of dogfooding, and as a closing out thought — as a motivating thread for filing this ticket, today I ran into an issue where I had to blow away my `target/` directory from Cargo. I do this all the time I feel like. I am constantly cleaning up Cargo directories all over my computer. It's just like, I might as well set NVMe space on fire, sometimes. Occasionally, there will be bad interactions with Cargo and sometimes tools like Nix and others will do silly things like copy `target/` directories that are 10GB+ into the Nix store. When those are large, that's really expensive and irritating, and that happens for a couple of reasons whenever I use things like Sapling or Jujutsu to write patches (e.g. because there's no `.git` repo so tools think they are free to disrespect `.gitignore`, etc.) There's a big complex chain of interactions here of course, but it's not nice to feel like I need to recompile things to avoid issues like that!

That can all be worked around, but then I have to rebuild dependencies, and all this other stuff. For some reason, that kind of sounds like a familiar problem! I wonder if there would be a build system where I didn't have to worry about my local space being used up, and with a persistent remote cache so deleting things wasn't harmful... Hence, this issue. It would be pretty heavenly if we could just `buck2 build //:buck2` and get caching for free! We should really aim high on this one I think, I think but that's just my opinion.