facebook / facebook/buck2

Sharing and distributing log files/build metadata?

Open
#441 11 comments 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
4.4k
Forks
394
PR merge metrics
No merged PRs in 30d

Description

`buck2 log` is pretty cool, since it lets you look at what build commands a user ran and what their output is, and you can do things like `log cmd` or `log replay` to watch the build. Is it possible or recommended to 'share' these log files? What would some infrastructure for doing that look like? Maybe something like:

- Users run `buck2 build ...` a bunch
- Their logs can get synchronized to something like an HTTP endpoint (maybe a proxy that just fronts an S3 bucket)
- Something goes wrong, a user asks for help, posts their Build ID.
- You could run `buck2 log cmd --trace-id $BUILD_ID` to find out what happened.
- This would (transparently?) download the log files from some endpoint (somehow?)

There could be other useful things to derive from this possibly, assuming you had these logs. For example, you could use these logs as a direct source of build analytic information from users to derive information about build times, etc.

Is there any kind of "thing" like this inside Meta? Does this seem interesting? To be truly useful it would need some support in the core executable, I think. It seems like something *kind of* like what `buck2 rage` does, right? Except I'd want it in all cases, not just failures.

I imagine this could look like the following for OSS users:

- A `.buckconfig` key like `buck2_logs.upload_address` is set to `https://buck2-logs.aseipp.dev/`
- All `.pb.zst` files are POST'd to `$UPLOAD_ADDRESS/upload` with their trace ID as a primary key, asynchronously, by the daemon
- When a user says `buck2 log cmd --trace-id $TRACE_ID`, then:
- HTTP GET `$UPLOAD_ADDRESS/trace/$TRACE_ID`
- This should return a json object containing metadata and a new address to find the log content at
- That new address is the public URL to download the `.pb.zst` file
- This allows users to materialize logs on demand
- Something something something authorization with a bearer token
- Ideally the HTTP protocol is simple enough to implement "by hand" in a short afternoon

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.