facebook / facebook/buck2

API design for BuckEvent data

Open
#226 7 comments 3 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
4.4k
Forks
394
PR merge metrics
No merged PRs in 30d

Description

Today, the build telemetry is done pretty much through Scribe (Meta's internal message queue system) into Scuba (Meta's telemetry platform).

This resulted in Buck2 implementing the EventSink trait for Scribe only:

```rust
/// A sink for events, easily plumbable to the guts of systems that intend to produce events consumeable by
/// higher-level clients. Sending an event is synchronous.
pub trait EventSink: Send + Sync {
/// Sends an event into this sink, to be consumed elsewhere. Explicitly does not return a Result type; if sending
/// an event does fail, implementations will handle the failure by panicking or performing some other graceful
/// recovery; callers of EventSink are not expected to handle failures.
fn send(&self, event: BuckEvent);

/// Sends a control event into this sink, to be consumed elsewhere. Control events are not sent to gRPC clients.
fn send_control(&self, control_event: ControlEvent);

/// Collects stats on this sink (e.g. messages accepted, rejected).
fn stats(&self) -> Option;
}
```

In here:

- `BuckEvent` is defined in `buck_data/data.proto` which is great for backward compatibility

- `ControlEvent` is an enum
```rust
/// An event that can be produced by Buck2 that is not intended to be presented to the user, but rather is used to
/// communicate with other parts of Buck2.
#[derive(Clone, From)]
pub enum ControlEvent {
/// A command result, produced upon completion of a command.
CommandResult(Box),
/// A progress event from this command. Different commands have different types.
PartialResult(PartialResult),
}
```
where `CommandResult` and `PartialResult` are both defined in `buck2_cli_proto/daemon.proto`. Not sure yet how backward
compatible guarentee this is just yet.

- `EventSinkStats` is a struct defined in `buck2_events` crate, not guarantee to be backward compatible.

---

It would be nice if we could define the telemetry data in a unified interface/data format similar to [Bazel's BEP](https://bazel.build/remote/bep) so that third parties could implement the server side for it. At the very least, having the data structures defined in a unified protobuf would help assuring that the future changes are backward compatible.

Contributor guide

Open the contributing guide

Research direction

Start by reading buck_data/data.proto and the EventSink trait described in the issue, then compare ControlEvent and CommandResult/PartialResult in buck2_cli_proto/daemon.proto and EventSinkStats in the buck2_events crate. Review Bazel's BEP as the stated reference. Done would mean a decided unified protobuf interface with documented backward-compatibility expectations for telemetry data.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.