quickwit-oss / quickwit-oss/quickwit
Support for OpenTelemetry logs with HTTP JSON/binary format
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Is your feature request related to a problem? Please describe.
The Quickwit only supports gRPC transport to send logs. The docs stated that this would be an easy feature to add :)
Describe the solution you'd like
It would be great to also see the support for HTTP JSON and/or binary (protobuf) format (so it can be easily used with e.g. opentelemetry_sdk in Rust).
Describe alternatives you've considered
No alternatives.
Additional context
What I've been experimenting with recently, is a way of having an additional proxy (such as HTTP API) solely responsible for the logs ingestion. Imagine the following, multi-tenant scenario:
- Some process which produces the logs is deployed across the multiple distinct consumers (tenants).
- The logs should be collected in a single place (Quickwit server), but each tenant, should have its own, fully isolated index compatible with Otel logs specification (thus ingesting all the logs into existing
/v1/logsendpoint is not enough). - There's an intermediary HTTP API proxy (custom application) which receives the logs from the mentioned process.
- The custom proxy sends further the logs to the Quickwit API depending on their source (unique tenant index).
My understanding is, that it could be only achieved by creating the index per tenant via API in the first place, and then invoking the ingest endpoint. Ideally, when receiving the Otel compatible logs by the custom proxy (in binary format, for example), it could be as simple as just invoking QW ingest API and passing the raw bytes to it (similar to having gRPC connection to /v1/logs as it's working currently).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing gRPC logs ingestion path and the /v1/logs endpoint mentioned in the issue. Review how the endpoint handles tenant-specific indexes, then define and test acceptance for receiving OpenTelemetry logs over HTTP in JSON and/or binary format while preserving the requested ingestion behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, rust
- Domain
- api, backend, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100