quickwit-oss / quickwit-oss/quickwit

Support for OpenTelemetry logs with HTTP JSON/binary format

Open
#5,562 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Is your feature request related to a problem? Please describe.
The Quickwit only supports gRPC transport to send logs. The docs stated that this would be an easy feature to add :)

Describe the solution you'd like
It would be great to also see the support for HTTP JSON and/or binary (protobuf) format (so it can be easily used with e.g. opentelemetry_sdk in Rust).

Describe alternatives you've considered
No alternatives.

Additional context
What I've been experimenting with recently, is a way of having an additional proxy (such as HTTP API) solely responsible for the logs ingestion. Imagine the following, multi-tenant scenario:

  • Some process which produces the logs is deployed across the multiple distinct consumers (tenants).
  • The logs should be collected in a single place (Quickwit server), but each tenant, should have its own, fully isolated index compatible with Otel logs specification (thus ingesting all the logs into existing /v1/logs endpoint is not enough).
  • There's an intermediary HTTP API proxy (custom application) which receives the logs from the mentioned process.
  • The custom proxy sends further the logs to the Quickwit API depending on their source (unique tenant index).

My understanding is, that it could be only achieved by creating the index per tenant via API in the first place, and then invoking the ingest endpoint. Ideally, when receiving the Otel compatible logs by the custom proxy (in binary format, for example), it could be as simple as just invoking QW ingest API and passing the raw bytes to it (similar to having gRPC connection to /v1/logs as it's working currently).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing gRPC logs ingestion path and the /v1/logs endpoint mentioned in the issue. Review how the endpoint handles tenant-specific indexes, then define and test acceptance for receiving OpenTelemetry logs over HTTP in JSON and/or binary format while preserving the requested ingestion behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, rust
Domain
api, backend, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.