getsentry / getsentry/sentry

Data Forwarding: Support batching, field filtering

Open
#112,653 4 comments 0 reactions 0 assignees View on GitHub
Feature Product Area: Settings - Integrations
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
21h 10m
Merged PRs (30d)
635

Description

### Problem Statement

Data engineering teams using Sentry's Data Forwarding (via SQS → S3) run into two friction points at scale:

No batching — each event triggers an individual S3 write, leading to a high volume of requests and increased costs on the receiving end.
No field filtering — the full event payload is always forwarded, but most teams only need a subset (e.g. stacktraces, account, device, and network details) for downstream enrichment in their data warehouse. There's no way to include/exclude specific fields.

These teams typically want to join Sentry data with other signals in a data warehouse to analyze bug impact, and the current forwarding setup makes that expensive to operate.

### Solution Brainstorm

Allow configuring a batch window (by time or event count) so multiple events are grouped into a single forwarding write
Allow customers to define an allowlist or denylist of fields to include in forwarded payloads
Add Databricks as a native forwarding destination alongside the existing SQS/S3 option (see also [ISWF-1681](https://linear.app/getsentry/issue/ISWF-1681/integrations-data-forwarding-support-for-databricks))

### Product Area

Settings - Integrations

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start in Settings - Integrations by tracing the existing SQS-to-S3 forwarding flow, then clarify whether batching, field filtering, Databricks support, or a narrower scope is intended; done should include an agreed configuration and validation for the selected forwarding behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.