Data Forwarding: Support batching, field filtering
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 21h 10m
- Merged PRs (30d)
- 635
Description
### Problem Statement
Data engineering teams using Sentry's Data Forwarding (via SQS → S3) run into two friction points at scale:
No batching — each event triggers an individual S3 write, leading to a high volume of requests and increased costs on the receiving end.
No field filtering — the full event payload is always forwarded, but most teams only need a subset (e.g. stacktraces, account, device, and network details) for downstream enrichment in their data warehouse. There's no way to include/exclude specific fields.
These teams typically want to join Sentry data with other signals in a data warehouse to analyze bug impact, and the current forwarding setup makes that expensive to operate.
### Solution Brainstorm
Allow configuring a batch window (by time or event count) so multiple events are grouped into a single forwarding write
Allow customers to define an allowlist or denylist of fields to include in forwarded payloads
Add Databricks as a native forwarding destination alongside the existing SQS/S3 option (see also [ISWF-1681](https://linear.app/getsentry/issue/ISWF-1681/integrations-data-forwarding-support-for-databricks))
### Product Area
Settings - Integrations
Contributor guide
Research direction
No files or tests are named. Start in Settings - Integrations by tracing the existing SQS-to-S3 forwarding flow, then clarify whether batching, field filtering, Databricks support, or a narrower scope is intended; done should include an agreed configuration and validation for the selected forwarding behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100