Support Avro Format for Filesystem and GCP Bucket Destinations
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### Feature description
We would love for DLT to support Avro as a destination format for file-based outputs (such as writing to the local filesystem or cloud buckets, including GCP). This should allow users to select Avro as the output format when configuring their pipelines, similarly to how Parquet is currently supported:
```
destination_config = {
"type": "filesystem",
"bucket": "gs://my-bucket",
"file_format": "avro"
}
```
**Additional context**
- Use case: Multi-source ingestion to GCP bucket, direct Avro ingest to BigQuery
- Motivation: Avro is better supported for schema management, performance, and compatibility with BigQuery than other formats for many enterprise use cases.
Would be great to see this as an option in DLT for production-data engineering workflows!
### Are you a dlt user?
Yes, I'm already a dlt user.
### Use case
Currently, DLT supports writing output files in JSONL, Parquet, and CSV formats to cloud storage (such as GCP buckets). However, direct Avro output is not available. Our team has a use case where DLT loads data from multiple sources and writes the output straight to a GCP bucket for ingestion into BigQuery. Avro is natively and efficiently supported by BigQuery, offering both schema compatibility and optimised loading performance for large datasets.
### Proposed solution
Right now, to generate Avro files, a pipeline must output in Parquet or another supported format, then perform a manual conversion or use another tool to turn the output into Avro before ingestion. This introduces extra steps, increases operational complexity, and is less efficient for direct BigQuery ingestion flows.
### Related issues
_No response_
Contributor guide
Research direction
Start by tracing filesystem destination configuration and the existing Parquet output handling for local and GCP bucket destinations. Confirm how pipeline configuration selects file formats and what tests cover those destinations; done means Avro can be selected and written for both filesystem and GCP bucket outputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, python
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100