dlt-hub / dlt-hub/dlt

Support Avro Format for Filesystem and GCP Bucket Destinations

Open
#3,301 0 comments 0 reactions 0 assignees View on GitHub
destination enhancement
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### Feature description

We would love for DLT to support Avro as a destination format for file-based outputs (such as writing to the local filesystem or cloud buckets, including GCP). This should allow users to select Avro as the output format when configuring their pipelines, similarly to how Parquet is currently supported:
```
destination_config = {
"type": "filesystem",
"bucket": "gs://my-bucket",
"file_format": "avro"
}
```

**Additional context**

- Use case: Multi-source ingestion to GCP bucket, direct Avro ingest to BigQuery
- Motivation: Avro is better supported for schema management, performance, and compatibility with BigQuery than other formats for many enterprise use cases.

Would be great to see this as an option in DLT for production-data engineering workflows!

### Are you a dlt user?

Yes, I'm already a dlt user.

### Use case

Currently, DLT supports writing output files in JSONL, Parquet, and CSV formats to cloud storage (such as GCP buckets). However, direct Avro output is not available. Our team has a use case where DLT loads data from multiple sources and writes the output straight to a GCP bucket for ingestion into BigQuery. Avro is natively and efficiently supported by BigQuery, offering both schema compatibility and optimised loading performance for large datasets.

### Proposed solution

Right now, to generate Avro files, a pipeline must output in Parquet or another supported format, then perform a manual conversion or use another tool to turn the output into Avro before ingestion. This introduces extra steps, increases operational complexity, and is less efficient for direct BigQuery ingestion flows.

### Related issues

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing filesystem destination configuration and the existing Parquet output handling for local and GCP bucket destinations. Confirm how pipeline configuration selects file formats and what tests cover those destinations; done means Avro can be selected and written for both filesystem and GCP bucket outputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.