pathwaycom / pathwaycom/pathway
Azube Blob Storage support
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 62.3k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
Is your feature request related to a problem? Please describe.
Pathway supports S3 and MinIO as object storage backends for reading pipeline data, but there is no equivalent connector for Azure Blob Storage. Users running Pathway on Azure infrastructure have no native way to read from or write to Blob Storage, and must resort to custom Python connectors that bypass engine-level guarantees.
Describe the solution you'd like
Add pw.io.azure_blob_storage.read and pw.io.azure_blob_storage.write, with an interface as close as possible to pw.io.s3.read and pw.io.s3.write (planned in #216).
Crates: azure_core, azure_storage, azure_storage_blobs — all MIT license. Already used by Pathway's persistence checkpoint layer; no new dependencies needed.
API:
pw.io.azure_blob_storage.read(
container,
path, # blob prefix / directory to read from
format, # "csv" | "json" | ...
...
)
pw.io.azure_blob_storage.write(
table,
container,
path, # blob prefix / directory to write into
format, # "csv" | "json" | ...
*,
write_interval,
...
)
The write connector follows the same buffering and flush semantics as planned for pw.io.s3.write (#216): rows are accumulated in an in-memory buffer, and on each flush call the connector checks whether write_interval has elapsed. If yes, the buffer is uploaded as a new blob under the configured prefix and the buffer is cleared. If no, the buffer is retained for the next flush cycle.
Describe alternatives you've considered
Azure Blob Storage exposes an Append Blob type that supports appending blocks to an existing blob, unlike S3. This could in principle allow a simpler write strategy. However, adopting it would diverge the implementation and semantics from the S3/MinIO connectors without a compelling benefit — the batched write approach is predictable, cost-efficient, and consistent across all object storage connectors. Append Blob support can be considered as a follow-up.
Additional context
Since the azure_storage_blobs crate is already in use for persistence, the authentication setup (connection strings, SAS tokens, managed identity) is already solved and can be reused directly.
Testing should follow the same pattern as the S3/MinIO integration tests. Coverage should include read, write, flush timing, buffer retention, multiple flush cycles, and format coverage (at minimum CSV and JSON). An Azurite container (the Azure Blob Storage emulator) should be used in the Docker Compose test suite in place of a real Azure account.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the existing S3/MinIO connector implementations and integration tests, then inspect the Azure Blob Storage usage in the persistence layer and the Docker Compose setup for Azurite. Done means read and write connectors match the planned S3 interface, support the listed formats and flush behavior, and integration coverage includes multiple flush cycles and buffer retention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, docker-compose, python, rust
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100