Azure / Azure/azure-sdk-for-python

Reading append blobs while they're being modified without triggering a ResourceModifiedError exception

Aperta
#39,817 5 commenti 0 reazioni 2 assegnatari Rivendicata da @jalauzon-msft Vedi su GitHub
Client customer-reported feature-request needs-team-attention Service Attention Storage
Lingua principale
Python
Stelle
5.6k
Fork
3.4k
Merge medio
2g 2h
PR unite (30g)
213

Descrizione

We have Azure Container App Jobs that read JSON files containing security log events from a storage account and then insert the events into Delta tables in another storage account..

These are append blobs containing [Advanced Hunting](https://learn.microsoft.com/en-us/defender-endpoint/api/raw-data-export-storage) events that are being written to every X minutes. We do not have any control of how they are being written.

Between invocations we are keeping track of how much of the file we have processed (offset) and we stream into a PyArrow buffer:

```python
buffer = pa.allocate_buffer(length)
output = pa.output_stream(buffer)

# some offset and length calculation

download = client.download_blob(
offset=offset,
length=length,
progress_hook=progress_callback,
)

bytes_read = download.readinto(output)
```

If we want to download 2GB and use the `max_single_get_size` with its default value of `32MB` then the the Python SDK will download multiple chunks of size 4MB. Unfortunately, if the blob is being written to at the same time we are downloading one of the chunks it will notice that the ETAG has changed and it will throw a `ResourceModifiedError`.

This thread explains in detail what is going on https://github.com/Azure/azure-sdk-for-python/issues/30233#issuecomment-1535567309.

While this makes sense for a regular blob, why is this behaviour necessary for an append blob? Is there any match condition that will allow us to ignore the ETAG change?

The only way I can think of is either:

1. Increasing the `max_single_get_size`. However, even when running in an ACA large downloads are unstable.
2. Doing the chunking ourselves.

Are we doing something wrong here?

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.