clearlydefined / clearlydefined/crawler

Crawler Azure Blob Writes Error With block list is invalid

Open
#664 1 comment 0 reactions 1 assignee Claimed by @ljones140 View on GitHub
Dominant language
JavaScript
Stars
60
Forks
38
Avg merge
1h 49m
Merged PRs (30d)
1

Description

# What

We are seeing a high amount of errors when writing to Azure Blob Storage with `The specified block list is invalid`

This error happens in the Clearly Defined Crawler when it upserts into Azure blob storage
[storage/storageDocStore.js](https://github.com/clearlydefined/crawler/blob/master/ghcrawler/providers/storage/storageDocStore.js#L48)

It uses the Azure streaming upload function.
```
await blockBlobClient.uploadStream(dataStream, 8 << 20, 5, options)
```
We have a high number of instances running and `The specified block list is invalid` could be caused by a race condition with the uploads. If we have more that one pod attempting to upload the same package at the same time.

Pod 1: Processing request A → uploads "go/golang/package-v1.json"
→ Starts uploadStream with blocks: block_00001, block_00002, block_00003...

Pod 2: Processing SAME request A → uploads "go/golang/package-v1.json"
→ ALSO starts uploadStream with blocks: block_00001, block_00002, block_00003...

Azure: "Wait, I have TWO sets of block_00001 for the same blob!"
→ "The specified block list is invalid" ❌

## Possible fixes

###Use upload rather than upload stream.
```
await blockBlobClient.upload(data, data.length, options)
```

This doesn't chunk the file into streams. We should consider this if the size of the blobs we are uploading are in <= 100 mb. We may need to continue to use the stream if the blobs are larger.

### Obtain Leases when uploading to blobs

Azure documentation recommends to use leases for concurrency race issues. [Azure docs](https://learn.microsoft.com/en-us/troubleshoot/azure/azure-storage/blobs/connectivity/invalid-blob-or-block-content-or-invalid-block-list#cause-2-put-operations-occur-simultaneously-for-a-blob)

That feels like a bigger change which would require a lot of testing.

### Solution for now

- Files <100MB: Use upload()
- Files ≥100MB: Use uploadStream()

And lets see if that alleviates the problem

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.