Azure / Azure/azure-storage-python

Not able to upload bigger files into azure blob container

未关闭
#704 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
343
派生
243
PR 合并指标
30 天内没有已合并 PR

描述

### Which service(blob, file, queue) does this issue concern?
Note: for package version >= 12.0.0 please post the issue here instead: https://github.com/Azure/azure-sdk-for-python/issues
for table service, please post the issue here instead: https://github.com/Azure/azure-cosmosdb-python.

### Which version of the SDK was used? Please provide the output of `pip freeze`.
azure-storage-blob==12.11.0

### What problem was encountered?
error : azure.core.exceptions.ServiceResponseError: ('Connection aborted.', timeout('The write operation timed out'))

### Have you found a mitigation/solution?
NO. for small size of lists below function is working well but when the size is bigger around 3000 dictionaries in the list then
the code is giving error as I mentioned above. plz help how can I solve this problem.

`# to upload json files in azure blob containers
def file_upload_blob(sitename, property_list, location="", containername=os.getenv('containerName')):
"""
property_list: a big list of dictionaries,
sitename: a string value
"""
# grab the blob connection string.
try:
# connect to the container client.
container_client = ContainerClient.from_connection_string(
conn_str=os.getenv("BLOB_STORAGE_CONNECTION_STRING"),
container_name=containername
)
time.sleep(3)
logger.info(f"connected to the container {containername}")

# perform file operations
filename = os.path.join(sitename,
"{sn}_{location}{ts}.json".format(
ts=datetime.datetime.now().strftime(r'%Y-%b-%d_%H-%M-%S'),
sn=sitename,
location=location)
)
try:
if property_list:
container_client.upload_blob(
name=filename,
data=json.dumps(obj=property_list, indent=4),
blob_type='BlockBlob'
)
logger.info(f'file "{filename}" has been uploaded in azure blob storage')
else:
container_client.upload_blob(
name=filename,
data=json.dumps(obj=['got empty file'], indent=4),
blob_type='BlockBlob'
)
logger.info('found empty file! .. uploading empty file')

except Exception as e:

logger.error(f"getting error while uploading file to blob storage:{e}")
traceback.print_exc()

except Exception as e1:
logger.error(f" blob storage error: {e1}")
# logger.info("blob storage connection error:%s" % e1)
`

贡献指南

打开贡献指南

调研方向

首先检查报告中的 azure-storage-blob==12.11.0 版本和仓库指南,该指南将版本 12 或更高版本的问题指向 Azure/azure-sdk-for-python。使用提供的 ContainerClient.upload_blob 调用和包含约 3,000 个字典的 property_list 重现超时;完成标准是确定大规模上传的可操作原因或已确认的限制。

由索引模型根据 Issue 内容生成。

评估

技术栈
azure, python
领域
backend, cloud
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。