Azure / Azure/azure-functions-python-library

Seems to be a limit on updated rows

オープン
#278 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
area:python-functions
主要言語
Python
スター
179
フォーク
79
平均マージ
2日 8時間
マージ済み PR(30日)
2

説明

#### Check for a solution in the Azure portal
Done. Nothing found.

#### Investigative information

- Timestamp: 2023-04-21 14:19:47.401
- Function App version: 4.17.3.3
- Function App name: cardAccess
- Function name(s) (as appropriate): PersonnelFileTrigger
- Invocation ID: 8a20b815-a601-4ab0-b61d-025147fef78b
- Region: EastUS

#### Repro steps

Provide the steps required to reproduce the problem:

Import an XMl file with more than 1,000 elements

#### Expected behavior

Import an XML file of ~4,000 personnel elements
All should appear in the target SQL server table

#### Actual behavior

Import an XML file of ~4,000 personnel elements

When I import only 1,000 records, it works fine. In the code:
start_recno = 0
end_recno = 1000

When I import only the 2nd thousand records, it works fine. In the code:
start_recno = 1000
end_recno = 2000

When I import 2,000 records I get a meaningless error message. Code:
start_recno = 0
end_recno = 2000

2023-04-21 14:19:53.808
Invalid column name 'Text4'. Invalid column name 'Text4'.
Error

Note: The target table indeed does have such a column, and it was populated by smaller runs. The reported Invalid column varies from run to run. Sometimes Text4, sometimes Text6 and Text7, sometimes MiddleName. Seemingly random, but all valid columns.

#### Known workarounds

none

#### Related information

Provide any related information

* Programming language used: Python
* Links to source: https://portal.azure.com/#view/WebsitesExtension/FunctionMenuBlade/~/code/resourceId/%2Fsubscriptions%2F8a237898-8564-4180-9423-9c9befb66299%2FresourceGroups%2Fcn100898%2Fproviders%2FMicrosoft.Web%2Fsites%2FcardAccess%2Ffunctions%2FPersonnelFileTrigger
* Bindings used

```json
{
"bindings": [
{
"name": "myblob",
"path": "ccure/Personnel/{name}",
"connection": "recurringintegrations_STORAGE",
"direction": "in",
"type": "blobTrigger"
},
{
"name": "PersonnelItems",
"type": "sql",
"direction": "out",
"commandText": "dbo.Personnel",
"connectionStringSetting": "live_100898"
},
{
"name": "CredentialItems",
"type": "sql",
"direction": "out",
"commandText": "dbo.Credential",
"connectionStringSetting": "live_100898"
}
]
}
```

Source

```python

import logging
import azure.functions as func
import xml.etree.ElementTree as ET
from datetime import datetime

def main(myblob: func.InputStream, PersonnelItems: func.Out[func.SqlRow], CredentialItems: func.Out[func.SqlRow]):

logging.info(f"Python blob trigger function processed blob \n"
f"Name: {myblob.name}\n"
f"Blob uri: {myblob.uri} \n")

XMLContent = myblob.read()
# create element tree object from string and start at root
root = ET.fromstring(XMLContent)

# iterate items
start_recno = 0
end_recno = 1000
recno = 0
debug = True
rows = []
for item in root.findall('./SoftwareHouse.NextGen.Common.SecurityObjects.Personnel'):
recno = recno + 1
if recno < start_recno:
continue
if recno >= end_recno:
break
now_datetime = datetime.now().replace(microsecond=0).isoformat()
row_d = {"ImportDate": now_datetime, "FileName": myblob.name, "RecordNumber": recno }
for child in item:
if child.tag == 'SoftwareHouse.NextGen.Common.SecurityObjects.Credential': # skip detail for now
continue
if "LastModifiedTime" in child.tag:
orig_date = child.text
date_obj = datetime.strptime(orig_date[:-9].strip(), "%m/%d/%Y %I:%M:%S %p")
iso_date = date_obj.isoformat()
row_d[child.tag] = iso_date
else: # "normal" elements
row_d[child.tag.strip()] = child.text.strip()
if debug:
logging.info(f"{row_d}")
# store in a list
rows.append(func.SqlRow(row_d))
# write the list of rows to the database
PersonnelItems.set(rows)
logging.info(f"row count: {recno}")

```

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

提供された Python の main 関数とその PersonnelItems の SQL 出力バインディングから始め、次にレポートで説明されている 1,000 件および 2,000 件のレコード実行を比較します。変化する無効な列エラーが、ライブラリによるバッチ SqlRow 出力の処理に起因するかどうかを確認します。約 4,000 件の人員要素を列エラーなしでインポートし、すべての行を保持できれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
azure, python, sql
領域
backend, cloud, database
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。