Azure / Azure/azure-functions-python-library
Seems to be a limit on updated rows
- Vorherrschende Sprache
- Python
- Sterne
- 179
- Forks
- 79
- Ø Merge
- 2 T. 8 Std.
- Gemergte PRs (30 T.)
- 2
Beschreibung
#### Check for a solution in the Azure portal
Done. Nothing found.
#### Investigative information
- Timestamp: 2023-04-21 14:19:47.401
- Function App version: 4.17.3.3
- Function App name: cardAccess
- Function name(s) (as appropriate): PersonnelFileTrigger
- Invocation ID: 8a20b815-a601-4ab0-b61d-025147fef78b
- Region: EastUS
#### Repro steps
Provide the steps required to reproduce the problem:
Import an XMl file with more than 1,000 elements
#### Expected behavior
Import an XML file of ~4,000 personnel elements
All should appear in the target SQL server table
#### Actual behavior
Import an XML file of ~4,000 personnel elements
When I import only 1,000 records, it works fine. In the code:
start_recno = 0
end_recno = 1000
When I import only the 2nd thousand records, it works fine. In the code:
start_recno = 1000
end_recno = 2000
When I import 2,000 records I get a meaningless error message. Code:
start_recno = 0
end_recno = 2000
2023-04-21 14:19:53.808
Invalid column name 'Text4'. Invalid column name 'Text4'.
Error
Note: The target table indeed does have such a column, and it was populated by smaller runs. The reported Invalid column varies from run to run. Sometimes Text4, sometimes Text6 and Text7, sometimes MiddleName. Seemingly random, but all valid columns.
#### Known workarounds
none
#### Related information
Provide any related information
* Programming language used: Python
* Links to source: https://portal.azure.com/#view/WebsitesExtension/FunctionMenuBlade/~/code/resourceId/%2Fsubscriptions%2F8a237898-8564-4180-9423-9c9befb66299%2FresourceGroups%2Fcn100898%2Fproviders%2FMicrosoft.Web%2Fsites%2FcardAccess%2Ffunctions%2FPersonnelFileTrigger
* Bindings used
```json
{
"bindings": [
{
"name": "myblob",
"path": "ccure/Personnel/{name}",
"connection": "recurringintegrations_STORAGE",
"direction": "in",
"type": "blobTrigger"
},
{
"name": "PersonnelItems",
"type": "sql",
"direction": "out",
"commandText": "dbo.Personnel",
"connectionStringSetting": "live_100898"
},
{
"name": "CredentialItems",
"type": "sql",
"direction": "out",
"commandText": "dbo.Credential",
"connectionStringSetting": "live_100898"
}
]
}
```
Source
```python
import logging
import azure.functions as func
import xml.etree.ElementTree as ET
from datetime import datetime
def main(myblob: func.InputStream, PersonnelItems: func.Out[func.SqlRow], CredentialItems: func.Out[func.SqlRow]):
logging.info(f"Python blob trigger function processed blob \n"
f"Name: {myblob.name}\n"
f"Blob uri: {myblob.uri} \n")
XMLContent = myblob.read()
# create element tree object from string and start at root
root = ET.fromstring(XMLContent)
# iterate items
start_recno = 0
end_recno = 1000
recno = 0
debug = True
rows = []
for item in root.findall('./SoftwareHouse.NextGen.Common.SecurityObjects.Personnel'):
recno = recno + 1
if recno < start_recno:
continue
if recno >= end_recno:
break
now_datetime = datetime.now().replace(microsecond=0).isoformat()
row_d = {"ImportDate": now_datetime, "FileName": myblob.name, "RecordNumber": recno }
for child in item:
if child.tag == 'SoftwareHouse.NextGen.Common.SecurityObjects.Credential': # skip detail for now
continue
if "LastModifiedTime" in child.tag:
orig_date = child.text
date_obj = datetime.strptime(orig_date[:-9].strip(), "%m/%d/%Y %I:%M:%S %p")
iso_date = date_obj.isoformat()
row_d[child.tag] = iso_date
else: # "normal" elements
row_d[child.tag.strip()] = child.text.strip()
if debug:
logging.info(f"{row_d}")
# store in a list
rows.append(func.SqlRow(row_d))
# write the list of rows to the database
PersonnelItems.set(rows)
logging.info(f"row count: {recno}")
```
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Rechercherichtung
Beginne mit der bereitgestellten Python-main-Funktion und ihrer SQL-Ausgabebindung für PersonnelItems und vergleiche dann die im Bericht beschriebenen Läufe mit 1,000 und 2,000 Datensätzen. Prüfe, ob die sich ändernden Fehler wegen ungültiger Spalten durch die Verarbeitung von gebündelter SqlRow-Ausgabe durch die Bibliothek verursacht werden; abgeschlossen ist die Aufgabe, wenn etwa 4,000 Personalelemente ohne Spaltenfehler importiert werden und alle Zeilen erhalten bleiben.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- azure, python, sql
- Bereich
- backend, cloud, database
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 25/100