Azure / Azure/azure-functions-host
Malformed ISO timestamps in cosmosdb input binding
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
#### Investigative information
- Timestamp: 2023-10-12 20:48:19.689
- Invocation ID: 973c894ee4ea126f36f3a7f27edcc6de
- Region: West Europe
#### Repro steps
In a python azure function I have this comos db input binding
```
@app.function_name(name="UpdateChart")
@app.schedule(schedule="0 */5 * * * *",
arg_name="mytimer",
run_on_startup=True)
@app.cosmos_db_input(arg_name="cosmos",
database_name="particle-photon-db2",
container_name="weather",
sqlQuery="SELECT TOP 4500 * FROM c WHERE c.published_at > '2020-10-06T07:45:16.081Z' AND c.event='weather-data-v1' ORDER BY c.published_at DESC",
connection="MyCosmosDBConnectionString")
@app.blob_output(arg_name="outputblob",
path="weather-app-container/chart_new.html",
connection="MyStorageConnectionString")
def update_chart(mytimer: func.TimerRequest, cosmos: func.DocumentList, outputblob: func.Out[str]):
dflist = []
for i, item in enumerate(cosmos):
# Append each item as a dictionary to list
dict_item = dict(item['data'])
logging.info(item['published_at'])
dict_item['published_at'] = item['published_at']
dflist.append(dict_item)
df = pd.DataFrame.from_records(dflist)
df['published_at'] = pd.to_datetime(df['published_at'])
...
```
#### Expected behavior
This worked in V1 of the azure functions just fine
#### Actual behavior
In V2 of azure functions with this hosts.json
```
{
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[4.0.0, 5.0.0)"
},
"extensions": {
"cosmosDB": {
"connectionMode": "Gateway"
}
}
}
```
it breaks for some data with the following exception:
```python
Result: Failure
Exception: ValueError: time data "2023-10-10T20:40:16Z" doesn't match format "%Y-%m-%dT%H:%M:%S.%f%z", at position 1429. You might want to try: - passing `format` if your strings have a consistent format; - passing `format='ISO8601'` if your strings are all ISO8601 but not necessarily in exactly the same format; - passing `format='mixed'`, and the format will be inferred for each element individually. You might want to use `dayfirst` alongside this.
Stack:
File "/azure-functions-host/workers/python/3.10/LINUX/X64/azure_functions_worker/dispatcher.py", line 479, in _handle__invocation_request call_result = await self._loop.run_in_executor(
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs)
File "/azure-functions-host/workers/python/3.10/LINUX/X64/azure_functions_worker/dispatcher.py", line 752, in _run_sync_func return ExtensionManager.get_sync_invocation_wrapper(context,
File "/azure-functions-host/workers/python/3.10/LINUX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper result = function(**args)
File "/home/site/wwwroot/function_app.py", line 35, in update_chart df['published_at'] = pd.to_datetime(df['published_at'])
File "/home/site/wwwroot/.python_packages/lib/site-packages/pandas/core/tools/datetimes.py", line 1112, in to_datetime values = convert_listlike(arg._values, format)
File "/home/site/wwwroot/.python_packages/lib/site-packages/pandas/core/tools/datetimes.py", line 488, in _convert_listlike_datetimes return _array_strptime_with_fallback(arg, name, utc, format, exact, errors)
File "/home/site/wwwroot/.python_packages/lib/site-packages/pandas/core/tools/datetimes.py", line 519, in _array_strptime_with_fallback result, timezones = array_strptime(arg, fmt, exact=exact, errors=errors, utc=utc)
File "strptime.pyx", line 534, in pandas._libs.tslibs.strptime.array_strptime
File "strptime.pyx", line 355, in pandas._libs.tslibs.strptime.array_strptime
```
And indeed, after some debugging, I confirmed, that the timestamps in 'published_at' are malformed.
Example: `"2023-10-10T20:40:16Z"` is wrong, when looking at the cosmos db data, there the timestamp string is:
`"2023-10-10T20:40:16.000Z"`. Note the missing trailing ".000" before the Z. I also confirmed, that all trailing 0 before the Z are stripped.
I could not find a way my code could introduce this bug. It most likely is introduced in the binding.
Summary: `"2023-10-10T20:40:16Z"` in 'published_at' handed into my azure function is wrong and should be `"2023-10-10T20:40:16.000Z"`
#### Known workarounds
I could to some evil string magic (fill up missing 0 before the Z), but I would rather not do it.
Contributor guide
Assessment
This issue has not been assessed yet.