Azure / Azure/azure-linux-extensions
Cannot efficiently query LinuxSyslogVer2v0 table
- Dominant language
- Python
- Stars
- 333
- Forks
- 278
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 4
Description
The Linux Azure Diagnostic 3.0 agent writes to the LinuxSyslogVer2v0 table in Azure Table Storage. We need to perform periodic queries based on Date and Time ranges in an efficient way. In the past, using Timestamp was fast and returned results quickly, but scanning for a 5 minute window with just Timestamp can take several minutes or more and we need results in seconds not minutes.
Example using C#
```
TableQuery query = new TableQuery().Where(
TableQuery.CombineFilters(
TableQuery.GenerateFilterConditionForDate("Timestamp", QueryComparisons.GreaterThanOrEqual, acct.StartingDateTime),
TableOperators.And,
TableQuery.GenerateFilterConditionForDate("Timestamp", QueryComparisons.LessThan, acct.EndingDateTime)));
```
The above query is extremely slow since we're only basing our query on Timestamp. We'd love to be able to use PartitionKey and RowKey but they don't seem to make any logical sense in terms of using them in a date-range based query.
Any suggestions?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the LinuxSyslogVer2v0 table and the C# TableQuery example in the issue, then review how its PartitionKey, RowKey, and Timestamp values are assigned. Reproduce the five-minute Timestamp query and measure its response time. Done means an agreed, documented approach that supports the required date-range results in seconds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp, linux
- Domain
- cloud, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100