Azure / Azure/azure-linux-extensions

Cannot efficiently query LinuxSyslogVer2v0 table

Open
#639 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
333
Forks
278
Avg merge
2d 9h
Merged PRs (30d)
4

Description

The Linux Azure Diagnostic 3.0 agent writes to the LinuxSyslogVer2v0 table in Azure Table Storage. We need to perform periodic queries based on Date and Time ranges in an efficient way. In the past, using Timestamp was fast and returned results quickly, but scanning for a 5 minute window with just Timestamp can take several minutes or more and we need results in seconds not minutes.

Example using C#

```
TableQuery query = new TableQuery().Where(
TableQuery.CombineFilters(
TableQuery.GenerateFilterConditionForDate("Timestamp", QueryComparisons.GreaterThanOrEqual, acct.StartingDateTime),
TableOperators.And,
TableQuery.GenerateFilterConditionForDate("Timestamp", QueryComparisons.LessThan, acct.EndingDateTime)));
```

The above query is extremely slow since we're only basing our query on Timestamp. We'd love to be able to use PartitionKey and RowKey but they don't seem to make any logical sense in terms of using them in a date-range based query.

Any suggestions?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the LinuxSyslogVer2v0 table and the C# TableQuery example in the issue, then review how its PartitionKey, RowKey, and Timestamp values are assigned. Reproduce the five-minute Timestamp query and measure its response time. Done means an agreed, documented approach that supports the required date-range results in seconds.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, linux
Domain
cloud, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.