AbsaOSS / AbsaOSS/atum-service
Enhance measurement querying
- 主要语言
- Scala
- 星标
- 6
- 派生
- 1
- 平均合并
- 2 天 7 小时
- 30 天内合并 PR
- 4
描述
# Atum reader requirements for measurement querying in Unify
This document captures limitations in the Atum `atum-reader` public API that force
awkward workarounds in `DataMeasurementService`
(`contexts/data-measurement/implementation`), and the changes we would like in the reader
to support the querying we need cleanly.
## Context
We store measurement checkpoints in the Atum service. Each checkpoint has:
- a **name** (e.g. the domain final checkpoint name `"Domain feeds data written"`), and
- **properties**, including `executionID` (written by the spark plugins via
`AtumSupport.Checkpoints.defaultCheckpointProperties`).
Two service queries need to filter checkpoints:
1. `listDomainMajorVersionMeasurements` — may filter by `executionIDs` (set) OR by
`checkpointName`, plus paginate.
2. `listDomainMajorVersionFinalMeasurements` — needs the latest checkpoint of a fixed
**name** AND (ideally) restricted to given `executionIDs`.
## Current reader API (as of `atum-reader_2.12` 0.8.0)
Both `FlowReader` and `PartitioningReader` expose three mutually-exclusive public methods:
| method | filters by |
| ----------------------------- | --------------------- |
| `getCheckpointsPage` | nothing (all) |
| `getCheckpointsOfNamePage` | checkpoint **name** |
| `getCheckpointsByPropertiesPage` | checkpoint **properties** (single key→value map) |
Key observation: the **private** `queryCheckpoints(..., checkpointName, checkpointProperties, ...)`
in both readers **already accepts and combines** name AND properties into the request query
params. Only the public wrappers artificially restrict callers to "name OR properties,
never both".
## Problems this causes
### Problem 1 — `listDomainMajorVersionMeasurements`: no multi-value property filter, no combined pagination
`executionIDs` is a *set*, but `getCheckpointsByPropertiesPage` accepts a single
property map and filters with equality only. There is no way to say
`executionID IN (id1, id2, ...)` in one request. So we must:
- issue one request **per executionID**, and
- because per-request pagination is meaningless once results from several executionIDs are
merged, exhaustively page each executionID (loop until `hasMore == false`), then combine.
This means the incoming `limit`/`offset` cannot be honoured for the executionID path
(we currently ignore them and return the full combined set with `hasMore = false`), and we
pay N sequential round-trips for N executionIDs.
### Problem 2 — `listDomainMajorVersionFinalMeasurements`: cannot combine name + property
We need "the checkpoint(s) named `domainFinalCheckpointName` **and** having
`executionID = X`". No public method combines name and properties, even though the private
`queryCheckpoints` already builds both params. As a result the method currently **ignores**
`executionIDs` entirely and filters by name only. This is a functional gap, not just an
efficiency one.
## Requested reader changes
Ordered by priority.
1. **Public method combining name + properties.** Expose what `queryCheckpoints` already
does internally, e.g.:
```scala
def getCheckpointsPage(
checkpointName: Option[String] = None,
checkpointProperties: Map[String, String] = Map.empty,
pageSize: Int = 10,
offset: Long = 0,
includeProperties: Boolean = false
): F[RequestResult[PaginatedResponse[...]]]
```
This alone unblocks Problem 2 and lets the final-measurements query filter by name and
`executionID` together.
2. **Multi-value / `IN`-list property filtering.** Allow a property to match any of several
values in a single request, e.g. `Map[String, Set[String]]` or a dedicated
`executionIDs` filter. This removes the N-requests-per-set workaround in Problem 1 and
restores meaningful server-side pagination across the whole matching set.
3. **Server-side combined pagination.** With (1) and (2), a single request returns a
correctly paginated page over the full filtered set, so `limit`/`offset` can be honoured
for the executionID path instead of being ignored.
贡献指南
这个仓库没有索引到贡献指南
评估
这个 Issue 还没有评估数据。