AbsaOSS / AbsaOSS/atum-service

Enhance measurement querying

未关闭
#478 1 条评论 1 个 reaction 已指派 1 人 已被 @lsulak 认领 在 GitHub 查看
enhancement
主要语言
Scala
星标
6
派生
1
平均合并
2 天 7 小时
30 天内合并 PR
4

描述

# Atum reader requirements for measurement querying in Unify

This document captures limitations in the Atum `atum-reader` public API that force
awkward workarounds in `DataMeasurementService`
(`contexts/data-measurement/implementation`), and the changes we would like in the reader
to support the querying we need cleanly.

## Context

We store measurement checkpoints in the Atum service. Each checkpoint has:

- a **name** (e.g. the domain final checkpoint name `"Domain feeds data written"`), and
- **properties**, including `executionID` (written by the spark plugins via
`AtumSupport.Checkpoints.defaultCheckpointProperties`).

Two service queries need to filter checkpoints:

1. `listDomainMajorVersionMeasurements` — may filter by `executionIDs` (set) OR by
`checkpointName`, plus paginate.
2. `listDomainMajorVersionFinalMeasurements` — needs the latest checkpoint of a fixed
**name** AND (ideally) restricted to given `executionIDs`.

## Current reader API (as of `atum-reader_2.12` 0.8.0)

Both `FlowReader` and `PartitioningReader` expose three mutually-exclusive public methods:

| method | filters by |
| ----------------------------- | --------------------- |
| `getCheckpointsPage` | nothing (all) |
| `getCheckpointsOfNamePage` | checkpoint **name** |
| `getCheckpointsByPropertiesPage` | checkpoint **properties** (single key→value map) |

Key observation: the **private** `queryCheckpoints(..., checkpointName, checkpointProperties, ...)`
in both readers **already accepts and combines** name AND properties into the request query
params. Only the public wrappers artificially restrict callers to "name OR properties,
never both".

## Problems this causes

### Problem 1 — `listDomainMajorVersionMeasurements`: no multi-value property filter, no combined pagination

`executionIDs` is a *set*, but `getCheckpointsByPropertiesPage` accepts a single
property map and filters with equality only. There is no way to say
`executionID IN (id1, id2, ...)` in one request. So we must:

- issue one request **per executionID**, and
- because per-request pagination is meaningless once results from several executionIDs are
merged, exhaustively page each executionID (loop until `hasMore == false`), then combine.

This means the incoming `limit`/`offset` cannot be honoured for the executionID path
(we currently ignore them and return the full combined set with `hasMore = false`), and we
pay N sequential round-trips for N executionIDs.

### Problem 2 — `listDomainMajorVersionFinalMeasurements`: cannot combine name + property

We need "the checkpoint(s) named `domainFinalCheckpointName` **and** having
`executionID = X`". No public method combines name and properties, even though the private
`queryCheckpoints` already builds both params. As a result the method currently **ignores**
`executionIDs` entirely and filters by name only. This is a functional gap, not just an
efficiency one.

## Requested reader changes

Ordered by priority.

1. **Public method combining name + properties.** Expose what `queryCheckpoints` already
does internally, e.g.:

```scala
def getCheckpointsPage(
checkpointName: Option[String] = None,
checkpointProperties: Map[String, String] = Map.empty,
pageSize: Int = 10,
offset: Long = 0,
includeProperties: Boolean = false
): F[RequestResult[PaginatedResponse[...]]]
```

This alone unblocks Problem 2 and lets the final-measurements query filter by name and
`executionID` together.

2. **Multi-value / `IN`-list property filtering.** Allow a property to match any of several
values in a single request, e.g. `Map[String, Set[String]]` or a dedicated
`executionIDs` filter. This removes the N-requests-per-set workaround in Problem 1 and
restores meaningful server-side pagination across the whole matching set.

3. **Server-side combined pagination.** With (1) and (2), a single request returns a
correctly paginated page over the full filtered set, so `limit`/`offset` can be honoured
for the executionID path instead of being ignored.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。