AbsaOSS / AbsaOSS/atum-service

Enhance measurement querying

オープン
#478 コメント 1 件 リアクション 1 件 担当者 1 名 @lsulak が担当を希望しています GitHub で見る
enhancement
主要言語
Scala
スター
6
フォーク
1
平均マージ
2日 7時間
マージ済み PR(30日)
4

説明

# Atum reader requirements for measurement querying in Unify

This document captures limitations in the Atum `atum-reader` public API that force
awkward workarounds in `DataMeasurementService`
(`contexts/data-measurement/implementation`), and the changes we would like in the reader
to support the querying we need cleanly.

## Context

We store measurement checkpoints in the Atum service. Each checkpoint has:

- a **name** (e.g. the domain final checkpoint name `"Domain feeds data written"`), and
- **properties**, including `executionID` (written by the spark plugins via
`AtumSupport.Checkpoints.defaultCheckpointProperties`).

Two service queries need to filter checkpoints:

1. `listDomainMajorVersionMeasurements` — may filter by `executionIDs` (set) OR by
`checkpointName`, plus paginate.
2. `listDomainMajorVersionFinalMeasurements` — needs the latest checkpoint of a fixed
**name** AND (ideally) restricted to given `executionIDs`.

## Current reader API (as of `atum-reader_2.12` 0.8.0)

Both `FlowReader` and `PartitioningReader` expose three mutually-exclusive public methods:

| method | filters by |
| ----------------------------- | --------------------- |
| `getCheckpointsPage` | nothing (all) |
| `getCheckpointsOfNamePage` | checkpoint **name** |
| `getCheckpointsByPropertiesPage` | checkpoint **properties** (single key→value map) |

Key observation: the **private** `queryCheckpoints(..., checkpointName, checkpointProperties, ...)`
in both readers **already accepts and combines** name AND properties into the request query
params. Only the public wrappers artificially restrict callers to "name OR properties,
never both".

## Problems this causes

### Problem 1 — `listDomainMajorVersionMeasurements`: no multi-value property filter, no combined pagination

`executionIDs` is a *set*, but `getCheckpointsByPropertiesPage` accepts a single
property map and filters with equality only. There is no way to say
`executionID IN (id1, id2, ...)` in one request. So we must:

- issue one request **per executionID**, and
- because per-request pagination is meaningless once results from several executionIDs are
merged, exhaustively page each executionID (loop until `hasMore == false`), then combine.

This means the incoming `limit`/`offset` cannot be honoured for the executionID path
(we currently ignore them and return the full combined set with `hasMore = false`), and we
pay N sequential round-trips for N executionIDs.

### Problem 2 — `listDomainMajorVersionFinalMeasurements`: cannot combine name + property

We need "the checkpoint(s) named `domainFinalCheckpointName` **and** having
`executionID = X`". No public method combines name and properties, even though the private
`queryCheckpoints` already builds both params. As a result the method currently **ignores**
`executionIDs` entirely and filters by name only. This is a functional gap, not just an
efficiency one.

## Requested reader changes

Ordered by priority.

1. **Public method combining name + properties.** Expose what `queryCheckpoints` already
does internally, e.g.:

```scala
def getCheckpointsPage(
checkpointName: Option[String] = None,
checkpointProperties: Map[String, String] = Map.empty,
pageSize: Int = 10,
offset: Long = 0,
includeProperties: Boolean = false
): F[RequestResult[PaginatedResponse[...]]]
```

This alone unblocks Problem 2 and lets the final-measurements query filter by name and
`executionID` together.

2. **Multi-value / `IN`-list property filtering.** Allow a property to match any of several
values in a single request, e.g. `Map[String, Set[String]]` or a dedicated
`executionIDs` filter. This removes the N-requests-per-set workaround in Problem 1 and
restores meaningful server-side pagination across the whole matching set.

3. **Server-side combined pagination.** With (1) and (2), a single request returns a
correctly paginated page over the full filtered set, so `limit`/`offset` can be honoured
for the executionID path instead of being ignored.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。