AbsaOSS / AbsaOSS/atum-service

Enhance measurement querying

Aperta
#478 1 commento 1 reazione 1 assegnatario Rivendicata da @lsulak Vedi su GitHub
enhancement
Lingua principale
Scala
Stelle
6
Fork
1
Merge medio
2g 7h
PR unite (30g)
4

Descrizione

# Atum reader requirements for measurement querying in Unify

This document captures limitations in the Atum `atum-reader` public API that force
awkward workarounds in `DataMeasurementService`
(`contexts/data-measurement/implementation`), and the changes we would like in the reader
to support the querying we need cleanly.

## Context

We store measurement checkpoints in the Atum service. Each checkpoint has:

- a **name** (e.g. the domain final checkpoint name `"Domain feeds data written"`), and
- **properties**, including `executionID` (written by the spark plugins via
`AtumSupport.Checkpoints.defaultCheckpointProperties`).

Two service queries need to filter checkpoints:

1. `listDomainMajorVersionMeasurements` — may filter by `executionIDs` (set) OR by
`checkpointName`, plus paginate.
2. `listDomainMajorVersionFinalMeasurements` — needs the latest checkpoint of a fixed
**name** AND (ideally) restricted to given `executionIDs`.

## Current reader API (as of `atum-reader_2.12` 0.8.0)

Both `FlowReader` and `PartitioningReader` expose three mutually-exclusive public methods:

| method | filters by |
| ----------------------------- | --------------------- |
| `getCheckpointsPage` | nothing (all) |
| `getCheckpointsOfNamePage` | checkpoint **name** |
| `getCheckpointsByPropertiesPage` | checkpoint **properties** (single key→value map) |

Key observation: the **private** `queryCheckpoints(..., checkpointName, checkpointProperties, ...)`
in both readers **already accepts and combines** name AND properties into the request query
params. Only the public wrappers artificially restrict callers to "name OR properties,
never both".

## Problems this causes

### Problem 1 — `listDomainMajorVersionMeasurements`: no multi-value property filter, no combined pagination

`executionIDs` is a *set*, but `getCheckpointsByPropertiesPage` accepts a single
property map and filters with equality only. There is no way to say
`executionID IN (id1, id2, ...)` in one request. So we must:

- issue one request **per executionID**, and
- because per-request pagination is meaningless once results from several executionIDs are
merged, exhaustively page each executionID (loop until `hasMore == false`), then combine.

This means the incoming `limit`/`offset` cannot be honoured for the executionID path
(we currently ignore them and return the full combined set with `hasMore = false`), and we
pay N sequential round-trips for N executionIDs.

### Problem 2 — `listDomainMajorVersionFinalMeasurements`: cannot combine name + property

We need "the checkpoint(s) named `domainFinalCheckpointName` **and** having
`executionID = X`". No public method combines name and properties, even though the private
`queryCheckpoints` already builds both params. As a result the method currently **ignores**
`executionIDs` entirely and filters by name only. This is a functional gap, not just an
efficiency one.

## Requested reader changes

Ordered by priority.

1. **Public method combining name + properties.** Expose what `queryCheckpoints` already
does internally, e.g.:

```scala
def getCheckpointsPage(
checkpointName: Option[String] = None,
checkpointProperties: Map[String, String] = Map.empty,
pageSize: Int = 10,
offset: Long = 0,
includeProperties: Boolean = false
): F[RequestResult[PaginatedResponse[...]]]
```

This alone unblocks Problem 2 and lets the final-measurements query filter by name and
`executionID` together.

2. **Multi-value / `IN`-list property filtering.** Allow a property to match any of several
values in a single request, e.g. `Map[String, Set[String]]` or a dedicated
`executionIDs` filter. This removes the N-requests-per-set workaround in Problem 1 and
restores meaningful server-side pagination across the whole matching set.

3. **Server-side combined pagination.** With (1) and (2), a single request returns a
correctly paginated page over the full filtered set, so `limit`/`offset` can be honoured
for the executionID path instead of being ignored.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.