airbytehq / airbytehq/airbyte

[source-gitlab] New child stream using SubstreamPartitionRouter and incremental_dependency only gets full refreshes

Offen
#50,962 3 Kommentare 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
area/connectors community connectors/source/gitlab team/use type/bug
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

### Connector Name

source-gitlab

### Connector Version

4.3.3

### What step the error happened?

During the sync

### Relevant information

I have added a new stream for [Merge Requests Discussions](https://docs.gitlab.com/ee/api/discussions.html#merge-requests) (for which I'm happy to open a PR to contribute it) as child of Merge Requests and I was expecting it to use the parent MRs cursor and thus filter Discussions based on it.

Relevant lines from `manifest.yaml` here:

```manifest.yaml
...
merge_requests_child_streams_retriever:
$ref: "#/definitions/retriever"
partition_router:
type: SubstreamPartitionRouter
parent_stream_configs:
- type: ParentStreamConfig
parent_key: "iid"
stream: "#/definitions/merge_requests_stream"
partition_field: "iid"
incremental_dependency: true
...
base_merge_requests_child_stream:
$ref: "#/definitions/base_full_refresh_stream"
retriever: "#/definitions/merge_requests_child_streams_retriever"

merge_request_discussions_stream:
name: "merge_request_discussions"
$ref: "#/definitions/base_merge_requests_child_stream"
retriever:
$ref: "#/definitions/retriever"
partition_router:
type: SubstreamPartitionRouter
parent_stream_configs:
- type: ParentStreamConfig
parent_key: "iid"
stream: "#/definitions/merge_requests_stream"
partition_field: "iid"
$parameters:
path: "projects/{{ stream_slice.parent_slice.id }}/merge_requests/{{ stream_slice.iid }}/discussions"
```

From command line, passing the right state it works as expected (i.e. something like `poetry run source-gitlab read --config secrets/config.json --catalog catalog_mr_discussions.json --state state_mr_discussions.json`) and the connector only pulls a small number of Discussions based on the Merge Requests cursor.

On Airbyte Platform, however, the connector never receives the state for Merge Request Discussions and therefore that always leads to a full refresh.

I verified that the connection saved the complete state since there is a `parent_state` object with the right Merge Requests cursor for the new Discussions stream (Ids and names have been anonymized):

```json
{
"streamDescriptor": {
"name": "merge_request_discussions"
},
"streamState": {
"states": [
// Partition states omitted for brevity
],
"parent_state": {
"merge_requests": {
"states": [
{
"cursor": {
"updated_at": "2025-01-07T14:28:40.290Z"
},
"partition": {
"id": 12345678,
"parent_slice": {
"id": "org123%2Fgroup-a%2Fgroup-b%2Fexample-project",
"name": "example-project",
"namespace_id": 11111111,
"namespace_name": "group-b",
"name_with_namespace": "organization / group-a / group-b / example-project",
"namespace_full_path": "org123/group-a/group-b"
}
}
}
]
}
}
}
},
```

But it seems that the connector does not get the full input state, I went into the workspaces docker volume and the `input_state.json` file only contains a state for Merge Requests and nothing else, which explains why the full refresh on Discussions:

```shell
root@56b7143deb47:/data# cat 141/0/input_state.json
[{"type":"STREAM","stream":{"stream_descriptor":{"name":"merge_requests"},"stream_state":{"states":[{"cursor":{"updated_at":"2025-01-07T14:28:40.290Z"},"partition":{"id":12345678,"parent_slice":{"id":"org123%2Fgroup-a%2Fgroup-b%2Fexample-project","name":"example-project","namespace_id":11111111,"namespace_name":"group-b","name_with_namespace":"organization / group-a / group-b / example-project","namespace_full_path":"org123/group-a/group-b"}}}]}}}]
```

I'm not sure if it's something specific to Gitlab and the stream I'm trying to add or something platform broad like the [issue described here](https://github.com/airbytehq/airbyte/issues/28278).

### Relevant log output

```shell

```

### Contribute

- [x] Yes, I want to contribute

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.