epam / epam/statgpt-backend

Auto-update jobs fail with 400 — missing `X-Source-Artefact-Urn` header on proxy structure requests

Open
#418 0 comments 0 reactions 1 assignee Claimed by @Fedir-Yatsenko View on GitHub
bug
Dominant language
Python
Stars
26
Forks
1
Avg merge
1d 17h
Merged PRs (30d)
25

Description

### StatGPT Backend version

latest

### Short description

Some `auto_update.py` jobs fail with `400 Bad Request` from the SDMX proxy because `conceptscheme`/`codelist` structure requests are sent **without** the `X-Source-Artefact-Urn` header, which the proxy needs to disambiguate artefacts that share the same identity.

### What steps will reproduce the bug?

1. Configure a channel backed by the StatGPT SDMX proxy (SDMX 3.0) with `allow_auto_update` enabled, for a dataflow whose concept schemes share an identity that requires disambiguation (e.g. `ESTAT:CS_NA(1.14)`).
2. Run the batch auto-update script (`statgpt/admin/auto_update.py`).
3. The job loads the dataset structure and the resulting conceptscheme request to the proxy returns `400`.

### What is the expected behavior?

Auto-update jobs load the dataset structure successfully. Structure requests (`conceptscheme`, `codelist`) issued by the auto-update / reindex flow include the `X-Source-Artefact-Urn` header, exactly like the runtime dataset-loading flow does.

### What do you see instead?

A `400` error on the conceptscheme request, e.g.:

Traceback

```
httpx.HTTPStatusError: Client error '400 ' for url 'http:///statgpt/sdmx-proxy/api/v0/sdmx/3.0/structure/conceptscheme///'
File "/home/app/statgpt/common/data/sdmx/v21/sdmx_client.py", line 360, in _perform_request
resp.raise_for_status()
File "/home/app/statgpt/common/data/sdmx/v21/sdmx_client.py", line 343, in _fetch
httpx_response = await self._perform_request(req)
File "/home/app/statgpt/common/utils/async_loading_cache.py", line 62, in get
value = await loader()
File "/home/app/statgpt/common/data/sdmx/v21/sdmx_client.py", line 325, in _get
return await self._cache.get(...)
File "/home/app/statgpt/common/data/sdmx/v21/sdmx_client.py", line 224, in _get_structure
return await self._get(...)
File "/home/app/statgpt/common/data/sdmx/v21/sdmx_client.py", line 118, in conceptscheme
return await self._get_structure(...)
File "/home/app/statgpt/common/utils/async_utils.py", line 69, in sem_coro
return await coro
```

### Additional information

Root cause

The `X-Source-Artefact-Urn` header is wired through `DataflowLoader` via the `structure_extra_headers` callback (`proxy_structure_extra_headers`). However, only one entry point passes it:

- `StatGptSdmxProxyDataSource._get_dataset` (v30) builds the loader **with** the header callback:
`DataflowLoader(sdmx_client, structure_extra_headers=proxy_structure_extra_headers)` — `statgpt/common/data/statgpt_sdmx_proxy/v30/datasource.py:98`.
- The base `SdmxDataSource._load_dataset_structure_message` builds the loader **without** it:
`DataflowLoader(sdmx_client)` — `statgpt/common/data/sdmx/v21/datasource.py:413`.

The v30 proxy datasource does **not** override `_load_dataset_structure_message`, so every flow that goes through it inherits the header-less loader. That method is used by `get_dataset_structure`, config validation, and `_resolve_urn`/`resolve_config` — i.e. the paths exercised during auto-update / reindex. As a result, conceptscheme/codelist requests from auto-update omit `X-Source-Artefact-Urn` and the proxy returns `400`.

Suggested fix direction

Ensure `_load_dataset_structure_message` builds the `DataflowLoader` with the appropriate `structure_extra_headers` for the proxy datasource (e.g. override it in the v30 proxy datasource, or make the header callback a property of the datasource so all loader construction sites use it consistently).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.