Auto-update jobs fail with 400 — missing `X-Source-Artefact-Urn` header on proxy structure requests
- Dominant language
- Python
- Stars
- 26
- Forks
- 1
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 25
Description
### StatGPT Backend version
latest
### Short description
Some `auto_update.py` jobs fail with `400 Bad Request` from the SDMX proxy because `conceptscheme`/`codelist` structure requests are sent **without** the `X-Source-Artefact-Urn` header, which the proxy needs to disambiguate artefacts that share the same identity.
### What steps will reproduce the bug?
1. Configure a channel backed by the StatGPT SDMX proxy (SDMX 3.0) with `allow_auto_update` enabled, for a dataflow whose concept schemes share an identity that requires disambiguation (e.g. `ESTAT:CS_NA(1.14)`).
2. Run the batch auto-update script (`statgpt/admin/auto_update.py`).
3. The job loads the dataset structure and the resulting conceptscheme request to the proxy returns `400`.
### What is the expected behavior?
Auto-update jobs load the dataset structure successfully. Structure requests (`conceptscheme`, `codelist`) issued by the auto-update / reindex flow include the `X-Source-Artefact-Urn` header, exactly like the runtime dataset-loading flow does.
### What do you see instead?
A `400` error on the conceptscheme request, e.g.:
Traceback
```
httpx.HTTPStatusError: Client error '400 ' for url 'http:///statgpt/sdmx-proxy/api/v0/sdmx/3.0/structure/conceptscheme///'
File "/home/app/statgpt/common/data/sdmx/v21/sdmx_client.py", line 360, in _perform_request
resp.raise_for_status()
File "/home/app/statgpt/common/data/sdmx/v21/sdmx_client.py", line 343, in _fetch
httpx_response = await self._perform_request(req)
File "/home/app/statgpt/common/utils/async_loading_cache.py", line 62, in get
value = await loader()
File "/home/app/statgpt/common/data/sdmx/v21/sdmx_client.py", line 325, in _get
return await self._cache.get(...)
File "/home/app/statgpt/common/data/sdmx/v21/sdmx_client.py", line 224, in _get_structure
return await self._get(...)
File "/home/app/statgpt/common/data/sdmx/v21/sdmx_client.py", line 118, in conceptscheme
return await self._get_structure(...)
File "/home/app/statgpt/common/utils/async_utils.py", line 69, in sem_coro
return await coro
```
### Additional information
Root cause
The `X-Source-Artefact-Urn` header is wired through `DataflowLoader` via the `structure_extra_headers` callback (`proxy_structure_extra_headers`). However, only one entry point passes it:
- `StatGptSdmxProxyDataSource._get_dataset` (v30) builds the loader **with** the header callback:
`DataflowLoader(sdmx_client, structure_extra_headers=proxy_structure_extra_headers)` — `statgpt/common/data/statgpt_sdmx_proxy/v30/datasource.py:98`.
- The base `SdmxDataSource._load_dataset_structure_message` builds the loader **without** it:
`DataflowLoader(sdmx_client)` — `statgpt/common/data/sdmx/v21/datasource.py:413`.
The v30 proxy datasource does **not** override `_load_dataset_structure_message`, so every flow that goes through it inherits the header-less loader. That method is used by `get_dataset_structure`, config validation, and `_resolve_urn`/`resolve_config` — i.e. the paths exercised during auto-update / reindex. As a result, conceptscheme/codelist requests from auto-update omit `X-Source-Artefact-Urn` and the proxy returns `400`.
Suggested fix direction
Ensure `_load_dataset_structure_message` builds the `DataflowLoader` with the appropriate `structure_extra_headers` for the proxy datasource (e.g. override it in the v30 proxy datasource, or make the header callback a property of the datasource so all loader construction sites use it consistently).
Contributor guide
Assessment
This issue has not been assessed yet.