elastic / elastic/eland

pandas_to_eland - cannot write into index by alias name

Open
#747 2 comments 0 reactions 0 assignees View on GitHub
enhancement topic:dataframe
Dominant language
Python
Stars
693
Forks
112
PR merge metrics
No merged PRs in 30d

Description

Hi all,

I followed this [tutorial](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#manage-time-series-data-without-data-streams) to rollover an index without data streams.

If I go step-by-step with DevTools directly in Kibana, then works fine.

Based on my use case, I have my data in a pandas DataFrame and want to use the pandas_to_eland() function to ingest the data into an aliased index. It seems that pandas_to_eland() can't work with an aliased index.

I tried two eland versions:
eland version - 8.15.3
eland version - 8.17.0
Elasticsearch Cloud version - 8.15.3

# Steps to reproduce in DevTools

This manual flow in DevTools is working as expected (prerequisites for the step with eland)

```
PUT _ilm/policy/my-rollover-policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "1GB",
"max_age": "1m"
}
}
}
}
}
}

PUT _index_template/my_template
{
"index_patterns": [
"my-index-*"
],
"template": {
"settings": {
"index.lifecycle.name": "my-rollover-policy",
"index.lifecycle.rollover_alias": "my-alias"
}
}
}

PUT /my-index-000001
{
"aliases": {
"my-alias": {
"is_write_index": true
}
}
}

POST /my-alias/_doc
{
"message": "Hello, World!",
"@timestamp": "2024-01-01T12:00:00Z"
}
```

# Error with pandas_to_eland()

After the index is configured and created with one document, I want to ingest a second document with pandas_to_eland:

```
import eland as ed
import pandas as pd

from elasticsearch import Elasticsearch
from elastic_transport import RequestsHttpNode

if __name__ == "__main__":

es_client = Elasticsearch(
cloud_id="ES_CLOUD_ID",
api_key=("ES_API_ID", "ES_API_KEY"),
node_class=RequestsHttpNode,
request_timeout=60,
max_retries=10,
retry_on_timeout=True,
)

data = {
"message": "Hello, World 2!",
"@timestamp": "2025-01-01T12:00:00Z"
}

df = pd.DataFrame([data])

print(df)

ed.pandas_to_eland(
pd_df=df,
es_client=es_client,
es_dest_index="my-alias",
es_if_exists="append",
)
```

```
Traceback (most recent call last):
File "/fs075/sd19a/destech/devel/aklein/lib_cell_device_stats/trunk/src/eland_ingest_github.py", line 44, in
ed.pandas_to_eland(
File "/fs075/sd19a/destech/devel/CAD_shared/python_environments/libs_statistics_tool/lib/python3.10/site-packages/eland/etl.py", line 180, in pandas_to_eland
dest_mapping = es_client.indices.get_mapping(index=es_dest_index)[
File "/fs075/sd19a/destech/devel/CAD_shared/python_environments/libs_statistics_tool/lib/python3.10/site-packages/elastic_transport/_response.py", line 186, in __getitem__
return self.body[item] # type: ignore[index]
KeyError: 'my-alias'
```

## output of self.body of `elastic_transport/_response.py`, line 186

```
{'my-index-000002': {'mappings': {}}, 'my-index-000001': {'mappings': {'properties': {'@timestamp': {'type': 'date'}, 'message': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}}}}}}
```

It seems that pandas_to_eland() expects the es_dest_index name `my-alias` in the response of self.body. The content of self.body shows that only the referenced alias indexes (`my-index-000001`, `my-index-000002`) are included and therefore a KeyError is raised for `my-alias`

Could anyone confirm that this behavior is wrong? Or advise me on what I'm doing wrong.

Thanks and best regards

Contributor guide

Open the contributing guide

Research direction

Start in eland/etl.py at pandas_to_eland(), especially the indices.get_mapping() lookup shown in the traceback. Check how Elasticsearch returns mappings when the destination is an alias, then run or add a regression test covering append to an aliased index; done means the DataFrame is ingested without the KeyError.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, pandas, python
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.