elastic / elastic/elasticsearch-py

Add option to `bulk` indexer to automatically add nested mapping where appropriate

Open
#855 1 comment 0 reactions 0 assignees View on GitHub
tracking
Dominant language
Python
Stars
4.4k
Forks
1.2k
Avg merge
17h 17m
Merged PRs (30d)
34

Description

I am using `bulk` to index this JSON dict:
```
{
"project.nested.repeated.addresses":[
{
"status":"current",
"city":"New York",
"zip":33333,
"state":"NY",
"address":"789 Any Avenue",
"numberOfYears":2
},
{
"status":"previous",
"city":"Hoboken",
"zip":44444,
"state":"NJ",
"address":"321 Main Street",
"numberOfYears":3
}
],
"project.nested.repeated.first_name":"Jane",
"project.nested.repeated.last_name":"Doe"
}
```

I would like the addresses field to be [nested in the Elasticsearch sense](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html).

I have code that [manually crawls the schema and adds nested mappings](https://github.com/DataBiosphere/data-explorer-indexers/blob/e0aace1c166aa9c7eb97e87d8ed791f0d5630fb8/bigquery/indexer.py#L96).

It would be nice if I didn't have to manually put `nested` everywhere. This seems like a common use case:

If there is a nested JSON field, make the corresponding Elasticsearch field nested.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the bulk indexer implementation and the schema-crawling example in bigquery/indexer.py around the linked lines. Define how nested JSON fields should be detected and mapped, then add coverage showing that bulk indexing automatically creates Elasticsearch nested mappings where appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.