Add heuristic for detecting string dtype -> text in Elasticsearch instead of always keyword
Open
enhancement
help wanted
topic:dataframe
- Dominant language
- Python
- Stars
- 693
- Forks
- 112
- PR merge metrics
- No merged PRs in 30d
Description
When calling `pandas_to_eland()` it'd be good to start creating `text` fields if data is:
- Longer than 255 characters (longer than default keyword)
- Contains 5+ spaces (ML magic number)
- Has a high cardinality of values compared to entries (This one is hard to detect)
Contributor guide
Research direction
Start by locating pandas_to_eland() and inspect its current string-to-Elasticsearch mapping behavior. Define and test how string length, whitespace, and cardinality should influence text versus keyword selection, including the unspecified high-cardinality case. Done means the behavior is covered and pandas data produces the intended Elasticsearch mapping.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, pandas, python
- Domain
- data, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100