make get-filter-values configurable
- Dominant language
- Clojure
- Stars
- 49.3k
- Forks
- 6.8k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 653
Description
**Is your feature request related to a problem? Please describe.**
This is a long pending we have and it's about time we discuss it. Right now our get-filter-values process will fire a query to the DB to get the distinct values and it will only get the first 1000, no matter the DB you use.
Relational DBs:
```
SELECT field
FROM table
GROUP BY field
ORDER BY field ASC
LIMIT 1000
```
MongoDB:
```
db.getCollection("collection").aggregate([
{
"$group": {
"_id": {
"field": "$field"
}
}
},
{
"$sort": {
"_id": 1
}
},
{
"$project": {
"_id": false,
"field": "$_id.field"
}
},
{
"$limit": 1000
}
])
```
There's a clear issue to this approach: it's tied to the ***first*** 1000 values, so on transactional DBs or in Mongo collections, the filter values that the user will be able to pick will be skewed to the older ones (the first ones inserted).
**Describe the solution you'd like**
If we're going to keep the filter limited to 1000, we then should ask the admin if they want to pick those from the beginning, the end or probably grab the distribution of the values and show the ones near the median.
Or we can make it configurable, so the admins set the number they want to grab from each filter (e.g they might want to grab 1000 distinct values on a certain field, while others less or more.
**Describe alternatives you've considered**
Inserting filter values manually on the DB, a pain
**How important is this feature to you?**
Important for customers that are interesting in having representative filter values
**Additional context**
Discussed on a customer call that use MongoDB and can reach the 1000 distinct values very quickly
Contributor guide
Research direction
Start by locating the get-filter-values implementation and the relational and MongoDB query-generation paths described in the issue. Clarify whether the intended change is a configurable limit, a selection strategy, or both; done should be defined by an agreed behavior that avoids always selecting only the first 1000 values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clojure, mongodb, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100